LouFranco.com - RSS

Ant Colony in F#

Lou Franco — 2008-04-30T21:09:38-04:00

My colleague, Rick, is playing around with F#. He made this ant colony in F# that's based on Rich's clojure one. If you know clojure, and you want to see what F# is all about, you might want to take a look.

Clojure Day Blog Roundup

Lou Franco — 2008-03-21T13:36:15-04:00

If you were there and blogged it, let me know:

Here's what I found so far:

20 Days of Clojure: Day 20

Lou Franco — 2008-03-21T07:47:30-04:00

Last night was our Clojure event with Rich Hickey. BTW, if you were there, regular meeting of the Western Mass Developers Group are every other Thursday at Panera Bread in Hadley (7pm) --- most of the time, we just sit around and talk -- very informal -- hope to see you there.

Back to the presentation -- hopefully the slides/video/etc will be up somewhere soon -- Rich focussed on the concurrency features of Clojure (Vars, Refs, and Agents). First he showed us the state of concurrency support in Java/C# today (locking) and identified the problem as having direct references to mutable objects. Clojure offers only immutable atoms and datastructures so that's how he addresses one part of the problem. However, since the mutability is how the world works, Clojure models change as a change to what a reference points to. So, if you view the world through refs, it will appear to change. Then he introduced the controlled ways in which refs can change.

1. Vars are essentially thread local variables. They are a reference whose changes can only be seen in a single thread -- each thread gets an independent and isolated view of the variable. Like thread-local variables -- they are a very simple way to take code that looks like it's manipulating global state and give each thread its own isolated copy.

2. Refs are mutable references to immutable objects. They can only be changed inside a transaction and all changes to all refs in a transaction appear to have happened at the time the transaction ends (to other threads). Inside the transaction, the changes are immediate. When two transactions are open and change the same ref, the second one to finish is automatically retried (so you should have no side-effects inside of transactions). All Refs read inside of a transaction will return the value that they had at the start of the transaction (so are consistent with each other).

3. Agents (which I have explained before) are for asynchronous changes to a reference. Each agent hold a value and you send actions to it to change the value. Only one action will happen to an agent at a time and you can read the current state of the agent any time. Rich spent a little time to caution us against any comparison to Erlang's actors. They are a different concept with different tradeoffs. Agents have the advantage that they can be read from any thread conveniently (just dereference) whereas actors require sending a message to read (which is asynchronous). Clearly, Erlang's benefit is location transparency (for distributed code) -- which is what it was designed for. Rich hinted that Clojure might have an Actor concept, but it would not be unified with Agents.

What was new to me with agents is that there are two types of message sending mechanisms -- send and send-off (which appear to be undocumented right now) -- Rich used send-off which dedicates a thread to the agent (rather than run the action from a thread-pool). This is how you have to do it if agents block at all (which ants do because they have a small sleep).

Then, he showed us an ant simulation -- I think he will make the code available. In short, there is a square world of cells, each one holds a Ref to what is in the cell (food, ant, pheromones, home base). Ants are represented by agents, which are sent a behave message. Behave picks a random action for the ant to do (move, turn, pick up food, drop food) and then mutates the world of cells in a transaction, then it sends itself another behave action. There is an agent responsible for painting the world, and another which evaporates pheromones in the background.

Anyway, the demo was impressive -- since painting makes a copy of the world in a transaction, it has a completely consistent view of the world. Refs make sure that all changes to them are in transactions, so you have language support for where you need cooperation (contrasted to locking, which is not enforced). Agents also help you model the real world in a way that a coordinated single-looped simulation (calling behave on ants in a loop, then paint over and over again) could not.

And clojure's agent mutating mechanism (sending a function to it), means that agents don't have to have any knowledge of the messages that might be sent to it (again contrasted to Erlang Actors). Finally, various messages can take different times to complete and that would be represented in the simulation --- some ants might complete several small actions in the time it took another to complete one (which would not be the case in a behave loop).

I'll have more on this when the slides, code, etc are available.

20 Days of Clojure: Day 19

Lou Franco — 2008-03-19T07:41:01-04:00

Ok, yesterday I decided to try to implement a defclass macro which would hobble multimethods and force you to have some kind of class inheritance for polymorphism. Today, I'll try to implement that macro.

First, I made this macro, which I think will be useful

  (defmacro sym2key [s]
    `(keyword (name (quote ~s)))
  )

Works like this:

  user=> (sym2key rect)
  :rect

(I wrote the above this morning, and after several hours, I didn't get too far)

I had to cheat a lot, but I finally got something -- here is my final OO minilanguage

  (defclass shape
    (defabstractmeth area)
  )
  (defclass rect
    (defctor (fn [w h] (setprop w) (setprop h) ))
    (defmeth area (fn [] (* (:w this) (:h this))))
  )
  (defclass circle
    (defctor (fn [r] (setprop r) ))
    (defmeth area (fn [] (* (:r this) (:r this) (. Math PI))))
  )

I got that to be processed by these macros:

  (defmacro sym2key [s]
    `(keyword (name (quote ~s)))
  )

  (defmacro defclass [name & body]
    (cons 'do
    (loop [classbody# body parts# nil]
      (if classbody#
        (recur (rest classbody#) (cons (concat (first classbody#) `(~name)) parts#))
        parts#)))

  )

  (defmacro defabstractmeth [name class] `(defmulti ~name :type) )

  (defmacro setprop [x] { (sym2key x) x })

  (defmacro defctor [fndef name]
    (let [  arglist# (second fndef)
        fnbody# (map second (nthrest fndef 2) )
        obj# (reduce merge
            {:type `(sym2key ~name)}
            (map assoc (repeat {})
              (map eval
                (map sym2key fnbody#)) fnbody#))
      ]

      `(defn ~name ~arglist# ~obj#)
    )
  )

  (defmacro defmeth [meth fndef name]
    (let [  arglist# (conj (second fndef) 'this)
        fnbody# (first (nthrest fndef 2))
        namesym# (eval `(sym2key ~name))

      ]

      `(defmethod ~meth ~namesym# ~arglist# ~fnbody#)
    )
  )

This code shows it in action:

  (prn (rect 10 20))
  (prn (area (rect 10 20)))
  (prn (circle 10))
  (prn (area (circle 10)))

Outputs:

  {:type :rect, :h 20, :w 10}
  200
  {:type :circle, :r 10}
  314.1592653589793

I am way too tired to explain this -- suffice to say, this is kind a crazy way to make something, but it sure beats not being able to make it. My ctor implementation is so crazy, that you should just ignore it -- defclass and defmeth are worth looking at (ctor is a major hack -- assumes only setprop calls and turns them into a map).

20 Days of Clojure: Day 18

Lou Franco — 2008-03-18T21:06:44-04:00

I just got around to watching Clojure TV on sequences, which I guess I should have done earlier, but I guess I was too lazy --- thanks, I'm here all week.

Ok -- since clearly, day 20 is going to be about Clojure Day in Northampton, I only have two days to explore more of clojure. I haven't even gotten to multimethods, which are very cool, but pretty simple to understand. I want to keep working with macros.

The hardest thing about thinking about macros is that not using a language that has one, I haven't started to think in macros yet. Kind of reminds me of Clint Eastwood in Firefox who has to keep remembering to think in Russian in order to fly his stolen super-secret Soviet jet.

Ok, since I think in OO, let me try to do some OO in clojure via macros -- maybe I get to use multimethods after all (let's hobble them with conventional OO polymorphism)

  (defclass rect
    (defctor (fn [w h] (setprop w) (setprop h) ))
    (defmeth area (fn [] (* w h)))
  )

  (defclass circle
    (defctor (fn [r] (setprop r)))
    (defmeth area (fn [] (* r r (. Math PI))))
  )

I want that to turn into

  (defn rect [w h] {:type :rect :w w :h h})
  (defmulti area :type)
  (defmethod area :rect [r]
    (* (:w r) (:h r)))

  (defn circle [r] {:type :circle :r r})
  (defmulti area :type)
  (defmethod area :circle [c]
    (* (. Math PI) (* (:r c) (:r c))))

I'm not even sure this is possible -- I just checked to see if it's legal to defmulti the same name, and it's not really because it kills the definition for rect. Luckily, I have a precedent to fall back on -- in Java/C# style OO, area is only polymorphic if it came from a superclass. I can change the class structure to this:

  (defclass shape
    (defabstractmeth area)
  )

  (defclass rect :extends shape
    (defctor (fn [w h] (setprop w) (setprop h) ))
    (defmeth area (fn [] (* w h)))
  )

  (defclass circle :extends shape
    (defctor (fn [r] (setprop r)))
    (defmeth area (fn [] (* r r (. Math PI))))
  )

This is so ugly, I don't even know if I want to go through with this. I'll try for tomorrow.

20 Days of Clojure: Day 17

Lou Franco — 2008-03-17T19:02:52-04:00

(clojure day in Northampton, MA is Thursday, March 20th)

I know I'll reveal myself as being tragically unhip for not using emacs or aquamacs or whatever (I'm a VI man after all), but I started this TextMate Bundle for clojure. I'll grow it as time goes by and probably move it to its own page or some repository for bundles once I figure this bundling stuff out.

I do some rudimentary syntax coloring and automatically expand defn[Tab], def[Tab], let[Tab], and if[Tab].

If you use TextMate and have some ideas for things to add, please send me a note.

20 Days of Clojure: Day 16

Lou Franco — 2008-03-16T14:06:02-04:00

Ok, time to look at macros. Being new to lisps, I'm especially new to macros. I have been meaning to read On Lisp for a few months, so like the SICP, I'll show you stuff from the macros chapter and then try to write them in clojure.

In, On Lisp (Chapter 7), Paul Graham writes:

The expression generated by the macro will be evaluated in place of the original macro call. A macro call is a list whose first element is the name of the macro. What happens when we type the macro call into the toplevel? Lisp notices that [it] is the name of a macro, and
builds the expression specified by the definition, then
evaluates that expression in place of the original macro call

Macros simply allow you to write code that gets executed at compile time to generate code, which is then run. There really isn't anything quite like them in popular non-lisp languages, though there are a couple of .NET languages with language altering macros. They are much more powerful than the preprocessor macros in C, and C++ template meta-programming can sometimes get you the effect, but templates were not designed for this, so it is too cumbersome most of the time.

We have been using macros in clojure throughout this series. In fact, since Lisps evaluate all of their arguments before calling a function, whenever you see code where all of the arguments are not evaluated, you are either looking at something built-in to the language (e.g. if) or a macro (e.g. cond). You can look at boot.clj to see the definitions of them. For example, here's lazy-cons:

  (defmacro lazy-cons [x & body]
    (list 'fnseq x (list* `fn [] body)))

Remember, we used lazy-cons to make lazy sequences. lazy-cons acts like cons, except it does not evaluate the rest-expr until rest is called on the resulting list. We can use macroexpand-1 (which returns the unevaluated macro expansion of a macro call) to see how this works:

  user=> (defn rfn [] (prn "Called") '(3))
  #
  user=> (rfn)
  "Called"
  (3)
  user=> (def lazylist (lazy-cons 1 (rfn)))
  #
  user=> (first lazylist)
  1
  user=> (rest lazylist)
  "Called"
  (3)
  user=> (rest lazylist)
  (3)
  user=> (macroexpand-1 '(lazy-cons 1 (rfn)))
  (fnseq 1 (clojure/fn [] (rfn)))
  user=> (first (eval (macroexpand-1 '(lazy-cons 1 (rfn)))))
  1
  user=> (rest (eval (macroexpand-1 '(lazy-cons 1 (rfn)))))
  "Called"
  (3)

So, a lazy-cons is just a short hand to using a fnseq and wrapping the rest in a function that will be called whenever rest is called on the fnseq. That's a typical use of a macro -- codifying common uses to make your programs shorter. And, lazy-cons cannot be implemented as a function, because if it were, its arguments would need to be evaluated, which defeats the purpose.

Ok, let's write some macros (Documentation on the clojure macro page and the clojure reader page). On Lisp, section 7.3 (Defining Simple Macros) presents this one for while.

  (defmacro while (test &body body)
    `(do () ((not ,test)) ,@body))

Here it is in clojure:

  (defmacro while [test & body]
    `(loop [] (if ~test (do ~@body (recur)))))

In clojure, you don't have many mutable objects -- here's how you use while with a thread-local var binding:

  (def x 10)
  (prn (macroexpand '(while (> x 0) (prn x) (set! x (dec x)))))
  (binding [x 10] (while (> x 0) (prn x) (set! x (dec x))))

Here's the output:

  (loop [] (if (> x 0) (do (prn x) (set! x (dec x)) (recur))))
  10
  9
  8
  7
  6
  5
  4
  3
  2
  1

20 Days of Clojure: Day 15

Lou Franco — 2008-03-15T13:43:40-04:00

Christophe Grand made two observations about my parallel prime code, which improve it quite a bit.

First, since primes-in-list is lazy, you have to (touch) it to get it to be completely created in the agent thread. That leaves you with this:

  (defn par-primes-in-list [potential-primes known-primes]
    (let [ x 10 x-times-2 (* x 2)]
      (! w1 (fn[w pp kp] (touch (primes-in-list pp kp))) (take x potential-primes) known-primes)
      (! w2 (fn[w pp kp] (touch (primes-in-list pp kp))) (take x (drop x potential-primes)) known-primes)
      (await w1 w2)

      (let [new-primes (concat @w1 @w2) ]
        (if (= (count new-primes) 0)
          (recur (drop x-times-2 potential-primes) known-primes)
          (let [real-primes
            (if (> (first new-primes) x-times-2) new-primes (primes-in-list new-primes []))]
            (lazy-cat real-primes (par-primes-in-list
              (drop x-times-2 potential-primes)
              (into known-primes real-primes))))))))

But, the bigger difference is picking the number of potential-primes to evaluate intelligently. I had experimented with trying to do this dynamically, but without the fix above, anything you do will probably result in primes being found lazily in the main thread. Here's Christophe's code:

    (defn par-primes-in-list [potential-primes known-primes]
        (let [p (first potential-primes) x (quot p 2)]
            (! w1 (fn[w pp kp] (touch (lazy-primes-in-list pp kp)))
                (take x potential-primes) known-primes)
            (! w2 (fn[w pp kp] (touch (lazy-primes-in-list pp kp)))
                (take (- p x) (drop x potential-primes)) known-primes)
            (await w1 w2)

            (let [new-primes (concat @w1 @w2) ]
                (lazy-cat new-primes
                    (par-primes-in-list
                         (drop p potential-primes)
                         (into known-primes new-primes))))))

Christophe's observation:

I use the property that no number in the interval [n .. 2*n-1] is divisible by n (except n itself) so it's safe to process the whole interval with the current set of known primes. Hence this interval can be divided into m intervals (where m is the number of cores) for parallel processing.

The one issue with Christophe's code it part lazy and part eager, so it's applicability depends on whether you are trying to get to a specific number the fastest or just generate primes as fast as possible. Since the number of primes to try grows as it searches, it lazily generates bigger and bigger lists of primes. I tried to find a good value to ask for where it would perform well, and chose 20390 which is big enough to show a difference and doesn't make Christophe's code do too much extra. Here are the results:

Christophe's: "Elapsed time: 87266.119 msecs"
Mine (with a touch added): "Elapsed time: 108841.354 msecs"
Sequential: "Elapsed time: 151075.292 msecs"

20 Days of Clojure: Day 14

Lou Franco — 2008-03-14T21:37:44-04:00

Ok, I'm finally getting somewhere. To start with, Rich pointed out that I could use recur to make my last prime finder stack safe and faster. Here it is:

  (defn primes-in-list [potential-primes known-primes]
    (let [p (first potential-primes)]
      (if (nil? p) '()
        (if (is-prime? p known-primes)
          (lazy-cat
            (list p)
            (primes-in-list (rest potential-primes) (conj known-primes p)))
          (let [rest-primes (rest potential-primes)]
            (recur rest-primes known-primes))))))

Here's how I parallelized it:

  (def w1 (agent 0))
  (def w2 (agent 0))

  (defn par-primes-in-list [potential-primes known-primes]
    (let [ x 10 x-times-2 (* x 2)]
      (! w1 (fn[w pp kp] (primes-in-list pp kp)) (take x potential-primes) known-primes)
      (! w2 (fn[w pp kp] (primes-in-list pp kp)) (take x (drop x potential-primes)) known-primes)
      (await w1 w2)

      (let [new-primes (concat @w1 @w2) ]
        (if (= (count new-primes) 0)
          (recur (drop x-times-2 potential-primes) known-primes)
          (let [real-primes
            (if (> (first new-primes) x-times-2) new-primes (primes-in-list new-primes []))]
            (lazy-cat real-primes (par-primes-in-list
              (drop x-times-2 potential-primes)
              (into known-primes real-primes))))))))

Basically, I send lists to the agents, wait for them, and then combine their answers. The original runs at about 100% of CPU, calculates 8000 primes in 21.6 seconds and this runs at about 125%, and calculates the primes in 17.1 seconds, or in about 80% of the time.

20 Days of Clojure: Day 13

Lou Franco — 2008-03-13T20:08:56-04:00

My brain still hurts from thinking about parallelizing prime number generation. I think the key is still to find primes two at a time in parallel, meaning that it's only useful if you want a list, which is necessary to find the nth prime anyway.

So, I am starting with a rewrite of prime number generation -- this one is simpler for me to understand and for some reason runs three times faster.

  (defn divisible? [x y]
      (= (rem x y) 0)
  )

  (defn is-prime? [p known-primes]
      (every? (fn [kp] (not (divisible? p kp))) known-primes)
  )

  (defn primes-in-list [potential-primes known-primes]
      (let [p (first potential-primes)]
          (if (is-prime? p known-primes)
              (lazy-cons
                  p
                  (primes-in-list (rest potential-primes) (conj known-primes p)))
              (primes-in-list (rest potential-primes) known-primes)
          )
      )
  )

  (def primes (primes-in-list (iterate inc 2) []))
  (time (reduce + (take 1000 primes)))
  (time (reduce + (take 1000 primes)))

It's also lazy. Not sure why it's so much faster. Now that I can provide a list of potentials, I'll be able to run two lists in parallel, and then I have to reduce them in the main thread. When I get the lists back, I need to make sure that they aren't divisible by each other (since they won't be in each others list of known primes).

20 Days of Clojure: Day 12

Lou Franco — 2008-03-12T19:37:46-04:00

So, Rich thinks I'm on a fool's errand, but I'm going to keep going with this.

I'm sorry to say I can't think of a less promising candidate for parallelization :( The 3 things you would want in something to parallelize are  independent data/work, moderate-to-course-grained work units and/or  complex coordination logic that would be simplified with threads.  (Note - wanting it to run faster is not enough :) Here you have none of those - the data/work is interdependent (more  later), the work is very fine grained (seq walk and a remainder test),  and the logic was pretty elegant.

So, in order to have something to parallelize, I need to get one of those things. Since I am trying to make a list of primes, I'll generate them two at a time. Each check is independent. There's probably better ways to do it, but to keep my life simple, I'll just generate pairs of primes lazily, but in parallel (each member of the pair in a separate thread).

Hmmm. I'm too tired to figure this out right now. So, for some content today, I can explain a mistake that Rich pointed out in my original prime sieve.

I defined primes as a function that returns a lazy seq, but a key benefit of lazy seqs is that they cache values -- so by creating a new lazy seq for each call to primes, I lose caching. The right way is to define primes as a seq this way:

  (def primes (sieve (iterate inc 2)))

Then, when you do this:

  (time (reduce + (take 1000 primes)))
  (time (reduce + (take 1000 primes)))

You get this:

  "Elapsed time: 1080.253 msecs"
  "Elapsed time: 5.36 msecs"

20 Days of Clojure: Day 11

Lou Franco — 2008-03-11T17:40:52-04:00

Ok, so now we've covered some basic clojure as lisp with immutable datastructures and built-in support for streaming (lazy) interfaces. I was playing around with Erlang right before I found clojure because I was looking for a language with direct support for concurrency, the lispiness and the JVM hosting were the deciding factor for me.

I'm just getting started with clojure's concurrency support, so you'll have to bear with me as I might struggle a bit. Yesterday, I showed you a lazy prime number sieve built using the seq interface. Today, I'm going to try to parallelize it. Since I don't really know what I'm doing, I'll take you through my thinking.

There are three ways in clojure to deal with memory in threads: dynamic vars, software transactional memory, and agents. I get dynamic vars and agents because I use things like them all the time. Dynamic vars are basically thread-local storage and agents are a kind of ActiveObject [pdf]. Thread local storage isn't going to work for me since I need to gather answers from my worker threads, not have each work independently (there may be dynamic vars in the solution, but it's not the way you communicate between threads).

ActiveObject is a pattern that serializes access to an object so that one thing happens to it at a time. Individual calls are put in a queue which is serviced by a thread assigned to the object, and that thread is the only one to touch the object. In clojure, there's actually a thread pool that services all agents. Now, that I think about it -- maybe this will work. I kind of wanted to play with STM's, but I think I understand this better and I feel I can still make it streamable.

Here's my naive idea. I'm going to make an agent per prime, then ask them all if a number is divisible by them, then wait for them all and if it is, make a new agent and go on. That seems really dumb unless agents are super-cheap. Let's see.

I'll start with something simple:

(defn divisible? [x y] (= (rem x y) 0))

I'll define my agents to be maps that look like this at the start {:prime 2} and then define this:

  (defn check-divisible [agent x]
    (merge agent {:last x :notdivisible (not (divisible? x (:prime agent)))}))

Here's a quick Repl session so far (no real agents yet)

  user=> (check-divisible {:prime 2} 3)
  {:last 3, :prime 2, :notdivisible true}
  user=> (check-divisible {:prime 2} 4)
  {:last 4, :prime 2, :notdivisible false}
  user=> (check-divisible {:prime 2} 5)
  {:last 5, :prime 2, :notdivisible true}

Ok, now I'll start. Here's something weird -- I just did a CPU snapshot running the sequential prime sieve from yesterday and it looks like this:

I don't know what to make of that -- I know purely functional programs are easy to parallelize, but this sieve seems particularly serial in implementation. Anyway, I'll press on:

Here's my agent-sieve

  (defn agent-sieve [a ints]
    (let [p (next-prime a ints)]
      (lazy-cons p (agent-sieve (cons {:prime p} a) (iterate inc (inc p))))))

I keep stuffing a new agent list into the rest of a lazy-cons -- just need to write next-prime:

  (defn next-prime [agents ints]
    (let [x (first ints)]
      (check-divisible-by-agents (take-nth 2 agents) x)
      (check-divisible-by-agents (take-nth 2 (drop 1 agents)) x)
      (apply await agents)
      (if (every? (fn [a] (or (not= (:last @a) x) (:notdivisible @a))) agents)
        x
        (next-prime agents (rest ints)))))

Ok, so I'm starting to see problems -- for one, I really want to short-circuit if an agent is divisible, but I also want to split up the work. So I decide to make a check-divisible-by-agents that will chain along each agent and stop when it finds an agent that divides into the prime. I start two chains each with half of the agents and wait -- this will mean that if one is done, the other won't know and it will keep going (which is bad). I press on, here's the chain

  (defn check-divisible-by-agents [agents x]
    (if (not= (count agents) 0)
      (! (first agents)
        (fn [a y rest-agents]
          (let [newstate (check-divisible a y)]
            (if (:notdivisible newstate)
              (check-divisible-by-agents rest-agents y))
            newstate))
        x (rest agents))))

So, I run a function on the first agent in the list (with !) that sets the new state to whether it was divisible. If it's not divisible then I recursively call the check on the rest of the agents.

I'm not getting primes yet -- but it's close. I don't think await works like I think it does -- even though I am chaining ! inside of !, I think that doesn't count for await. Here's is next-prime rewritten.

  (defn next-prime [agents ints]
    (let [x (first ints)]
      (check-divisible-by-agents (take-nth 2 agents) x)
      (check-divisible-by-agents (take-nth 2 (drop 1 agents)) x)
      (if (every?
          (fn [a]
            (await a)
            (or (not= (:last @a) x)
              (:notdivisible @a)))
        agents)
          x
          (next-prime agents (rest ints)))))

Ok, I get primes now

To tell the truth, I can't believe I got this working at all, but running (time) on the serial and parallel versions give me this:

sequential: "Elapsed time: 9.517 msecs"
parallel: "Elapsed time: 251.025 msecs"

So, obviously, this is not a good way to do this. I don't know yet if it's because I'm using too many agents or the extra work I'm doing because I'm not short-circuiting. Anyway, that's enough for now.

Update: After a night's rest, I think I know what's going on with the other thread in the sequential version. My guess is the GC. The only other alternative I can come up with is that under the hoods clojure parallelizes some things I'm using. I started a thread on parallelizing prime sieves on the clojure google group.

(clojure day (March 20) in Northampton, MA is coming soon)

20 Days of Clojure: Day 10

Lou Franco — 2008-03-10T20:51:29-04:00

Woo hoo -- half way through. Well, not yet, but at the end of this entry.

Yesterday, we saw sequences. Now, here's a way to use them: the Prime Sieve of Eratosthenes. Here it is in Scheme from the SICP lectures:

(define (sieve s)
  (cons-stream (head s)
   (sieve (filter
    (lambda (x)
     (not
      (divisible? x (head s))))
    (tail s)))))

(define primes
  (sieve (integers-from 2)))

And here is a complete clojure program (all undefined functions are defined as part of the seq interface)

(defn sieve [s]
  (lazy-cons (first s)
    (sieve (filter
     (fn [x]
      (not
       (= (rem x (first s)) 0)))
     (rest s)))))

(defn primes []
  (sieve (iterate inc 2)))

(prn (take 10 (primes)))

Here's how it works. The function sieve is a sequence and takes a sequence (in this case we pass in all numbers greater than or equal to two).

Working from the inside, this filter:

    (filter
     (fn [x]
      (not
       (= (rem x (first s)) 0)))
     (rest s))

Filters the rest of the passed in stream based on whether the items are divisible by the first, so only the ones not divisible are passed. So for (iterate inc 2) we get a stream of all numbers not divsible by two.

The first of that stream is 3, and it is passed into sieve, so the resulting stream is all those not divisible by two (already filtered) or three. Once the stream is returned (lazily), the heads are lazy-cons'd on (that means that you get a stream with a given first and the rest set to the given stream). The first head was 2, the second was 3, the next one is 5 (the first number not divisible by 2 and 3), and so on.

(clojure day (March 20) in Northampton, MA is coming soon)

Update: It may look like sieve is an infinitely recursive function, but it's not. lazy-cons is a macro that doesn't evaluate its argument until rest is called on its return value. I haven't looked at the implementation, but I imagine its implemented similarly to how cons-stream in the SICP lecture is.

Update: This prime sieve has been corrected here.

20 Days of Clojure: Day 9

Lou Franco — 2008-03-09T12:16:59-04:00

News from Rich on a optimization he found for vectors:

I re-read Okasaki's paper yesterday and woke up today with an idea for speeding up the Clojure vector conj and pop. They are now 2-5x fasterthan they were. The trick is to keep the tail of the vector out of the tree until it is a full (32-element) node. Note that this greatlyspeeds up vector construction, which typically consists of repeatedly conj'ing onto [].

Today, I'm going to look at sequences. I think it's helpful to go back to the SICP -- this time, the lecture on stream machines:

This is just a small part of the whole SICP stream machine lecture (and Streams Part II).

Sequences in clojure are lazy like stream as described in the video, and if you watch the whole lecture, you will see a rich stream library built up that is present in clojure.

This means that many operations can be incredibly efficient, but seem like they are operating on gigantic data structures at once (since my day job is image processing, this is especially intriguing to me -- maybe more on that later). To really see this, you need to play with infinite sequences. In clojure, it's easy to create an infinite sequence with cycle -- (cycle [1 2 3]) returns a sequence which continuously yields 1, 2, 3, 1, 2, 3, etc. If you use sequence operations on it, you can get a feel for how laziness works because it's obvious that these functions could not possibly be consuming the whole sequence. Here's a clojure Repl script.

user=> (def cyc123 (cycle [1 2 3]))
#
user=> (every? (fn [x] (= 1 x)) cyc123)
false
user=> (take 5 cyc123)
(1 2 3 1 2)
user=> (take 5 (filter (fn [x] (= x 1)) cyc123))
(1 1 1 1 1)
user=> (take 5 (drop 1 cyc123))
(2 3 1 2 3)
user=> (take 5 (interleave cyc123 cyc123))
(1 1 2 2 3)
user=> (take 5 (take-nth 3 cyc123))
(1 1 1 1 1)
user=> (take 10 (for [x cyc123] (< x 3) x))
(1 2 1 2 1 2 1 2 1 2)
user=> (mapcat (fn [x y] (list x y)) cyc123 '(10 11 12))
(1 10 2 11 3 12)

More on this tomorrow.

20 Days of Clojure: Day 8

Lou Franco — 2008-03-08T11:14:55-05:00

Rich Hickey comments on yesterday's post on the clojure google group:

An interesting point of comparison with purely-functional solutions such as that which Okasaki writes about is that the Clojure vector can be log₃₂N specifically because I can use mutation - when cloning, i.e. the tree is log₃₂N deep and a node can be copied in O(1) time due to arraycopy + indexed array set. And yes, the vectors are optimized for lookup - in practice, log₃₂N blows logN out of the water.

Exactly -- PersistentVector's internals are not purely functional. New objects (during construction only) make an array copy of Object arrays from existing vectors, then make changes to the copied array before returning. As Rich points out, this makes some operations O(1). And for comparison, a clojure vector of 32k elements is only 2 levels deep, but if implemented with a log₂N structure would be 15 deep (with average access in maybe 7 operations). Since access by index is a very common operation, you can see that this would make a big difference.

Today, I want to look at PersistentHashMap.java which is the implementation of clojure hash-maps. There are actually four implementations of the map interface (hash-map, sorted-map, struct-map, and array-map), but hash-map is what the reader will give you for when you build up a map with {}.

A comment on the top of the source file says that it's a persistent version of Phil Bagwell's Hash Array Mapped Trie. According to that paper, it looks like the bits of hashes of the key are used to traverse a trie. Let's see how it's done in clojure.

The first thing I like to look at is how lookup is done, that gives me a sense of the structure. So, to start with, here's entryAt:

public IMapEntry entryAt(Object key){
return root.find(RT.hash(key), key);
}

Simple enough -- it uses the root node to start the traversal. The root node is an INode, and clojure uses polymorphism to get different behavior for different nodes. There are five types of nodes, EmptyNode, LeafNode, HashCollisionNode, BitmapIndexedNode, and FullNode. An empty hash-map has its root set to an instance of EmptyNode, which is an implementation of the Null Object design pattern, and find() on it returns null (by the way, the entire structure is a Composite design pattern, and by using each node type to handle part of traversal, I guess he's also using a Chain of Responsibility -- I mention this because there are many OO design patterns used in the Java implementation of clojure -- some by name, so if you are going to look at the code, you'll want to be familiar with these patterns).

If I assoc() onto a hash-map, root.assoc() is called which is supposed to return a new root. EmptyNode.assoc() simply creates a LeafNode with the hash, key and value, so a map with one key has its root set to a LeafNode.

If I call find() on a LeafNode, that's pretty simple too. If this is the node I am looking for, it returns this, otherwise null.

If I assoc onto a LeafNode, here is what can happen:

If the key and value match the LeafNode, then it is just returned -- nothing is created since you haven't changed anything.
If just the key is the same as this node, a new LeafNode is created with the same key and the new value and returned (remember, this is a persistent data-structure, so we do not change the LeafNode, we create new nodes to build up a new hash-map).
If the hash is the same, but the key is different, then a new HashCollisionNode is created that holds onto this LeafNode and a new one that has the same key and the new value. HashCollisionNodes simply hold onto an array of LeafNodes whose keys have the same hash.
If the new key doesn't have the same hash (the common case), then a BitmapIndexedNode will be created. At first it contains just the old LeafNode in its array of nodes, and then assoc() is called on it with the new key and value.

Case #4 is where we start to build up the trie. The hash of the key is going to be used to traverse the trie later, and a BitmapIndexedNode knows if it contains the LeafNode you are looking for. A BitmapIndexedNode has a bitmap of 32 bits. The bit corresponding to the bottom five bits of the hash of the LeafNode is turned on to indicate that the LeafNode is to be found in this branch (it can have 32 branches). For example if the bottom five bits of the key hash are 01100, then the 12th bit of the bitmap is turned on, and the node is put at this branch.

If the bitmap bit for the new leaf node is off, then a new BitmapIndexedNode is created with a new bitmap with the new bit turned on, and the node is put in its array which will be one bigger than the one we started with. If we kept doing this and hit each bit, the 32nd assoc would result in a FullNode -- which is a simple case of a BitmapIndexedNode, except it knows that if it had a bitmap, all bits would be set. Again, remember, we create a new node because we never change nodes in a persistent data structure.

If the new leaf node has a bottom five bits that match a node already in this BitmapIndexedNode (the corresponding bit is already turned on), then we assoc() this key/value onto the node with the same five bits.

I haven't mentioned this yet, but assoc takes a shift argument -- so far, that has always been 0 and that meant to use the bottom five bits (use a 0 shift) of the hash in any BitmapIndexedNodes. On this assoc call, we're now going to pass in (shift+5) so that the next 5 bits will be used on any BitmapIndexedNodes created under here. Later when we traverse, we'll use the bottom five bits on the first BitmapIndexedNode we see, then the next five bits on the next one and so on. When we've used up all 32 bits of the hash, we should be at our LeafNode or a HashCollisionNode (meaning all 32 bits matched).

So here's find():

If the node is an EmptyNode, it returns null.
If the node is a LeafNode, it returns itself if the key matches, otherwise null.
If the node is a HashCollisionNode, it calls find() on each LeafNode it holds, if any return themselves, then that is returned, otherwise null.
If the node is a BitmapIndexedNode, it first checks to see if it has a node where the five bits being used matches and then calls find() on it, if it doesn't, it returns null.
If the node is a FullNode, it acts like a BitmapIndexedNode with all bits of the bitmap set, so it just calls find() on the node corresponding the five bits in the hash at its level.

The data structure is persistent because it never changes constructed nodes. During an assoc, all non-traversed paths will be shared, and only the path that changed will be new.

I should also mention that FullNode is purely an optimization -- a BitmapIndexedNode could be used in its place, but it saves some branch calls to have them sprinkled throughout the map. Also, using a bitmap in BitmapIndexedNode is also an optimization. An INode array of 32 elements could have always been constructed with null signifying that there was no nodes in that branch, but that would be very wasteful in space, and so by using the bitmap, BitmapIndexedNodes have an INode array of the exact length they need.

20 Days of Clojure: Day 7

Lou Franco — 2008-03-07T21:06:51-05:00

Ok, I'm going to try to describe how vectors are implemented in clojure. The implementation is in:

   src/jvm/clojure/lang/PersistentVector.java

if you want to see the code. I'm going to try to go through the important bits here.

The PersistentVector object has three members: two ints named cnt and shift, and an Object[] named root. A simple PersistentVector built from this line:

   (def v1 [1 2 3])

would look like this (using curly braces for root because it is a Java array):

   cnt: 3
   shift: 0
   root: { 1, 2, 3 }

cnt is the count of elements and (count v1) simply returns it and is O(1) -- I'll explain shift later, and root is the object array. When shift is 0, this line:

   (nth v1 1)

Just simply resolves to root[1], and returns 2, and in this simple case is also O(1). If I do this:

   (def v2 (assoc v1 1 4))

which returns a new vector, but with the value at index 1 set to 4, you get another PersistentVector that looks like this:

   cnt: 3
   shift: 0
   root: { 1, 4, 3 }

The 1 and the 3 are shared between the v1 and v3 array. If I do this:

   (conj v1 5)

I'll get yet another PersistentVector that looks like this:

   cnt: 4
   shift: 0
   root: { 1, 2, 3, 5 }

with the 1, 2, and 3 shared with v1 (and the 1 and 3 shared with v2). This is all very simple until, you conj onto a vector of 32 elements. When the root array has 32 elements, then adding one more element (33) returns a new PeristentVector that looks like this (assume the starting array had 1, 2, 3, ... 32)

   cnt: 33
   shift: 5
   root: { {1, 2, 3, ..., 32 }, { 33 } }

Root is a Java Object[] with the 0th element set to the Object[] from the input vector to conj (not a clone, but the actual one), and the next element is an array of just the new value. If I conj onto that, I get a new Vector:

   cnt: 34
   shift: 5
   root: { {1, 2, 3, ..., 32 }, { 33, 34 } }

Now, I can explain shift. When a method that takes an index is called, a bit shift is done on it to determine how many levels deep in the structure we need to go to get to the part of the datastructure that has the element at that index. For instance, here is the main Java for nth(i):

      Object[] arr = root;
      for(int level = shift; level > 0; level -= 5)
         arr = (Object[]) arr[(i >>> level) & 0x01f];
      return arr[i & 0x01f];

so, when i < 32, then i >>> level is 0, and arr will be root[0] (the array of 32 elements). Then we return arr[i & 0x01f] (which is i % 32), to get the ith element in that array.

When i == 32, then (i >>> level) is 1, arr is root[1], and then we return arr[i%32] which is the 0th element. Now, if I do an assoc to set the 0th element to 100, I get this PersistentVector:

   cnt: 34
   shift: 5
   root: { {100, 2, 3, 4, ..., 32 }, { 33, 34 } }

assoc calls a recursive function (doAssoc). First it clones root so that the new root is an array of two elements, each an object array. Then it determines that index 0 is in the 0th branch and does an assoc on that, decrementing the shift by 5 and setting the index to (index % 32). This recursive call clones the array at root[0]. Since shift is now 0, it is at the base case of the recursion, and so it sets root[0][0] to 100. All of the numbers and the entire array at root[1] is shared with the starting vector. Here is the Java code for that (doAssoc is called with the arguments shift, root, the index, and the new value) -- the return is the new root:

   private static Object[] doAssoc(int level, Object[] arr, int i, Object val){
      Object[] ret = arr.clone();
      if(level == 0)
         {
         ret[i & 0x01f] = val;
         }
      else
         {
         int subidx = (i >>> level) & 0x01f;
         ret[subidx] = doAssoc(level - 5, (Object[]) arr[subidx], i, val);         }
      return ret;
   }

By now, you might have realized that we are building up a tree. If I keep conjing on numbers in order starting at 1, eventually I will conj on 65. If so, I get this

   cnt: 65
   shift: 5
   root: { {1, 2, 3, 4, ..., 32 }, { 33, 34, 35, ... 64 }, { 65 } }

Graphically, it looks like this (for 1..32 elements)

+-------------+
|{1,2,3,...32}|
+-------------+

and then for up to 32², it looks like this:

             +-------+
             | {   } |
             +--+-+--+
                | |
       +--------+ +----------+
       |                     |
+------+------+     +--------+-------+
|{1,2,3,...32}|     |{33,34,35,...64}|
+-------------+     +----------------+

And, we get another level at 32²+1 (1025) elements, and another at 32k+1. This explains how index access is O(log₃₂N). Another important point, is that not only is a PersistentVector immutable from an interface perspective, but it's internally immutable, and each individual Object[] that makes up the tree is never changed in a constructed vector. If it needs to change in a new vector, it is cloned and the changed clone is used in the new PersistentVector -- the benefit of this is that the vector never needs to lock. This may seem like a small point, but I believe that this is not a requirement of Persistence -- but this extra work makes vector very suitable for concurrent applications.

This implementation is different from the one described in the Okasaki paper from yesterday. Rich appears to be trading away O(1) consing for better index performance, which was O(log₂N) in that paper.

(clojure day (March 20) in Northampton, MA is coming soon)

Update: Made a few corrections

20 Days of Clojure: Day 6

Lou Franco — 2008-03-06T00:19:51-05:00

So far, I've been working on the "clojure is a dialect of Lisp" part of the description of clojure. One of the things Lisps allow is purely functional programming, but to enforce it you need immutable datatypes. One of the things that bothered me as a C programmer, is that I could never understand how it worked for anything more complicated than a string. Then I saw persistent data-structures. Their existence in clojure is what made me think that purely functional programming could work in real programs.

The clojure page on data structures, but more importantly check out clojure TV on collections.

If you're interested in how these data structures are implemented, you want to read Okasaki. I would start here with his description of a persistent random access list, which I think is one of the more accessible.

Tomorrow, I'm going to try to expand on this by describing how vectors are implemented in clojure.

20 Days of Clojure: Day 5

Lou Franco — 2008-03-05T07:04:31-05:00

This SICP lecture shows some of the power of Lisp by building a symbolic differentiator in a very small amount of code, and is also the first one to introduce the idea that code is data and data is code (watch from the beginning to 29:00 -- at 13:00 he starts talking about using Lisp as the data format):

Here is the first symbolic differentiator he writes (but, in clojure)

    (defn atom? [x]
        (not (instance? clojure.lang.IPersistentCollection x)))

    (defn const? [expr val] (and (atom? expr) (not= expr val)))
    (defn same-var? [expr val] (and (atom? expr) (= expr val)))
    (defn arith? [expr sym] (and (not (atom? expr)) (= (first expr) sym)))
    (defn sum? [expr] (arith? expr '+))
    (defn prod? [expr] (arith? expr '*))

    (defn a1 [expr] (frest expr))
    (defn a2 [expr] (first (rrest expr)))
    (defn m1 [expr] (frest expr))
    (defn m2 [expr] (first (rrest expr)))

    (defn deriv [expr val] (
        cond
            (const? expr val) 0
            (same-var? expr val) 1
            (sum? expr)
                (list '+ (deriv (a1 expr) val) (deriv (a2 expr) val))
            (prod? expr)
                (list '+ (list '* (m1 expr) (deriv (m2 expr) val))
                    (list '* (m2 expr) (deriv (m1 expr) val))
                )
            (:else) nil
        )
    )

The things to note:

Quote syntax, which uses '
first and rest which are car and cdr. Clojure also supports combinations like frest (cadr) and rrest (cddr)
In clojure, extra parens are usually not necessary -- look at how cond is implemented to use variable arguments rather than a single list argument (which would require extra parens)
My first usage of Java interop (using instance? to find out if an expression is atomic or a collection)

20 Days of Clojure: Day 4

Lou Franco — 2008-03-04T13:31:47-05:00

Continuing from yesterday, we made our first function that takes a function and returns a function. Go to 43:00 in this video to see why (you can stop at 54:00):

In this example, we take the first-order functions idea further and define a derivative function (that takes and returns functions) and a newton function (which takes a function and uses fixed-point). Here it is in clojure (you can find fixed-point and other functions in yesterday's post):

    (defn square [x] (* x x))

    (defn deriv [f]
        (let [dx 0.000001]
            (fn [x] (/ (- (f (+ x dx)) (f x)) dx))))

    (defn newton [f guess]
        (let [df (deriv f)]
            (fixed-point (fn [x] (- x (/ (f x) (df x)))) guess)))

    (defn sqrt [x] (newton (fn [y] (- x (square y))) 1))

(clojure day (March 20) in Northampton, MA is coming soon)

20 Days of Clojure: Day 3

Lou Franco — 2008-03-03T12:53:51-05:00

Ok, so first we made a very hard-coded sqrt, then we improved it by adding the concept of fixed points, which made the algorithm easier to see in the code. Here's how we can improve it even further. Go to 32:30 in this video (stop at the break at 43:00):

First we define an average-damp function. This function takes a function as its argument and returns another function which takes a number and returns the average of that number and result of calling the function argument on the number. It's easier to say that in clojure:

    (defn average-damp [f]
        (fn [x] (average x (f x))))

f is a function, average-damp creates a new function that takes x (a number) and that new function averages x and f(x).

Now that I have that, I define sqrt like this (you can find average and fixed-point in yesterday's post):

    (defn sqrt [x] (fixed-point (average-damp (fn [y] (/ x y))) 1))

Tomorrow, we'll use this to implement another way of finding sqrt.

20 Days of Clojure: Day 2

Lou Franco — 2008-03-02T12:04:27-05:00

Yesterday, we built a simple sqrt using Heron of Alexandria's algorithm. The problem with the code is that it had some generically useful ideas, but it was all hard-coded to only work with sqrt. Also, it was not clear what exactly the algorithm was.

In the SICP lecture they fix it by passing functions around since Lisp treats functions as first-class data-types. Here's the fix explained (23:30 in the video, stop at 32:30):

See the video for the Scheme implementation. Here it is in clojure.

    (defn sum ([] 0)
        ([x] x)
        ([x & more] (+ x (apply sum more) )))

    (defn average
        ([] 0)
        ([x] x)
        ([x & more]
            (/ (+ x (apply sum more)) (+ 1 (count more)))))

    (defn abs [x] (if (< x 0) (- x) x))

    (defn close-enough? [u v]
        (let [tolerance 0.00001]
            (< (abs (- u v)) tolerance)))

    (defn fixed-point [f start]
        (loop [old start new (f start)]
            (if (close-enough? old new)
                new
                (recur new (f new)))))

    (defn sqrt [x] (fixed-point (fn [y] (average (/ x y) y)) 1 ))

    (prn (sqrt 9))

A fixed point of a function f is a value y such that f(y) = y. This improved code, which implements the same algorithm as before, uses a generic fixed-point function that takes a function and a guess and tries to find the fixed point by setting the next guess to f(guess). You can see it all in the video. The things in clojure you need to know:

fn [arg1 arg2] is lambda (λ), so where fixed-point takes a function as the first argument, we create one on the fly to be (average (/ x y) y) where x is the argument to sqrt (closed around our function) and y is the argument defined by fn.
Since clojure compiles to Java and uses Java calling conventions, it does not have automatic tail-recursion. Instead, you use the recur special operator which will create a constant space loop back to the last loop or function definition. Loop/recur is explained on the Functional Programming page of the clojure site (at the bottom).

But, even this can be improved, as we'll see tomorrow.

20 Days of Clojure: Day 1

Lou Franco — 2008-03-01T10:11:57-05:00

Ok, this might be hard to keep up, but nothing ventured ...

Rich Hickey is going to be coming to Northampton on March 20th to talk about clojure. In preparation for that, I am going to blog every day in March about clojure -- then, hopefully we can post the results of his talk.

If you don't have clojure yet -- go get it. Start here to learn basic clojure syntax (there is also a download link). Clojure is trivial to get running and you definitely want to play with it a little to understand the rest of this. The quick description of clojure is that it's a dialect of Lisp that runs on the JVM. It has support for immutable and persistent data-structures and a concurrent programming model.

Today I'm going to start with some of the work I did to learn clojure through translating SICP lectures. In the first lecture, the first interesting program starts at 56:20 in this video:

Starting there presumes you know some basic Lisp syntax. This is the program he eventually writes to calculate the square root (in Scheme):

    (define (improve guess x)
        (average guess (/ x guess)))

    (define (good-enough? guess x)
        (< (abs (- (square guess) x)) .001))

    (define (try guess x)
        (if (good-enough? guess x)
            guess
            (try (improve guess x) x)))

    (define (sqrt x) (try 1 x))

I have left out implementations for square and average here. This code and the algorithm is explained in the video. Here it is in clojure (I defined average and square -- you could just use Java, but I am trying to keep it all in clojure for now)

    (defn sum ([] 0)
        ([x] x)
        ([x & more] (+ x (apply sum more) )))

    (defn average
        ([] 0)
        ([x] x)
        ([x & more]
            (/ (+ x (apply sum more)) (+ 1 (count more)))))

    (defn abs [x] (if (< x 0) (- x) x))

    (defn square [x] (* x x))

    (defn improve [guess x]
        (average guess (/ x guess)))

    (defn good-enough? [guess x]
        (< (abs (- (square guess) x)) 0.001))

    (defn try-sqrt [guess x]
        (if (good-enough? guess x)
            guess
            (try-sqrt (improve guess x) x)))

    (defn sqrt [x] (try-sqrt 1 x))

    (prn (sqrt 9))

If you take this and put it in a file named sqrt.clj, you can run it like this (after downloading clojure):

    java -cp clojure.jar clojure.lang.Script sqrt.clj

And it will print out:

    65537/21845

Explanation of the differences between the two examples:

clojure uses defn to define functions and puts arguments in []
sum and average implementations show how clojure can overload on arity (number of arguments)
try is reserved for exception handling, so I renamed the try function to try-sqrt
the answer is in clojure's rational number format, since all of the computation was adding and averaging integers and rationals, the result is a rational number and clojure preserves that.

Tomorrow, I'll improve this so to make it more generic. A lot of what I used in this can is explained in the clojure Functional Programming support page.

Book Review: Collective Intelligence, Part I

Lou Franco — 2008-01-16T18:38:24-05:00

I'm at about the halfway point in Programming Collective Intelligence by Toby Segaran, and even if the second half was blank, I'd be telling everyone to read this book. It's been a while since I've read a technical book this good. It excels because of a combination of three factors:

Useful algorithms
Shown in real-life situations
With real data that you can get yourself

The basic idea of the book is to show different ways of using user-generated datasets to extract information and make decisions. Each chapter tackles a different problem and shows a few ways to approach it. All of the examples are in Python, but even if you don't know Python, you should be fine if you know any structured language. Practically every example either uses some dataset that he tells you how to get, or a public API that is free to access.

This book is the first I've read that relies on the current maturity level of the web. A few years ago, it wouldn't have been possible to provide so many relevant examples. In a few years, the API's might be out-of-date -- but don't let that stop you -- the algorithms are timeless.

The choice of Python is a good one for a few reasons. Firstly, it's pretty clear to read even if you don't know it. Secondly, list-comprehensions are a pretty good way to think about data crunching (which these algorithms do a lot), and, finally, there are many freely available Python libraries to deal with the public APIs he uses.

To give you an idea of Toby's approach, I'm going to run through my favorite chapter so far, the one on Optimization. The quick description of the problem domain should be familiar to every programmer -- solve NP complete problems like the traveling salesperson. Basically, Optimization techniques can be used in situations where (1) you can't analytically solve the problem (2) the solution set is too large to exhaustively search (3) you can express the cost of a possible solution such that better solutions have a lower cost than worse ones and (4) solutions that are "near" each other have similar values for cost.

Toby's first example is how a family can choose flights so that they arrive at the same airport from all over the country at nearly the same time and depart for home a few days later at nearly the same time. The cost function is a combination of the actual costs of the tickets and the difference in time between the earliest and latest flights. He goes on to provide a dataset of flights, describes setting up optimization problems in general, and shows the python code to implement the cost function.

Then he introduces a few optimization algorithms, implements them, and discusses their relative strengths and weaknesses. The next section applies these techniques to real data on Kayak.com using the standard Python DOM parser, minidom. The final section shows how to apply these techniques to other problems like pairing students in a dorm and network visualization. Other chapters get a similarly exhaustive treatment.

I really hope that this is the future of technical books.

Django, WebFaction, RapidWeaver and Christmas

Lou Franco — 2006-12-25T16:38:07-05:00

Atalasoft closes for the week between Christmas and New Year's, so I have time to catch up on some things I've been wanting to do -- namely move this site to WebFaction. I chose WebFaction once I decided that I wanted to learn Django (after reading this great comparison of Django and Rails). Following along the forums, there were more than a few good mentions of WebFaction as a good Django host.

A couple of weeks into Django and a couple of days into WebFaction, so far, I'm pretty happy. It's Christmas today, so even though I had some questions for WebFaction, I really didn't expect them to get back to me -- so I was pretty happy that they do -- I guess I should expect 365 24x7 from an ISP, but I hadn't been getting it before.

In addition to new hosting and a new webapp framework -- I've also settled on RapidWeaver for content management. I moved to Mac for home a couple of years ago, and I still hadn't found a good desktop CMS that was as good as CityDesk on the PC. RapidWeaver has beautiful themes built in, and a lot of the features that I've gotten used to. The built-in blog is nice (would be nicer with an import from RSS), but it didn't take too long to transfer everything over.

All of this work is to get this site going again.

API UI

Lou Franco — 2006-03-15T12:44:16-05:00

I saw this article on Reddit that applies principles from human-factors design to API design. I think that's a great idea, but this suggestion sounds weird to me.

Now we could use progressive disclosure to help reduce the complexity of that JButton class: put the expert stuff in an object returned by a getExpertKnobs() method, and the graphics subsystem hooks in an object returned by a getIntegrationHooks() method, and you would be left with a button API that had just a handful of methods—the basic methods we all need.

This brings the number of methods in a button from about 100 to about 10, which sounds great until you try to subclass. Overriding a method that's in the object returned by the getExpertKnobs method would be a little tricky. It's likely that you don't even know the real class of that object because those methods are likely to return references to interfaces, plus the component code is in charge of constructing the objects, so there would have to be a way plug into that too, so that your subclass is constructed. I agree that the interface would be much simpler for most users, but it is much more complex for subclassers.

There's also the issue of what would be in the expertKnobs object. Some of the methods in JButton that would go in there actually come from JComponent or Component. In fact, as we progressively go down the class hierarchy, we are adding methods into the class that might belong together, which means that there is probably an expertKnobs class for each component class. But, the getExpertKnobs() method is probably introduced early in the hierarchy -- what does it return? Will I have to downcast to get to the button's expert knobs?

I don't have a solution -- but these issues probably drove the design of JButton more than the idea that they just sloppily added a 100 methods without giving a thought to the complexity of the class.

But the point is taken, and I like the idea of using hard won knowledge from other domains in software design.

Another recent blog on this topic was Steve Hawley's Principle of Least Astonishment essay -- the last topic, on layering APIs, has his approach to this problem.

UML Cheatsheet

Lou Franco — 2006-02-25T12:39:40-05:00

Here is a download for a UML Cheatsheet I made for my UML presentation this week. It's released under this creative commons license. When I'm done with the slides, I will post them here as well.

Update: Here are the slides.

WinFX Generation

Lou Franco — 2006-02-18T12:36:37-05:00

Been looking around WinFX and specifically XAML recently. Not sure I think writing XML is a great way to write a GUI (but billions of web pages clearly prove me wrong). My first impression is that one of the benefits is going to be automated creation of XAML -- not from tools, but from our own programs.

It's been 10 years since HTML was a household word and I still feel like it's easier to hand write it than use a WYSIWYG tool. I also never really liked using dialog editors. Even good ones like Delphi have too many limitations, especially when it comes to layout. So, I'm not expecting much from the tool support for XAML. But since it's just a text format, it should be pretty easy to generate from a simpler format (or from models). I'm looking forward to playing around with that idea.

Presentation Links

Lou Franco — 2006-02-11T12:32:31-05:00

Here are some links I've collected on giving presentations. I have been revisiting them lately in preparation for my talk next month.

Kathy Sierra, in her Creating Passionate Users blog, thinks that we shouldn't have slides. She also has this crash course on learning theory (with pictures and a workbook).
Seth Godin has a free-book called Really Bad PowerPoint. First tip: "No more than six words on a slide. EVER"
Guy Kawasaki has the 10/20/30 rule of PowerPoint.
The Beyond Bullets blog has great tips.
The Zen master of presentations, Steve Jobs, vs. Bill Gates.

Apple XCode CPlusTest Port

Lou Franco — 2006-02-09T12:04:35-05:00

C++ (and Cocoa) unit testing is built into XCode on OSX. I am writing an application right now that I plan to port to Windows. When I wanted to see how portable my code was, I decided to get all of the unit tests passed on Windows.

Unfortunately, XCode does not use CppUnit, which is already portable. Here is some code I wrote that you can use to run your XCode generated unit tests on Windows.

UML Primer at WMass Dot Net Users Group

Lou Franco — 2006-02-09T12:01:16-05:00

I'll be giving the next presentation at the Western Mass Dot Net Users Group on the basics of UML. Details:

Presentation:
Get an introduction to the Unified Modeling Language. Learn the basics of Class, Sequence, and Use Case diagrams. The emphasis will be on UML sketching, but UML code generation will be discussed.

This session will be platform independent. Please bring an open mind, your ideas, and experience to share with the group. Software developers, project managers, students and anyone interested in software development on any platform are encouraged to attend.

Meeting Details:
Tuesday, February 7 at 6:00 PM
Fazzi Associates, Inc.
243 King St. Suite 236, Northampton

Servware

Lou Franco — 2006-02-06T11:58:36-05:00

Despite his rants against C++, Stevey's Drunken Blog Rants, is one of my favorite new blogs. Actually, since it hasn't been updated in about nine months, and was written mostly in 2004, it's not really new. It was rediscovered by social bookmarking sites (I found it on reddit.com). I was reminded of it while thinking about resuscitating this site.

There are a lot of great articles, but this one about server-side service software is one of my favorites (given that I've spent a lot of time thinking about the subject). Another great article on this topic is Paul Graham's The Other Road Ahead.

Western Mass .Net Users Group

Lou Franco — 2006-02-06T11:54:20-05:00

The old .NET SAPs is rechristened under the name: Western Mass .NET Users Group (and a spiffy new website). Tomorrow's meeting topic is XML.

Speaking of XML, I saw this recently: Don't Invent XML Languages by Tim Bray. In addition to pointing to this great list of XML Languages, he introduces "The Big Five" XML Languages and where they should be used.

Suppose you’ve got an application where a markup language would be handy, and you’re wisely resisting the temptation to build your own. What are you going to do, then?

The smartest thing to do would be to find a way to use one of the perfectly good markup languages that have been designed and debugged and have validators and authoring software and parsers and generators and all that other good stuff. Here’s a radical idea: don’t even think of making your own language until you’re sure that you can’t do the job using one of the Big Five: XHTML, DocBook, ODF, UBL, and Atom.

The Chameleon Developer

Lou Franco — 2004-03-14T11:50:12-05:00

I was talking to a developer friend today about developers who don't get the importance of communication. Look through any want-ads for developers and right after the alphabet soup of technologies you have to know is the same requirement—must have “excellent written and verbal communication skills”. You'd be hard pressed to find a programmer job without those requirements.

But, expressing yourself clearly is only half of communication. The more important aspect is listening to and understanding other people. This is what separates good and great developers. And the most important skill in understanding is empathy—not only understanding someone, but being able to be them.

To be a successful developer, you have to learn more than just the software aspect of your business. Developers who understand this are more valuable and valued more. If you want to do this effectively, you have to think like a saleperson, a marketer, a CEO, your customers, the press, etc. You have to think like them because they aren't all going to try to think like you (but if they're going to try, by all means, help them).

Another part of empathy is thinking about how they will interpret your words. For instance, avoid rhetorical tics like “think about it” and “you're not seeing the big picture”. Not only are they annoying, but it puts people on the defensive.

If you understand—not only what someone is saying, but why—you'll have a much better chance of expressing yourself. So, the next time you have a communication gap with a non-developer, stop talking and ask yourself how much you even understand about them.

Software Addins as a Business

Lou Franco — 2004-03-12T11:46:11-05:00

I've written a couple of add-ins for CityDesk, and of course, it's tempting to think about trying to make some money off of them. I'm already a partner in another add-ins business (Spheresoft, which makes MS Office addins -- check out Highlighter if you view realtime data in Excel), so I know a little about what goes into the development, support, marketing, sales, licensing, etc. of doing this kind of thing.

Add-ins are a great business for small companies, because they are much easier to implement than a full-blown product and the software you are adding on to will help with marketing. Another add-in that Spheresoft will be publishing soon, changes the nature of spreadsheet modeling, but we'd never want to compete with Excel on thousands of other tiny features, so it makes more sense for this to be an add-in.

But, with any business, you have to decide how much effort you are willing to put in and what you are expecting back. The market of people who have Office is huge and I have a reasonable expectation of making a good enough return on Office add-ins to warrant the time spent writing and supporting them.

I'm not sure I can do the same for add-ins for Fogcreek products. I'm glad that people find them useful, and I do them because I need them as well. I would never make enough money to recoup support costs I would have to undertake if I charged for them.

So, they will remain free and pretty much unsupported for now.

Code Generation from Requirements

Lou Franco — 2004-03-12T11:24:42-05:00

This article on Domain Specific Languages from Martin Fowler got me thinking on code generation in general. A project I'm working on right now has many opportunities for generating code—not from a domain specific language, but from the requirements documentation and other source information.

I am working on a project now with a very detailed requirements document. For instance, practically all of the strings shown to the user are in there. Since the project uses string resources, there's no reason to manually copy and paste—the string resource is easily generated from the document.

The project also has a lot of images. And images are associated with each other (some images are the small version of another, or meant to be the same object but from another angle). Luckily, the artist used a directory and naming convention that makes these associations easy to figure out, and generate an XML resource descriptor and object building code from.

Tying together this information (each image has a string descriptor) is also easy, given the detail in the document and the naming convention.

Actual logical code generation is almost possible as well, as the project is a kind of complicated finite state machine. In this case the document doesn't go far enough to be a source. A Domain Specific Language might help here, and leaves me with source that might be able to be maintained more easily that a Java representation.

This brings up a good lesson for those writing requirements, even if you don't know how to program. If you follow conventions closely, it might be a good starting point for generating code.

Furl

Lou Franco — 2004-03-09T11:22:14-05:00

I've been playing around with the Furl Beta. It's looking pretty good. Essentially, it's a personal online cache of web pages that you can view, search, and share. Better than bookmarking because you can search the contents of it later, and it's accessible from any browser. You can even provide RSS feeds to your Furl content. Collaborative research on the internet just got a whole lot easier.

One drawback, there does not appear to be any way to make the cache private, so I can't possibly use it, because there is no way I want a public database of every site I find interesting or useful. I definitely would share some of it with some people, but totally public is not an option.

Perhaps there will be a pay service for a private space.

Update: My bad. Just mark the item private when you Furl it. Duh. Didn't see that on the Furl dialog and still don't see how to make a public item private (except to change its topic to personal, but then I lose the utility of topics). Anyway, now the biggest drawback is that the first dialog comes up at the wrong size and there is no scrollbar, so you can't see the buttons.