Mark Probert on the Ruby-Talk mailing lists asks: “I am not sure how to create a set of Rake rules to do the following. Can anyone prove assistance?”

I had planned for the next Rake tutorial to go into using prepackaged task libraries, but Mark’s question highlights an interesting problem (and resolution) with rules. I posted an answer to Mark on the list, but why waste a perfectly good explaination when I can recycle it here.

The Problem

Mark has two separate source directories (he calls them src_a and src_b, but I suspect they are more creatively named in real life. Both directories contain .c files. However, the kicker is that all the object files are to be placed in a single directory (named obj) no matter which source directory contained the original C code.

The rule we introduced in the last tutorial isn’t powerful enough to move the .o file into the obj directory. We need to tweek it just a bit.

The .o File Rule

Here is the rule in question.

rule '.o' => '.c' do |t|
  sh "cc -c -o #{t.name} #{t.source}" 
end

To recap, the rule specifies how to create a .o file from a similarly named .c file. But as noted above, the .c are in a different location. We can fix this by giving the rule a general purpose function that transforms the object file name into the correct source file name.

Finding the Source File.

But how do we find the source file? Assuming we have a constant named SRC that contains a list of all our source files, this simple find command will do the trick (assume objfile is the name of the object file):

SRC.find { |s| File.basename(s, '.c') == File.basename(objfile, '.o') }

Wrapping this in a method (named find_source) gives us a nice way to find the source file.

Tweeking the Rule

We can now write the rule like this…

rule '.o' => lambda { |objfile| find_source(objfile) } do |t|
  sh "cc -c -o #{t.name} #{t.source}" 
end

The Whole Rakefile

Just a couple notes about the Rakefile

1. Note that we invoke the OBJDIR task directly in the rule. Because it is a rule, there is no opportunity to list OBJDIR as an explicit dependency. By invoking it directly inside the rule, we will build that directory if it is needed (but only if it is needed).
2. If searching the SRC list has performance problems (because SRC is very long), then an alternative is to create a mapping of object names to source names at the top of the file. Then finding the source name is a simple hash lookup.

Rakefile

require 'rake/clean'

PROG = "foo" 
LIBNAME = PROG
LIBFILE = "lib#{LIBNAME}.a" 

SRC = FileList['**/*.c']

OBJDIR = 'obj'
OBJ = SRC.collect { |fn| File.join(OBJDIR, File.basename(fn).ext('o')) }

CLEAN.include(OBJ, OBJDIR, LIBFILE)
CLOBBER.include(PROG)

task :default => [:build, :run]

task :build => [PROG]

task :run => [PROG] do
  sh "./#{PROG}" 
end

file PROG => [LIBFILE] do
  sh "cc -o #{PROG} -L . -l#{LIBNAME}" 
end

file LIBFILE => OBJ do
  sh "ar cr #{LIBFILE} #{OBJ}" 
  sh "ranlib #{LIBFILE}" 
end

directory OBJDIR

rule '.o' => lambda{ |objfile| find_source(objfile) } do |t|
  Task[OBJDIR].invoke
  sh "cc -c -o #{t.name} #{t.source}" 
end

def find_source(objfile)
  base = File.basename(objfile, '.o')
  SRC.find { |s| File.basename(s, '.c') == base }
end

Alternatives

On possible alternative is to replace the rule with a loop that explicitly creates tasks to compile each .c file. It might look something like this:

SRC.each do |srcfile|
  objfile = File.join(OBJDIR, File.basename(srcfile).ext('o'))
  file objfile => [srcfile, OBJDIR] do
    sh "cc -c -o #{objfile} #{srcfile}" 
  end
end

What I like about this solution is the ability to put the OBJDIR dependency directly in the task definition.

Ruby and celestial mechanics… two great tastes that taste great together.

I saw this mentioned on the Ruby-Talk mailing list. The Kali Code for Dense Stellar Systems by Piet Hut and Jun Makino is a conversation between two astrophysists as they build a computer simulation of the N-Body problem. And they are using Ruby to write the simulation. It is a great introduction to the Ruby language in addition to being a fun problem to solve.

My first job out of college was at Cape Canaveral working on deep space trajectory programs. It has been many years since I flexed my math muscles in celestial mechanices, so it was kinda fun to see that stuff again. I’ve only read through the second chapter, but I’m eagerly looking forward to the rest.

If you are looking to learn Ruby and aren’t frightened by a bit of math (well, perhaps more than a bit), you might enjoy this.

Rake is a tool for controlling builds. In this part of the Rake tutorial, we see how to organize the Rake actions to apply to many similar tasks.

In the RakeTutorialIntroduction, we talked about the basics of specifying dependencies and associating actions to build the files. We ended up with a nice Rakefile that built our simple C program, but with some duplication in the build rules.

But First, Some Extra Rake Targets

But before we get into all that, lets add some convience targets to our Rakefile. First of all, it would be nice to have a default target that is invoked when we don’t give any explicit task names to rake. The default target looks like this:

   task :default => ["hello"]

Until now, the only kind of task we have seen in Rake are file tasks. File tasks are knowledgable about time stamps on files. A file task will not execute its action unless the file it represents doesn’t exist, or is older than any of its prerequisites.

A non-file task (or just plain “task”) does not represent the creation of a file. Since there is no timestamp for comparison, non-file tasks always execute their actions (if they have any). Since the default task does not represent a file named “default”, we use a regular non-file task for this purpose. Non-file tasks just use the task keyword (instead of the file keyword).

Here are a couple of other really useful tasks that I almost always include in a Rakefile.

Remove temporary files created during the build process.

Remove all files generated during the build process.

clean tidies up the directories and removes any files that generated as part of the build process, but are not the final goal of the build process. For example, the .o files used to link up the final executable hello program would fall in this category. After the executable program is built, the .o files are no longer needed and will be removed by saying “rake clean”.

clobber is like clean, but even more aggressive. “rake clobber” will remove all files that are not part of the original package. It should return a project to the “just checked out of CVS” state. So it removes the final executable program as well as the files removed by clean.

In fact, these tasks are so common, Rake comes with a predefined library that implements clean and clobber.

But every project is different, how do we specify which files are to be cleaned and clobbered on a per project basis?

The answer is File lists.

File Lists to the Rescue

A file list is simply a list of file names. Since a lot of what Rake does involves files and lists of those files, a file list has some special features to make manipulating file names rather easy.

Suppose you want a list of all the C files in your project. You could add this to your rake file:

  SRC = FileList['*.c']

This will collect all the files ending in ”.c” in the top level directory of your project. File lists understand glob patterns (i.e. things like "*.c") and will find all the matching files.

By the way, no matter where you invoke it, rake always executes in the directory where the Rakefile is found. This keeps your path names consistent without depending on the current directory the user interactive shell.

The clean and clobber tasks use file lists to manage the files to remove. So if we want to clean up all the .o files in a project we could try …

  CLEAN = FileList['*.o']

(CLEAN is the file list associated with the clean task. I’ll let you guess the name of the file list associated with clobber).

The Rakefile So Far …

With the addtion of a few extra tasks, our Rakefile now looks like this. Notice the require ‘rake/clean’ line used to enable the clean and clobber tasks.

  require 'rake/clean'

  CLEAN.include('*.o')
  CLOBBER.include('hello')

  task :default => ["hello"]

  file 'main.o' => ["main.c", "greet.h"] do
    sh "cc -c -o main.o main.c" 
  end

  file 'greet.o' => ['greet.c'] do
    sh "cc -c -o greet.o greet.c" 
  end

  file "hello" => ["main.o", "greet.o"] do
    sh "cc -o hello main.o greet.o" 
  end

Ok, now its time to address the redundant compile commands.

Dynamically Building Tasks

The command to compile the main.c and greet.c files is identical, except for the name of the files involved. The simpliest and most direct way to address the problem is to define the compile task in a loop. Perhaps something like this …

  SRC = FileList['*.c']
  SRC.each do |fn|
    obj = fn.sub(/\.[^.]*$/, '.o')
    file obj  do
      sh "cc -c -o #{obj} #{fn}" 
    end
  end

Just a couple things to note about the above code.

• The dependencies are not specified. This is a common where we specify the dependents at one place and the actions in another. Rake is smart enough to combine the dependencies with the actions.

• Although the task was named after the .o (which is, after all, what we want to generate), the file list is defined in terms of the .c files. Why?

The simple reason is that file lists search for file names that exist in the file system. We have no guarantee that the .o files even exist at this point (indeed, the will not after invoking the clean task). The .c are source and will always be there.

Rake Can Automatically Generate Tasks

Defining tasks in a loop is pretty cool, but is really not needed in a number of simple cases. Rake can automatically generate file based tasks according to some simple pattern matching rules.

For example, we can capture the above logic in a single rule … no need to find all the source files and iterate through them.

  rule '.o' => '.c' do |t|
    sh "cc -c -o #{t.name} #{t.source}" 
  end

The above rule says that if you want to generate a file ending in .o, then you if you have a file with the same base name, but ending in .c, then you can generate the .o from the .c.

t.name is the name of the task, and in file based tasks will be the name of the file we are trying to generate. t.source is the name of the source file, i.e. the one that matches the second have of the rule pattern. t.source is only valid in the body of a rule.

Rules are actually much more flexible than you are led to believe here. But that’s an advanced topic that we will save for another day.

Final Rakefile

Here is our final resule. Notice how we use the SRC and OBJ file lists to manage our lists of scource files and object files.

  require 'rake/clean'

  CLEAN.include('*.o')
  CLOBBER.include('hello')

  task :default => ["hello"]

  SRC = FileList['*.c']
  OBJ = SRC.ext('o')

  rule '.o' => '.c' do |t|
    sh "cc -c -o #{t.name} #{t.source}" 
  end

  file "hello" => OBJ do
    sh "cc -o hello #{OBJ}" 
  end

  # File dependencies go here ...
  file 'main.o' => ['main.c', 'greet.h']
  file 'greet.o' => ['greet.c']

Up Next

In our next tutorial, we will look at using Rake to handle some tasks other than compiling C code.

Received via EMail:

I have just started using the excellent Rake tool (thanks, Jim!) and I am at a bit of a loss on how to proceed. I am attempting to create unit test for some C++ code I am creating, […]

To adderess this lack, I’m going to post several Rake tutorial articles that will take you through some of the basics. Eventually, I’ll organize the articles into a document somewhere.

Here’s the first one!

The Problem

We will start with a very simple build problem, the type of problem that make (and now rake) were desiged to deal with.

Suppose I am a C programmer and I have a simple C program consisting of the following files.

main.c

  #include "greet.h"
  int main() {
      greet ("World");
      return 0;
  }

greet.h

  extern void greet(const char * who);

greet.c

  #include <stdio.h>
  void greet (const char * who) {
      printf ("Hello, %s\n", who);
  }

(Yes, it really is the old standard "Hello, World" program. I did say we were starting with the basics!)

To compile and run this collection of files, a simple shell script like the following is adequate.

build.sh

  cc -c -o main.o main.c
  cc -c -o greet.o greet.c
  cc -o hello main.o greet.o

For those not familiar with compiling C code, the cc command is the C compiler. It generates an output file (specified by the -o flag) from the source files listed on the command line.

Running it gives us the following results …

  $ build.sh
  $ ./hello
  Hello, World

Building C Programs

Compiling C programs is really a two step process. First you compile all the source code file into object files. Then you take all the object files and link them together to make the executable.

The following figure illustrates the progression from source files to object files to executable program.

Our program is so small that there is little benefit in doing more than the three line build script above. However, as projects grow, there are more and more source files and object files to manage. Recompiling everything for a simple one line change in a single source file gets old quickly. It is much more efficient to just recompile the few files that change and then relink.

But how do we know what to recompile? Keeping track of that would be quite error prone if we tried to do that by hand. Here is where Rake become useful.

File Dependencies

First, lets take a look at when files need to be recompiled. Consider the main.o. Obviously if the main.c file changes, then we need to rebuild main.o. But are the other files that can trigger a recompile of main.o?

Actually, yes. Looking at the source of main.c, we see that it includes the header file greet.h. That means any changes in greet.h could possibly effect the main.o file as well.

We say that main.o has a dependency on the files main.c and greet.h. We can capture this dependency in Rake with the following line:

  file "main.o" => ["main.c", "greet.h"]

The rake dependency declaration is just regular Ruby code. We take advantage of the fact that we can construct hash arguments on the fly, and that Ruby doesn’t require parenthesis around the method arguement to create a file task declaration that reads very naturally to the humans reading the rake file. But its still just Ruby code.

Likewise, we can declare the dependencies for creating the "greet.o" file as well.

  file "greet.o" => ["greet.c"]

greet.c does include stdio.h, but since that is a system header file and not subject to change (often), we can leave it out of the dependency list.

Finally we can declare the dependencies for the executable program hello. It just depends on the two object files.

  file "hello" => ["main.o", "greet.o"]

Notice that we only have to declare the direct dependencies of hello. Yes, hello depends on main.o which in turn depends on main.c. But the .c files are not directly used in building hello, so they can safely be omitted from the list.

Building the Files

We have carefully specified how the files are related. Now we need to say what Rake would have to do to build the files when needed.

This part is pretty simple. The three line build script that we started with contains all the commands needed to build the program. We just need to put those actions with the right set of dependencies. Use a Ruby do / end block to capture actions …

The result looks like this:

Rakefile

  file 'main.o' => ["main.c", "greet.h"] do
    sh "cc -c -o main.o main.c"
  end

  file 'greet.o' => ['greet.c'] do
    sh "cc -c -o greet.o greet.c"
  end

  file "hello" => ["main.o", "greet.o"] do
    sh "cc -o hello main.o greet.o"
  end

Trying it out

So, let’s see if it works!

  $ rake hello
  (in /home/jim/pgm/rake/intro)
  cc -c -o main.o main.c
  cc -c -o greet.o greet.c
  cc -o hello main.o greet.o

The command line rake hello instructs rake to look through its list of tasks and find one called "hello". It then checks hello’s dependencies and builds them if required. Finally, when everything is ready it builds hello by executing the C compiler command.

Rake dutifully reports what it is doing as it goes along. We can see that each compiler invocation is done in the correct order, building the main program at the end. So, does the program work? Let’s find out.

  $ ./hello
  Hello, World

Success!

But what happens when we change a file. Lets change the greet function in greet.c to print "Hi" instead of hello.

  $ xemacs greet.c
  $ rake hello
  (in /home/jim/pgm/rake/intro)
  cc -c -o greet.o greet.c
  cc -o hello main.o greet.o
  $
  $ ./hello
  Hi, World

Notice that it recompiles greet.c making a new greet.o. And then it needs to relink hello with the new greet.o. Then it is done. There is no need to recompile main.c since it never changed.

What do you think will happend if we run Rake again?

  $ rake hello
  (in /home/jim/pgm/rake/intro)
  $

That’s right … nothing. Everything is up to date with its dependencies, so there is no work for Rake to do.

Ok, sure. Rake is a bit of overkill for only two source files and a header. But imagine a large project with hundreds of files and dependencies. All of a sudden, a tool like Rake becomes very attractive.

Summary

What have we learned? Building a Rakefile involves identifying dependencies and the actions required to create the target files. Then declaring the dependencies and actions are as simple as writing them down in standard Ruby code. Rake then handles the details of building

What’s Up Next

We notice that even our small example has a bit of duplication in it. We have specify how to compile both C file separately, even though the only difference is the files that are used. The next installment will look at fixing that problem as well as introduce non-file based tasks, rules and file lists.

Until then … Code Red, Code Ruby.

Changes

• Fixed bug where missing intermediate file dependencies could cause an abort with —trace or —dry-run. (Brian Chandler)
• Recursive rules are now supported (Tilman Sauerbeck).
• Added tar.gz and tar.bz2 support to package task (Tilman Sauerbeck).
• Added warning option for the Test Task (requested by Eric Hodel).
• The jamis rdoc template is only used if it exists.
• Added fix for Ruby 1.8.2 test/unit and rails problem.
• Added contributed rake man file. (Jani Monoses)
• Fixed documentation that was lacking the Rake module name (Tilman Sauerbeck).

What is Rake

Rake is a build tool similar to the make program in many ways. But instead of cryptic make recipes, Rake uses standard Ruby code to declare tasks and dependencies. You have the full power of a modern scripting language built right into your build tool.

Availability

The easiest way to get and install rake is via RubyGems …

  gem install rake    (you may need root/admin privileges)

Otherwise, you can get it from the more traditional places:

Thanks

Lots of people provided input to this release. Thanks to Tilman Sauerbeck for numerous patches, documentation fixes and suggestions. And for also pushing me to get this release out. Also, thanks to Brian Chandler for the finding and fixing —trace/dry-run fix. That was an obscure bug. Also to Eric Hodel for some good suggestions.

— Jim Weirich

Playing with continuations is great fun, but thinking deeply about them does tend to stretch the brain cells. If you always wanted to understand continuations better, but found them a bit hard to comprehend, Mikael Brockman has written a wonderful little introduction on the topic (via Tim Bray).

If you are interested in real world uses of continuations, see SeaSide and Borges.

  $ gem list --local rake

  *** LOCAL GEMS ***

  rake (0.4.15.1, 0.4.15, 0.4.14, 0.4.13, 0.4.12.1, 0.4.12, 0.4.4)
      Ruby based make-like utility.
  $
  $ rake --version
  rake, version 0.4.15.1
  $
  $ rake _0.4.12_ --version
  rake, version 0.4.12

Since rake is installed as a gem, you can run the executable of any installed version. By default, gems will run the latest one. But if you wish to run an older version, gems will accomodate you.

XP and Ruby

As I mentioned in XpCincyAndRuby, the Cincinnati XP Users Group is all set to take a look at Ruby this month.

Mark Windholtz got us started with introductions (we had some newcomers to the group!) and laid out the plan for the evening.

We thought we would start off with something a bit fun, and something that would allow us to get into Ruby fairly quickly. I had adapted the "Paper, Rock, Scissors" RubyQuiz of the week before into something that could be run over DRb. The goal was to run Paper/Rock/Scissors players on our laptops, coordinating with a central game server (using DRb and Rinda).

The result was quite fun. As you can see, at least one of the programs was quite explicit about the state of its own playing ability (click to enlarge the pictures).

Mystery Guest

We did have one guest who was obviously a bit camera shy. Must have something to do with the project he is currently working on. It seems Chris has reached some good milestones recently and had time to come over and check out the Rails crowd. Welcome Chris! And don't worry, we won't tell any of your Java friends (wink, wink).

After the Meeting

Afterwards, we all headed to Unos and enjoyed some appetizers and drinks. This is where all the important discussions happen.

Next month we will be diving into Rails itself. If you are in the Cincinnati area, feel free to stop by and join us. We meet the first Tuesday of every Month.

Cheers!

TagSurf (tagsurf.com)

I’m not sure how to describe this. It is kinda like a web forum with tags applied to each message. What makes it really interesting is that if you use a URL for a tag, then the posting is "about" that web page. So, if you wish to talk about this posting (yes, the one you are reading right now), then create a message with the link onestepback.org/index.cgi/Tech/Web/MoreWebApps.rdoc as a tag. Or, even easier, click the "tagsurf-it" link in the article banner above and be taken immediately to TagSurf with the tags prefilled and ready to go. (I’ve also added a TagSurf link below marked for feedback). Give it a try.

Yes, you will have to sign up for TagSurf, but it is free. It is also very alpha, so be warned. But it is definitely cool. Read about how TagSurf came to be at Russell Beattie's Blog.

Google Maps: (maps.google.com/)

Try it, you’ll like it!

Nuff said.

Ta-Da Lists: (tadalists.com)

I’ve mentioned Ta-Da lists before, but I started using it as a temporary holding area for some ideas for an article on "10 Things Java Programmers Should Know About Ruby". I just dumped suggestions on to the list with minimal editting. Then I made the list public so that folks could see what had already been suggested.

The list wasn’t up two days before it was noticed by some Python guy who picked up on the "Fixes what’s wrong in Python" link (ack … exactly the message I was not trying to send, and it wasn’t even aimed at Python). Then a Smalltalk guy (hi Avi!) blogs about it. I’ve gotten more attention on this article (which I haven’t written yet) than most articles I actually do write.

Part of the problem was that there was no place in the Ta-Da list to explain its purpose. It was just a raw list of items without any context. Given the title, I can see why some folks assumed it was the finished product.

So I bemoaned the problem of no list description in Ta-Da. And then today, as I was editing another list, I noticed that the Ta-Da lists now do have a description field. Wow! Did I miss that? I would have sworn it wasn’t there before. Perhaps did the Ta-Da folk read my mind and knew exactly what I wanted. Anyways, I’m impressed. Thanks guys.

So, anyways, my apologies to the Python crowd. The "10 Things …" list was never targeted at Python and was an unedited compilation of feedback I had received. As Ian (the Python guy) points out a comment …

For Python and Ruby to try to take users from each other is a losing game — we’re both small players, and there’s much better and bigger pools of developers we should be trying to attract.

Ok, that’s enough for this rather rambling post.

Code Red

Code Ruby (or Python).

Paper Rock Scissors …

The weeks Ruby quiz involves writing a simple program to play the game "Paper, Rock, Scissors". Soon after the comments started on the quiz, someone submitted this 12 line "cheating" program:

  class Cheater < Player
    def initialize opponent
      Object.const_get(opponent).send :define_method, :choose do
        :paper
      end
    end

    def choose
      :scissors
    end
  end

It cheats in that it dynamically modifies its opponent object so that it always returns :paper, making it trivial to beat. While it does cheat, it does show some clever "thinking outside the box" approach to playing Paper, Rock, Scissors.

This reminds me of a story …

Greed

A number of years ago, I used to teach C++ in GE’s after hours education program. One of the projects assigned to the class was writing a player for the game of Greed. Greed is a simple dice game where you score points by rolling a set of 5 dice. You can continue to roll as long as you continue to make points on each roll. However, a row that has no point value causes you to lose all the points for a turn, so there is some value in stopping while you are ahead (hence the name Greed).

The game framework handles the dice rolling. All the student’s prgram had to do was decide whether or not to roll again given the the current state of the dice.

As part of the course, I pointed out that the encapsulation and data hiding features of C++ made it difficult to "cheat" by effecting the framework or other player objects. One particular student took this as a challenge and announced that he had a "cheating" player program that stayed within the "rules" setup by the C++ langauge, but yet would always win.

How did he do it? This cheating player would record the state of the seed of the standard library’s random number generator. It would then do trial rolls of the dice until it got a high scoring roll. Then it would reset the random number seed to the value that produced the high scoring roll, and indicate to the game framework that it wanted to roll again. The framework would comply and give the player the high scoring roll that it was setup to do. The cheating player would win the game in a single turn.

The cheat was easily defeated (once the cheat was understood) by using a private random number generator that wasn’t accessible to the player programs. But I learned two things from this exercise:

1. Programmers love challenges, especially the kind that say "You can’t do this …"
2. Secure software is hard to create, particularly because of point 1.

• Feedback