Commit 7031e53

@@ -0,0 +1,25 @@
+#!/usr/bin/env ruby
+
+WHITESPACE = Regexp.new('[[:blank:]]+')
+SPLITTABLES = [';', '(', ')']
+
+def tokenize(code)
+  pattern = Regexp.new("[^#{Regexp.escape(SPLITTABLES.join)}]+")
+  output = []
+  tokens = code.chomp.strip.split(WHITESPACE)
+  tokens.each do |token|
+    prefix, stem, suffix = token.partition(pattern)
+    output << prefix.split('') unless prefix.empty?
+    output << stem unless stem.empty?
+    output << suffix.split('') unless suffix.empty?
+  end
+
+  output.flatten
+end
+
+
+code = ARGV[0]
+tokens = tokenize(code)
+
+puts "Input: #{code.inspect}"
+puts "Tokens: #{tokens.inspect}"

@@ -21,4 +21,49 @@ first is bigger
 
 Chapter 11:
 
+Figure 11.9
+
+1. Identify the tokens in each of the following statements. (You do not need to classify them; just identify them.)
+    a. `if (a == b1) a = x + y;`
+
+      | token | type                 |
+      | ----- | ----                 |
+      | if    | if condition         |
+      | (     | left paren           |
+      | a     | symbol               |
+      | ==    | comparison operator  |
+      | b1    | symbol               |
+      | a     | symbol               |
+      | =     | assignment operator  |
+      | x     | symbol               |
+      | +     | addition operator    |
+      | y     | symbol               |
+      | ;     | statement terminator |
+
+    b. `delta = epsilon + 1.23 - sqrt(zz);`
+
+      | token   | type                  |
+      | -----   | ----                  |
+      | delta   | symbol                |
+      | =       | assignment operator   |
+      | epsilon | symbol                |
+      | +       | addition operator     |
+      | 1.23    | floating point number |
+      | -       | subtraction operator  |
+      | sqrt    | symbol                |
+      | (       | left paren            |
+      | zz      | symbol                |
+      | )       | right paren           |
+      | ;       | statement terminator  |
+
+    c. `print(Q);`
+
+      | token | type                 |
+      | ----- | ----                 |
+      | print | symbol               |
+      | (     | left paren           |
+      | Q     | symbol               |
+      | )     | right paren          |
+      | ;     | statement terminator |
+
 Chapter 12: