Copper Thoughts

Too Much Abstraction

2011-07-02T05:36

Object Oriented & Duck Typed

If you're not familiar with Python, you owe yourself a few minutes to check it out. It's a beautiful language that lets you get stuff done quickly, with little ceremony.

That being said, I just can't understand why one would want to write object-oriented code in the language, even given that it really pushes you to write in the OO-style. The primary reasons for writing classes (the poor man's types) is so that you can abstract away a a piece of information, or create a new data type.

The latter use makes sense, though most of the structures you'd want to use in Python are there, and the ones that aren't are generally better implemented in C. The former use mostly results in boilerplate code. You don't get the beautiful code that results from having static types; no, the language is duck-typed. You have no safety, and no gentle enforcement. Then you have the normal sins of OO: you have your getters, you have your setters, and then you have some domain-specific query and processing methods. In this case, your domain is your class. And what a sad, small domain that is.

A slightly more appealing option is to inherit from core data structures like UserDict. But you're still writing more code than you need to be.

Instead, use the core data structures.

Data as Data

I recently found myself need to convert from X arbitrary unit to Y arbitrary unit (and so forth) defined by a text file of equalities. A sample:

4 foo = bar
bar = 3 baz
baz = 2 nar
quark = 7 cthulu
foo = f
bar = b
f = 15 hur

The idea was to allow me to convert any arbitrary amount of one unit to another, even if it meant many conversions in between. Say, for instance, that I wanted to see how many nars 27 foos was. You could eye the relatively simple list above and see that you'd need to follow the unit path of foo > bar > baz > nar, and apply scalar transformations to the original amount along the way. The transformation looks like this: 27 foo > 6.75 bar > 20.25 baz > 40.5 nar.

But how to represent that in Python? A directed, weighted graph seemed like the best choice to me, so that's what I implemented. The direction indicated the direction of the conversion, with the weight being the scalar by which the units differ by.

A graph is a set of two sets. The first is a set of nodes, the second is a set of edges. I decided to use a tuple of sets, one of nodes and one of edges. Nodes are strings representing the units (e.g. foo, bar), and edges are tuples of edges and the conversion weight (i.e. (foo, bar, 1/4)). Here's how I wrote that in my code:

"""
graph = (set(node), set(edge))
node = String
edge = (node[origin], node[dest], Fraction(C))
       where C is the fractional conversion factor between node[origin] and 
       node[dest]
"""

Note that it's a comment. Every method I need to access and manipulate my data is already there, and everyone who might read my code is familiar with them. I could provide a simple interface if I wanted (e.g. nodes() or edges()), but that would be a little superfluous. Alternatively, this is how I could have described by structure if I had been writing OO code:

class Graph(Object):
    def __init__(self):
    self.nodes = set()
    self.edges = set()

def nodes(self):
    return self.nodes

def edges(self):
    return self.edges

def addEdge(self, src, dst, c):
    self.edges.add(src, dst, c)

def addNode(self, node):
    self.nodes.add(node)

Which appears to be the same thing to me, though obfuscated and verbose, not to mention a bit slower.

Hedging

Maybe I'm way off the mark here, and the increasing amount of functional programming I have been exposed to has tainted my otherwise pristine intellect. But why hide the data structures behind your datastructure any more than necessary? And in this case, the obfuscation only serves to name the actions being carried out in the context it takes place.

It's important to note, lest a strawman be made, that this post isn't meant to disparage the abstractions that classes, types, and interfaces provide a programmer. Nor should it apply in the case of an API, where an interface should be specific and clear.

Instead, I find myself struggling and questioning the need for what I perceive to be ineffectual abstractions. I suppose much of depends on just how much power you're comfortable giving yourself and whoever may use your code, and how much you need to hide in order to keep the code grokable.

Clojure Atlas

2011-05-30T06:00

Though I know I'm late to the party, I've been enjoying poking around Chas Emerick's newest project, Clojure Atlas. It's particularly pleasant to use it in conjunction with the Github page for Clojure, move to the relevant .clj and browse around it with the Atlas as accompaniment.

The Atlas is essentially a live graph connecting functions, macros, "Classes", reader syntax and just about everything else you can think of. A single click brings up the source code, as well as the documentation. My one complaint is that the graph takes too long to stop moving, making it difficult to quickly navigate: according to his Twitter (and my memory), this is something he's working on. My one wish for the product: integration with ClojureDocs, a great resource for burgeoning Clojurists and a way to make Clojure Atlas more accesible to those who are newer to the language, as well as make it more of an encyclopedia.

It costs $40 if you buy the Atlas for Clojure 1.2 and 1.3 (coming soon), or just $25 if you'd like to buy the 1.2 version. It is definitely worth a look.

Lovely Pianobar

2011-05-29T06:18

If you love using Pandora, but really wish they had a command-line (or at least non-Flash) version of the desktop application, this post will make your day. An absolutely wonderful CLI utility called pianobar solves this problem for you, and a few more you didn't know you had. Something to note: there are no advertisements when using pianobar; this may be violating Pandora's terms of service. Because I think Pandora is supplying a great service, I pay for the $3/month Pandora+ membership and would not be hearing advertisements anyway: if this method seems like a great way to listen to music I'd recommend you pay the $36 too.

This tutorial will only cover setting up this environment on a Mac, though if you're running Linux you should be able to do much of the same using a few different applications in some cases.

What You'll Need

With that said, you can build pianobar from source, or use a package manager like brew to install it. Occasionally you need to update pianobar in order to get the newest auth keys. This should not be the difficult step.

You'll also want to install Alfred (Alfred deserves a whole post to itself) to get this to work, though you could accomplish similar results with KeyboardMaestro or another application launcher like LaunchBar. In order to do this using Alfred, you'll need to buy the powerpack (basically, an upgrade) for around $20. Any way you have of executing scripts via keyboard shortcut will work as well, as will your command-line itself, though. Alfred is definitely worth twenty dollars.

Finally, you will need Growl if you don't already have it, and its CLI utility growlnotify. If you're opposed to Growl for whatever reason, you don't need it, but it's a nice method of visual feedback.

I'm going to assume you already use a terminal multiplexor like tmux or screen, but if you don't you owe it to yourself to look into it.

The Execution

If you don't already have one, make a .config/ directory in your home folder and add a pianobar/ directory there as well. Here you'll make a file called config and write the following in it:

user = username@email.com
password = plaintext_password
event_command = /Users/username/.config/pianobar/event-cmd

Setting your username and password here allows you to start up pianobar and immediately begin listening: it's no fun to have to type your username and p/w in every time (though if you're using tmux/screen you should rarely have to anyway). event_command lets you designate an executable that will accepts output from pianobar on every event (e.g. starting a new song, hating a song, explaining a song).

In that file, at /Users/username/.config/pianobar/event-cmd put this shell script. In short, it parses the output from pianobar sent with every event and, depending on what event is occurring, sends a certain message to Growl.

In the same directory put an icon you'd like to use, and call it icon.png. I used this simple pixel eighth note for my icon. It's also possible to specify different icons for different events, as you can see in the shell script.

Next you'll need to create a FIFO named "ctl" (with mkfifo) in the same directory. You can send commands to this file and pianobar will respond to them appropriately. Now is probably a good time to start up pianobar in a tmux session, too. Make sure everything is working so far.

Finally, we get to tie this all together:

Here you can see my current (rapidly growing) list of shell commands Alfred handles for me. More pertinent are the "p" and "pp" commands. The former runs: echo "{query}" > ~/.config/pianobar/ctl, while the second just toggles play by passing the FIFO "p". In this way, I can open Alfred with CMD+SPACE,p type "p" and a space, then pass any command to pianobar from there. Most often I send "n" for "next song" or "e" for "explain", which displays the title, artist and album of the song via Growl. If I wanted to, I could bind any function keys I'd like to these commands as well, but typing 3 characters works fine for me.

So I have a lightweight, easily controlled music setup. If you're cheap, the cost is free. If you'd like to support Pandora as well as get a fantastically useful application, that'll be $46.

If you'd like to expand more on this setup, be sure to read the pianobar man pages, which are very helpful; you can customize the keys used to control pianobar, manage even more events and more.

The Ins and Outs of Clojure: Part One

2010-11-13T21:50

written about Clojure 1.2

It is a truth universally acknowledged, that a programmer using Clojure will want to perform IO. Let me help you out (put).

I'll go over some of the basics of IO, focusing on what you can use Clojure to do directly. I'll move on after the basic introduction, to some of the more interesting and generally useful classes that Java offers, giving a little context for each.

In

Reading files in is generally one of the first things I want to do when playing with a new language, so I'll start there. Before I get started though, I should mentioned that in Clojure, strings are always encoded using UTF-16. Generally this saves time and worry, but it's something to keep in mind should you run into problems on the encoding front.

slurp

Clojure comes with a handy little function called slurp that takes in a string representing a filename (or, really, pretty much anything; a File, a stream, a byte array, URL, etc) and returns a string containing the contents of your file. It's pretty handy if you just need to get some information from a file that's relatively small, and you'll be parsing it yourself.

(slurp "/Users/ihodes/Desktop/afile.txt")
=> "A little bit\nof information here."

A nice thing about slurp is that you can easily build up file paths with str. For example, say you want to output to a file based on information you find at runtime:

(slurp (str "/Users/" username "/Desktop/" filename))

But slurp is pretty basic, and once your files get large enough, totally impractical. Nonetheless, it's a handy function to know about.

As a useful and comical aside, the function spit is the counterpart to slurp, except that instead of reading input, spit does output. More on this in a future article, though.

line-seq

One of my favorite IO functions has got to be line-seq; line-seq takes a reader object (which must implement BufferedReader) and returns a lazy sequence of the lines of the text the reader supplies. This is handy when you're dealing with files (if this offends you, let's be Unixy here for now and say that everything is a file) that are too big to merely slurp, but that are \newline delimited (or CR/LF delimited, if you're of the Windows persuasion).

(use '[clojure.java.io '(reader)])
(take 2 
  (line-seq (reader "bobby.txt")))
=> ("Bobby was a good boy," "and didn't complain too much")

Notice how we take 2 from the sequence we get from using line-seq. We can take as much or as little as we need; we won't be reading much (Clojure will read a bit more than you tell it to in order to get more IO performance, but let's not worry about that) more than we specify. We can do anything we want with the resulting seq; that's the beauty of line-seq and the ubiquitous sequence abstraction.

Back in the day, Clojurists had to sink a little lower than the clojure.java.io namespace to use line-seq; two Java classes were needed. One of these Java classes is the most wondrous and amazing thing just below the surface of the more elegant and beautiful Clojure code; BufferedReader. Here's how we used to do it;

(import '(java.io FileReader BufferedReader))
(take 2 
  (line-seq (BufferedReader. 
              (FileReader. "bobby.txt")))
=> ("Bobby was a good boy," "and didn't complain too much")

This might give you a better sense of what's going on when you use reader, though in reality reader is far more complicated than just that: you can trust it to handle a variety of "readable things" and return to you a BufferedReader if possible.

FileReader will return a Reader on a file, and BufferedReader takes and buffers a Reader, as you might have extrapolated from the name. Readers are basically just objects upon which a few methods (like read, skip and close) may be enacted and expected to return reasonable results. line-seq essentially reads up until a line-delimiter and returns the read chunk as an element in the sequence it is generating.

While on the subject of files, I should probably mentioned the file function, from clojure.java.io. file takes in an arbitrary number of string arguments, and pieces them together into a file hierarchy, returning a File instance. This can come in handy.

Rivers? inputStreams? Brooks?

Streams are an especially useful class of readers. Oftentimes you're reading in text; that's what Readers do. But often you need to read in a stream of bytes; that's where you need to use clojure.java.io's input-stream.

(use '[clojure.java.io '(reader)])
(def g (input-stream "t.txt"))
(.read g)
=> 105 
(.read g)
=> 115
(char (.read g)) 
=> \space

As you can see, instead of getting characters from this file (like we get when we use a reader), we're getting integer byte values. This can be useful when reading, for example, a media file.

In general, strings are always UTF-16, which are 16-bit pieces of data, whereas byte-streams are 8-bit pieces of data. It bears repeating that the stream operators should be used when you're not dealing with strings: they are not trivially interchangeable, as they might be in other languages where strings are syntactic sugar for byte arrays.

RandomAccessFile

Finally, let me introduce to you a spectacularly useful Java class. RandomAccessFile is a class which allows you to quickly jump around in a large file, and read bytes from it.

(import '(java.io RandomAccessFile))
(def f (RandomAccessFile. "stuff.txt" "r"))

Note the second argument of the constructor, "r"; this indicates that we're opening the file just for reading. Now that we have f, we can use it to navigate and read the file:

(.read f)
=> 105
(.length f)
=> 2015 ;; this is the number of bytes this file is in length
(.skipBytes f 20)
(.getFilePointer h)
=> 21 ;; the position we're at in the file
(.read f)
=> 89

As you can see, you can jump around (quickly!) through a file, and read from the parts you want, and skip the parts you do not want. The key methods/functions here (among many others that can also be useful; be sure to check the documentation) are read, length, skipBytes, seek and getFilePointer.

Closing

Every file that is opened should be closed, and what we've been doing is a little unsafe. In order to close an open reader/file, we should use the close method on it; in the above example, when you're done with f, simply execute (.close f) to tell the file system that you're done with the file. Alternatively, and more idiomatically, you can open your files with the handy with-open binder:

(with-open [f (RandomAccessFile. "stuff.txt" "r")]
           (.read f))

When you're done with f, Clojure will close it, and you won't have to worry one iota about it.

Digging Deeper

Should slurp and line-seq not be enough for your reading needs (and chances are that, should you code enough in Clojure, they won't always been), you might want to explore clojure.java.io some more, as well as some of the Java classes (namely, those stemming from Reader and BufferedReader, as well as InputStream and BufferedInputStream) mentioned above. See my previous article on using Java if you're unfamiliar with using Java.

Next up is an introduction to the "outs" of Clojure and Java. Stay tuned!

I owe a big thank you to Phil Hagelberg for reading over this essay and offering advice. If you don't already, you should be using his Leiningen for both dependency management and a stress-free development environment.

Fun With Set Theory and Lisp

2010-09-01T15:40

A few weeks ago a provoking question was posted on StackOverflow asking if, for example, Common-Lisp's numberp could be written using only the primitives that John McCarthy presented in his now famous essay introducing LISP.

My answer was yes, but it wasn't the kind of yes you might expect. In his essay, McCarthy did not even mention numbers. There are no primitives to deal with them; only primitives to construct and deconstruct lists (i.e car, cdr, cons), as well as a few to enable the creation and naming of functions (i.e. lambda, label). But there are no number primitives: no direct interaction with the ALU, no way to directly access and manipulate integers and floats. So we have figure something else out.

It's All Lists From Here

So how can you represent numbers?

I'm going to eschew using the troublesome cons cell in my explanations. Instead, we'll use the more common list notation that's both easier to explain and easier to write. Of course, I'll be using the cons function (how could I not?) and you should at least have an idea of how that works.

It turns out that there are a number of ways to represent the natural numbers. Paul Graham postulates one way of representing them in his essay The Roots of Lisp. I'll state here, slightly tidied up, his suggestion:

Let the empty list be zero.
Let 1 = (cons 0 ())
Let 2 = (cons 1 ())
etc.

In this way we can represent all the natural numbers. It'd be trivial to write a function number? that would test to see if if its argument had the form of a number. There are a number (hah!) of things we could do with this representation, but I think we can do better.

Edit: Phil Hagelberg reminds me of another, very important, representation of the natural numbers I'd be remiss to leave out: the Church Encoding. Church Encoding is way of representing the natural numbers using only lambda calculus: there are no lists involved. Instead, a number n is representing by a higher order lambda abstraction (a function), and returns the result of that function being composed with itself n times. There's actually a fair amount that can be done with this representation, but it's outside the scope of this essay. For the curious, the afore-linked Wikipedia article does a fine job of covering the basics.

The More Lists, the Merrier

But really, that's not enough lists, is it? (((((((((()))))))))) is not nearly an unwieldy enough definition for 10. We need more lists.

Let's take a step back and look at set theory; a branch of math and CS that's informed a lot of underpinning of CS and programming in general. Set theory has already provided us with a handy way of defining numbers based on lists. Well not lists, sets, but sets are basically lists. In fact, contrary to popular/programming belief, sets can hold the same thing multiple times, but in set theory that means nothing. We'll say the same thing about lists: "when you say my list a contains b and b, I just hear that b is in a".

So how do we represent numbers using only sets? Well, we start with the empty set {} being zero. But, and here's where things get weird, we now define a function S called the "successor function". The successor of n is the union of n and the set of n, where union is a function of n and m that returns the set which contains all the elements in n and all the elements in m. S(n) = {n and all the elements in n}. To make this clearer, let's look at a few examples.

S(0) = union(0 {0}) = {0} ;; we'll call this 1
S(1) = union(1 {1}) = union({0} {{0}}) 
       = {0 {0}} ;; and this 2
S(2) = {0 1 2}  ;; 3
S(3) = {0 1 2 3} ;; 4

At the very least, this is an interesting definition of the natural numbers. The predecessors of each number are contained within it. And the set of all the natural numbers has a form similar to an individual natural number. Neat.

Let's see how we can represent this with lists. Our goal is a function, S, that returns the successor of a given list.

(define S
  (lambda (n)
    (union n (cons n ()))))

Perfect, though we don't exactly have a union function yet:

(define union 
  (lambda (n m)
    (cond ((null? n) m)
           (else (cons (car n)
                       (union (cdr n) m))))))

Now we do. You can test it out in your own REPL, or online with TryScheme. We're going to move on now.

So What?

Thats a valid question. At this point, it seems as though we've merely complicated Mr. Graham's elegant and simple proposition. It turns out that we've actually made things easier, in the long run.

Before we even get to addition and the like, let's look at one example of where we've made things easier and more elegant. Let's look at an ordering on the natural numbers; <. With this definition, we don't need anything else to define it: n < m if (and only if) n is in m. That's a straight forward definition, and one we can check rather quickly with a Lisp function you might have on hand anyway (The Little Schemer calls it member?).

But first we need a better eq?. The current eq? operates only on atoms; we need an equal? that works on our list representation of numbers. To keep things simple, we're going to assume our lists are ordered. That will be the case if we build our numbers with just S.

(define equal?
  (lambda (n m)
    (cond ((and (null? n) (null? m)) #t)
          ((or (null? n) (null? m)) #f)
          (else (and (equal? (car n) (car m))
                     (equal? (cdr n) (cdr m)))))))

(define member?
  (lambda (a l)
    (cond ((null? l) #f)
          (equal? a (car l)) #t)
          (else (member? a (cdr l))))))

(define <
    (lambda (n m)
       (member? n m)))

This works because all our natural numbers are either the empty list or a list of natural numbers, thanks to our nicely defined S function.

It turns out that this successor function is also nice for defining and working with ideas like ordinals and cardinals, and just generally makes out theorems easier to prove and reason about. But, without writing a book on it, suffice it to say there's a certain elegance about it. I'll leave it at that.

In Addition…

But back to our problem: numbers on a simple Lisp machine. We've now got a suitable representation of the natural numbers (though I've neglected to introduce the axioms required to fully justify this; let's just use common sense and carry on), and we have a way of comparing two numbers.

We also have a way of adding one to a number in S. How about adding two natural numbers together?

(define zero?
  (lambda (n)
    (equal? () n)))

(define A 
  (lambda (n m)
    (cond ((zero? m) n)
          (else (S (A n (P m)))))))

That is a nice recursive definition, but we have a mysterious new P function now, as well. P stands for "predecessor", and it gives us the natural number than m is a successor of.

Before defining P, let me present the set theoretical definition of addition:

A(n, 0) = n
A(n, S(m)) = S(A(n, m))

This is also recursive, but it has the unfair advantage of what the Clojurist in me calls the "de-structuring" of its operands. A can check to see if its second operand is a successor, and if it is will take the successor of the result of another A's value.

With Lisp, we can check to see is m is zero, and if it isn't, we know that it is a successor, we just don't know what it is a successor of. That is where P comes in.

(define R
  (lambda (fn acc l)
     (cond ((null? l) acc)
           (else (R fn (fn acc (car l)) (cdr l))))))

(define P
  (lambda (n)
    (cond ((zero? n) #f) ;; really, undefined
          (else (R union () n)))))

And out of nowhere comes a higher-order function! While set theory may have some expressive advantages over Lisp, we've got some special sauce, too. In this case (R union () n) is equivalent to the set theoretical "union over a set" that the Zermelo-Fraenkel Union Axiom gives provides. Most people call R "reduce".

You might notice that we get duplicates in our lists, but as I said before, we don't pay those any attention. To keep things clean and working well, we'll implicitly apply a helper function, distinct, to the result of every operation. distinct is left as an exercise to the reader; but this is a very important exercise: some of our function will not work if distinct isn't applied to the result of each function.

With a proper P defined, A now works flawlessly. Now let's define M, multiplication and B, subtraction:

(define M
  (lambda (n m)
    (cond ((zero? m) ())
          (else (A n (M n (P m)))))))

(define B                                                         
  (lambda (n m)
    (cond ((zero? m) n)
          (else (P (B n (P m)))))))

It gets easier and easier as we build off of our previous functions, and combine them to yield more complex, more useful abstractions. Finally, let's write our number predicate;

(define length
  (lambda (n)
    (cond ((null? n) ())
          (else (S (length (cdr n)))))))

(define number?
  (lambda (n)
    (equal? n (length n))))

And there you have it; we've got the basics of our natural numbers, represented using McCarthy's basic LISP.

But, Wait! There's More

Ah, we've left off the integers. And floats!

Fortunately, this isn't a big deal. With our newly defined natural numbers, we can represent integers as a tuple of natural numbers, where the first natural number in the tuple represents the "negative" portion of the integer, and the second natural number represents the "positive" portion of the integer.

In this manner we can represent all the integers. Of course, we'll have to create new functions for adding, subtracting and the like, but that becomes a matter of building new, higher level abstractions with our existing functions.

For example, we can define adding two integers like this:

(define first
  (lambda (n)
    (car n)))

(define second 
  (lambda (n)
    (car (cdr n))))

(define makeint
  (lambda (n p)
    (cons n (cons p ()))))

(define A2
  (lambda (x y)
     (makeint (A (first x) (first y)) 
              (A (second x) (second y)))))

Something important to keep in mind, however, is that different tuples can represent the same integer. For instance, the integer (2, 5) is the same as the integer (0, 3); they're both what we would have called "3" back in the natural numbers section. In fact, it makes sense to call both of them "3" here. But the second example here is preferable, I'd say. This is something that is dealt with in set theory by equivalence relations; we can do the same thing in Lisp. Below I'll define a function which returns the "simplest" version of a given integer; one where either the negative portion is zero, the positive portion is zero, or they are both zero.

(define simplint
  (lambda (z)
    (cond ((or (zero? (first z)) (zero? (second z))) z)
          (else (makeint (P (first z)) 
                         (P (second z)))))))

Subtraction, multiplication and other operations on integers begin to fall into place with these definitions. Things don't get more complicated as you move further away from your base representation (unless you try to formally prove all of my assertions): if anything, they get easier. This is a consequence of appropriate abstraction.

So far, we have arbitrarily large integers available to us (limited in practice only by memory and processing power). But we're still missing a type of number most programmers expect to have available to them: the float. Well, we can do that too.

First, let's make sure we know what a float can and cannot represent. Floats can't represent all real numbers: that is to say, every float is representable by a ratio of two numbers. In fact, traditional floats can't even represent all rationals: for instance, 1/3 is 0.333… repeating forever. We would run out of bits to represent it on a real computer. But we can represent all numbers a float can, and more, with ratios. And, as I'm sure has now become clear, we definitely have ratios in the form of a tuple of integers, the first representing the numerator and the second the denominator. These are the rational numbers.

Again, an equivalence relation on the rationals is called for, as are new operations. This time, though, they are left as an exercise for the reader.

Final Words

Though a lot more can be said about the intersection of set theory and Lisp, I'll stop while this post is still of Internet-length. If there's enough interest (mostly on my part), perhaps this will be extended into a series of essay on the subject, mirroring the progression of a elementary set theory book.

This essay is a product of my investigation into the possibility of a truly programmable and reprogrammable Lisp environment, one in which such a definition of the natural, integer and rational numbers would not only be feasible and efficient, but idiomatic. I'm a long way from explaining how such a system would work, but I thought I should record at least some of the byproducts of my thought process so that someone might obtain some value from my mental wanderings.

I should note that while I used the operators and and or, neither were in McCarthy's original LISP. Consider (and a b) a stand-in for (cond (a b) (else #f)) and (or a b) for (cond (a #t) (else b)), and all is well.

If there are errors in my explanations or code samples, please let me know. I certainly simplified much of the content so that it could fit in about 2000 words, but it should be correct.