Erlang on Probably Programming

Walking Journal: 45 miles, starting a Twitter client chain

Fri, 25 Sep 2009 03:37:22 +0000

I’ve been neglecting both walking and posting on here, though I’ve neglected posting more, obviously.

I’ve been very busy with work and side projects and my house trying to fall apart on me, so I’ve been out of free time lately.

Anyway, my total is now up to 45 miles of walking. Not too much more than last time, really. I’ve gone on some walks around the neighborhood with Angie, and I’m not counting them, mostly because I don’t know how long they are, really.

I decided I wanted to start a “chain,” where every day I do something to work toward a goal I’ve set for myself (yes, I mean other than the walking, though that can be a chain as well).

Fortunately enough for me, I found Don’t Break The Chain, where you can keep track of one or more “chains” of things you want to do every day.

The Twitter Client Chain

So I’m creating a web-based Twitter client, as a project for me to try out all the different techniques and technologies that I want to use, but don’t have the chance to work with as much as I’d like during my day job. Also, ideally, it will at some point become a supplementary source of income.

That’s a quite a way off until I have more free time in a month or two, but for now, I will satisfy myself with making some small amount of progress on it each day (hopefully).

I’m writing this client in Erlang, using the awesome Nitrogen web framework with the equally awesome MochiWeb web server. My goals for this project include using test driven development and continuous deployment and fully embracing the mantra of “release early, release often”.

The plan is to have something up and running on a server as soon as possible. By something, I mean even less than a full-featured twitter client. But as soon as something is up, I can start getting feedback, and see what I should work on next. If I’m creating a product for users, there’s no use in developing what I want.

I need to develop what my users want.

I started this chain up about a week ago, and broke it after the first day. Now I’m trying again.

Tests and pre-commit hooks

During my previous, one-day chain, I managed to get a test harness (using the EUnit Erlang unit test framework) set up which will find any modules ending with _tests, stopping at the first error.

At the moment, I have a shell script which runs the tests. When it starts to take too long, I will switch over to an Erlang module which loops through, so I don’t get the startup times of the Erlang interpreter for each test. Or if anyone knows an easier way (perhaps there’s already a module that does that somewhere?).

    #!/bin/bash
    
    for modfile in `ls ebin/*_tests.beam`
    do
        mod=`basename $modfile .beam`
        echo Running $mod:
        erl -noshell -pa ./ebin -s runtest main ${mod%.*}
        [ $? -eq 0 ] || exit 1
    done
    [ $mod ] || exit 1

The runtest module consists of the following:

    -module(runtest).
    -export([main/1]).
    
    main(Mod) ->
        case eunit:test(Mod) of
            error -> halt(1);
            _ -> init:stop()
        end.

It’s all kind of thrown together, but the point is incremental improvement, so I’ll get back to it when/if I need to. Like I said, there’s probably something else out there that will do a better job at this than I did.

So, today, my “chain link” was partially taking that script and setting it up to run from the GIT pre-commit script, and partially writing this blog post.

Getting it set up to run as a pre-commit script just involved making sure it returns non-zero whenever there is a test error (or if no tests were run, which happens when make hasn’t been run yet).

The pre-commit script itself (placed in .git/hooks/pre-commit and set executable) is pretty simple:

    #!/bin/sh
    ./tests.sh
    if [ $? != 0 ]
    then
            echo "===================================="
            echo " Commit stopped due to test failure"
            echo "===================================="
            exit 1
    fi

Chain length: 1!

What’s next?

Next up, I’m acutally going to start on the meat of the project. Since I haven’t fully decided on what GUI library I want to use, nor what GUI testing framework I want to use, I’m going to start on the backend with code to manage user login info and queries (timeline searches, DMs, etc.).

I’ve got some great ideas (I think) on ways to enhance the way people have conversations over Twitter (and eventually other mediums), so you can be sure that there will be more posts about this in the future.

Nitrogen module auto-reloading

Sat, 20 Jun 2009 19:27:14 +0000

This is a really simple tip, but it was handy for me, and it could be helpful for someone else, too.

While playing around with Nitrogen, which is a great project by the way (more in a later blog post, I guarantee it), I noticed that I had to restart the server to re-load modules as I changed them (or I could manually, reload them, I know, I know..).

Since I was using Mochiweb as my backend, and I’m used to the development version of Mochiweb auto-reloading modules as you recompile, I wanted the same functionality here.

Turns out it’s as simple as changing the default start.sh from

    #!/bin/sh
    cd `dirname $0`
    
    echo Starting Nitrogen.
    erl \
    	-name nitrogen@localhost \
    	-pa ./ebin -pa ./include \
    	-s make all \
    	-eval "application:start(appname)"

to this

    #!/bin/sh
    cd `dirname $0`
    
    echo Starting Nitrogen.
    erl \
    	-name nitrogen@localhost \
    	-pa ./ebin -pa ./include \
    	-s make all \
    	-s reloader \
    	-eval "application:start(appname)"

… and bam, modules now reload as you recompile them.

Untiny that url!

Sat, 11 Apr 2009 20:36:21 +0000

There has been some talk about and arguments against and responses to issues about using rev=“cononical” for referencing shorter URLs instead of the automated use of TinyURL when posting to sites like Twitter.

I must say that I agree with Ben Ramsey (see “arguments agains” above) in suggesting we use rel=“alternate shorter” instead.

I also like the idea that Chris Shiflett had of using a HTTP header and a HEAD request to make it so you neither have to retrieve the entire requested page nor parse any HTML. I’d stick with Ben’s suggestion, however, and make the header something like “X-Alternate-Shorter:“, rather than “X-Rev-Canonical”. What’s the harm in calling it something that actually makes sense?

The idea of using HTTP HEAD requests to solve the problem inspired me to come up with a more immediate solution to one of the problems introduced by using url shortening services: uncertainty about where a URL leads.

This problem can be solved on the client side, which requires no work on the part of Twitter (meaning this is more likely to be put into use sooner).

Since most URL shortening services use an HTTP redirect to do their job, all it takes is a HEAD request to the tiny URL in question, and then a look at whatever “Location:” header is returned to see what the real URL is. In fact, you don’t even really need to do a HEAD request in most cases, since most URL shortening services don’t return any body, since they are just redirecting you anyway.

Read on for more information and implementations of an untinyurl function in various languages.

There’s actually already a site online that offers the service of un-shortening URLs for you at UnTinyURL.com, but I wouldn’t suggest using that in any sort of automated system, and it’s of limited usefulness since you don’t really want to have to go to this site just to see what site you’re about to go to. Most people will just click a link, even if it means they might get RickRolled.

For those comfortable with the commandline, a simple curl call can give you the same basic info:

    curl -I http://tinyurl.com/c8f5bz
    HTTP/1.1 301 Moved Permanently
    X-Powered-By: PHP/5.2.9
    Location: http://probablyprogramming.com/2009/04/11/untiny-that-url/
    Content-type: text/html
    Date: Sun, 12 Apr 2009 01:26:08 GMT
    Server: TinyURL/1.6

Toss in a grep and an awk, and you get your URL in a single line, perfect if you’re handing shortened URLs in a shell script for some reason:

    $ curl -s -I http://tinyurl.com/c8f5bz | grep Location | awk '{print $2}'
    http://probablyprogramming.com/2009/04/11/untiny-that-url/

Here’s untinyurl in Python:

    import httplib
    import urlparse
    
    def untinyurl(tinyurl):
        url = urlparse.urlsplit(tinyurl)
        req = urlparse.urlunsplit(('', '', url.path, url.query, url.fragment))
        con = httplib.HTTPConnection(url.netloc)
        try:
            con.request('HEAD', req)
        except:
            return None
        response = con.getresponse()
        return response.getheader('Location', None)

And here’s a version in PHP. It’s a bit longer and uglier than the Python version because I’m using the low-level fsockopen function to do my HTTP request rather than using cUrl or the HTTP extension. The reason I did this is because every PHP install will have fsockopen, whereas not every install will have cUrl or the HTTP extension.

    <?php
    
    function untinyurl($tinyurl) 
    {
        $url = parse_url($tinyurl);
        $host = $url['host'];
        $port = isset($url['port']) ? $url['port'] : 80;
        $query = isset($url['query']) ? '?' . $url['query'] : '';
        $fragment = isset($url['fragment']) ? '#' . $url['fragment'] : '';
    
        $sock = @fsockopen($host, $port);
        if (!$sock) return $tinyurl;
        
        $url = $url['path'] . $query . $fragment;
        $request = "HEAD {$url} HTTP/1.0\r\nHost: {$host}\r\nConnection: Close\r\n\r\n";
    
        fwrite($sock, $request);
        $response = '';
        while (!feof($sock)) {
            $response .= fgets($sock, 128);
        }
        $lines = explode("\r\n", $response);
        foreach ($lines as $line) {
            if (strpos(strtolower($line), 'location:') === 0) {
                list(, $location) = explode(':', $line, 2);
                return ltrim($location);
            }
        }
        return $tinyurl;
    }

I’m not too familiar with Ruby, but after poking around for a little bit, I came up with this Ruby version. Holy crap, Ruby, that was easy and short!

    require 'net/http'
    require 'uri'
    def untinyurl(tinyurl)
      Net::HTTP.get_response(URI.parse(tinyurl))['location'] or tinyurl
    rescue
      tinyurl
    end

And one more, and Erlang implementation (that <SEMI> is supposed to be a semicolon, but something is wrong with the syntax highlighter Erlang plugin). Be sure you call “inets:start()” before calling this function.

    -module(untinyurl).
    -export([untinyurl/1]).
    untinyurl(TinyUrl) ->
        case http:request(head, {TinyUrl, []}, [{autoredirect, false}], []) of
            {ok, {_Status, Headers, _Body{% templatetag closevariable %} -> proplists:get_value("location", Headers, TinyUrl);
            _ -> TinyUrl
        end.

Interesting how the Erlang and Ruby implementations look pretty similar.

I’ve made the source code available at GitHub. If you would like to contribute an untinyurl implementation in another language or have a bug-fix or suggestion for an improvement of one of the implementations I have so far, either email me, send me a pull request on GitHub, or post a comment here.

First steps to an Erlang OpenID consumer

Wed, 21 Jan 2009 03:18:14 +0000

I’m working on a little project in Erlang and I wanted to use only OpenID for my authentication. It turns out there is currently no Erlang OpenID consumer library (or if there is, I couldn’t find it).

So I’ve started writing my own. So far I’ve got the first necessary step complete: HTML-based discovery.

I’m starting with version 1.1, simply because it is shorter and requires less (I don’t want to implement the XRI or Yadis protocols just yet).

It turns out that mochiweb comes with an HTML parser, so I used that, since I’m using mochiweb for my application. The parsed HTML comes back as a series of nested tuples of the format {<<“tag”>>, Attributes, Children}, where “tag” is the tagname (the root will be <<“html”>>, for example), Attributes is a proplist of that tag’s attributes, and Children is a list of more tuples of the same format and/or binaries with the contents of text nodes. Everything is represented as binaries, so I use those directly rather than converting between strings and binaries.

Here’s the code which finds the link tags with rel=“openid.server” and rel=“openid.delegate” (if it is there):

    get_openid_server(Identifier) ->
        NormalizedIdentifier = normalize_identifier(Identifier),
        case http:request(NormalizedIdentifier) of
            {ok, {_Status, _Headers, Body{% templatetag closevariable %} ->
                HtmlTokens = mochiweb_html:parse(Body),
                find_openid_tags(HtmlTokens);
            _ ->
                {error, http_error}
        end.
    
    normalize_identifier(Ident = "http://" ++ _Rest) ->
        Ident;
    normalize_identifier(Ident) ->
        "http://" ++ Ident.
    
    find_openid_tags(HtmlTokens) ->
        case find_tag(<<"head">>, [HtmlTokens]) of
            {<<"head">>, _Attrs, Children, _Rest} ->
                case find_tag_with_attr(<<"link">>, {<<"rel">>, <<"openid.server">>}, Children) of
                    not_found ->
                        {error, openid_server_not_found};
                    ServerAttrs ->
                        Server = proplists:get_value(<<"href">>, ServerAttrs),
                        case find_tag_with_attr(<<"link">>, {<<"rel">>, <<"openid.delegate">>}, Children) of
                            not_found ->
                                [{server, Server}];
                            DelegateAttrs ->
                                Delegate = proplists:get_value(<<"href">>, DelegateAttrs),
                                [{server, Server}, {delegate, Delegate}]
                        end
                end;
            not_found ->
                {error, no_head_tag}
        end.
    
    find_tag(_TagName, []) ->
        not_found;
    find_tag(TagName, [{TagName, Attributes, Children} | Rest]) ->
        {TagName, Attributes, Children, Rest};
    find_tag(TagName, [{_OtherTag, _Attributes, Children} | Rest]) ->
        find_tag(TagName, Children ++ Rest);
    find_tag(TagName, [_Other | Rest]) ->
        find_tag(TagName, Rest).
    
    find_tag_with_attr(_TagName, {_AttrKey, _AttrVal}, []) ->
        not_found;
    find_tag_with_attr(TagName, Attr = {AttrKey, AttrVal}, Tags) ->
        case find_tag(TagName, Tags) of
            not_found ->
                not_found;
            {TagName, Attributes, Children, Rest} ->
                case proplists:get_value(AttrKey, Attributes) of
                    AttrVal ->
                        Attributes;
                    _ -> 
                        find_tag_with_attr(TagName, Attr, Children ++ Rest) 
                end
        end.

(the <PIPE>s above should be “|“s and the <SEMI>s should be “;“s. Not sure why the syntax highlighter is doing that to them…

Assuming the above code is put into a module called “openid”, you get the following:

    1> openid:get_openid_server("blog.paulbonser.com")
    [{server,<<"http://www.livejournal.com/openid/server.bml">>},
     {delegate,<<"http://misterpib.livejournal.com/">>}]
    2>

As I said, this is the first step. Hopefully I’ll have some time very soon to get on with the next couple of steps, and then I’ll be done.

Storing Hierarchical Data in CouchDB

Fri, 04 Jul 2008 18:20:52 +0000

Much to my surprise, my last post generated more traffic in a single day than my blog has ever gotten in a single month. Apparently people are quite interested in making web applications with Python. I’ve started on part two, but since so many people showed interest I want to spend more time on it than I spent on the last one. So instead, you get this post.

So I’ve been fiddling around with CouchDB lately. Since it’s common to store tree-based data, and it’s kind of a pain to do so in your standard relational DB, I thought it would be a good exercise to see how hard it is to store hierarchical data in CouchDB.

Turns out it’s pretty easy.

For comparison, you might want to check out this article on storing trees in a relational database. It covers how to store tree-like data using both Adjacency Lists and Modified Preorder Tree Traversal (that’s a mouthful). I’ll cover how I put the data into CouchDB and some of the ways you might want to pull it out.

Storing the Tree

Rather than keeping track of parents as in the Adjacency List method or ‘left’ and ‘right’ as in the Modified Preorder Tree Traversal method, I store the full path to each node as an attribute in that node’s document. I then use this data in the views to organize the data as I need it.

The test data I used is as follows:

    [
        {"_id":"Food",   "path":["Food"]},
        {"_id":"Fruit",  "path":["Food","Fruit"]},
        {"_id":"Red",    "path":["Food","Fruit","Red"]},
        {"_id":"Cherry", "path":["Food","Fruit","Red","Cherry"]},
        {"_id":"Tomato", "path":["Food","Fruit","Red","Tomato"]},
        {"_id":"Yellow", "path":["Food","Fruit","Yellow"]},
        {"_id":"Banana", "path":["Food","Fruit","Yellow","Banana"]},
        {"_id":"Meat",   "path":["Food","Meat"]},
        {"_id":"Beef",   "path":["Food","Meat","Beef"]},
        {"_id":"Pork",   "path":["Food","Meat","Pork"]}
    ]

In a real system you’d probably want to use some sort of UUID instead of descriptive strings, since conflicts between node names could be bad. In fact, it’d probably be much faster to just use numbers, since comparisons on numbers are generally much faster. For the purposes of this post, however, it’s much easier to understand if it’s descriptive text.

Once that data is in your DB, it’s time to get it out again!

Retrieving the whole tree

The CouchDB map function to retrieve the whole tree is nice and simple:

    function(doc) {
        emit(doc.path, doc)
    }

Using the path as the key, the documents will be sorted as above, with each parent immediately followed by its children.

One option to get the data into an actual tree would be to add a reduce function to the view:

    function(keys, vals) {
        tree = {};
        for (var i in vals)
        {
            current = tree;
            for (var j in vals[i].path)
            {
                child = vals[i].path[j];
                if (current[child] == undefined) 
                    current[child] = {};
                current = current[child];
            } 
            current['_data'] = vals[i];
        }
        return tree;
    }

Note: don’t use this reduce function, since it doesn’t take the rereduce parameter into account, and would most likely not work correctly if a rereduce was done.

I chose to write a similar function in Python and use that to generate my tree on the client side:

    class TreeNode(dict): pass
    
    def tree_from_rows(list):
        tree = {}
        for item in list:
            current = tree
            for child in item.value['path']:
                current = current.setdefault(child, TreeNode())
            current.data = item.value
        return tree

This code does the job nicely and allows me to use the same function to build a tree from several different views without duplicating code.

Getting a subtree

To get all the nodes which are underneath a specific node, I implemented the view’s reduce function as follows:

    function(doc) { 
        for (var i in doc.path) { 
            emit([doc.path[i], doc.path], doc) 
        } 
    }

Again, this is pretty simple. The only difference from the last view is that I can now query this view with a startkey and endkey (see the CouchDB HttpViewApi) to get only nodes under a certain node. I could actually do that with the previous view, except I’d have to include the full path to the node in my startkey, which is a bit too much.

For example, if you had CouchDB running on your machine right now with my example data loaded and went to http://localhost:5984/tree/_view/tree/descendants?startkey=[“Fruit”]&endkey=[“Fruit”,{}]

How Many Descendants

Getting the number of descendants for a given node is simple. The view is as follows:

    'descendant_count': {
        'map':    'function(doc) { for (var i in doc.path) { emit(doc.path[i], 1) } }',
        'reduce': 'function(keys, values) { return sum(values) }'
    }

This will count the parent node as well, so you will probably want to subtract one from it at some point. To use this view simply call it with the key parameter set to the id of the desired root node.

Getting the immediate children of a node

Sometimes you just want to get a list of nodes which are immediately under a given node. This can be done by using a map with the following map function:

    function(doc) { 
        emit([doc.path.slice(-2,-1)[0], doc.path], doc) 
    }

This map function simply takes the second-to-last element from the path and uses that as the first element in the key. You can query this view in the same way as the “getting a subtree” view above.

Adding a node

Adding a node to the tree is fairly simple. Set the new node’s path to be the path of the desired parent node with the new node’s ID appended to the end. That’s it.

Deleting a node

Deleting a node is a bit trickier since any given node may have some number of children. You can get the list of nodes in the subtree as outlined above and then do a bulk update to delete each of them.

Depending on the data being stored, deleting the whole sub-tree might not ever be something you want to do, in a discussion forum, for example, you might want to simply delete a single offensive post, leaving any replies which might have been posted. Even in this case, it’s more likely that you’d want to set a flag indicating the deletion rather than actually deleting the post.

Moving a node to another parent

This is an instance where being able to update just certain fields in a document would be handly, since bulk-updating a large chunk of documents could start to kill performance.

Either way, if something needs to be reparented, it’s just a matter of getting all nodes which are children of a certain node, then doing a bulk update to change their paths to wherever they need to be.

This part worries me a bit, because there’s a chance that somebody else could add a new child node while you are in the process of moving the sub-tree, leaving that new node dangling by itself in a sub-tree which no longer exists. I’m not sure of the best approach to avoid such a problem.

Conclusion

After my initial experimentation, it seems that CouchDB could potentially do a good job handling hierarchical data. It’s simpler to understand and implement than Modified Preorder Tree Traversal, but still has the advantage of being able to get a whole tree in a single query, unlike the Adjacency List model.

I wrote some python code to load in my test data and query the various views I created. It requires CouchDB-Python, which can be gotten via that link or from EasyInstall by running easy_install CouchDB.

My code can be found in the appropriate spot on my Gitweb.

Have some questions? Have an idea for a better way of storing hierarchical data in CouchDB? Any other comments? Then leave a comment below!