<![CDATA[Pig, satisfied]]> 2013-03-24T21:49:02+00:00 http://benbc.github.com/ Octopress <![CDATA[Visualizing CloudFormation templates]]> 2013-03-24T19:45:00+00:00 http://benbc.github.com/blog/2013/03/24/visualizing-cloudformation-templates I’ve been using CloudFormation a lot recently to manage AWS resources. I’m impressed with a lot of the functionality it provides and it has allowed us to delete a lot of code on my current project which was needed to handle edge cases in provisioning AWS resources (eventual consistency issues, for example).

After a couple of months using it seriously, I only have four complaints about CloudFormation:

  • Stack lifecycle operations (create/update and delete) are not idempotent.
  • Notification topics can only be set up at stack-creation time, so if the topic gets accidentally deleted there is no way to get notifications from the stack.
  • Creation and modification of resources in the stack is serialized, which makes operations on large stacks very slow.
  • The template syntax is horrible: hard to write and even harder to read.

I can’t claim to have a better solution to the syntax problem. Given the constraints, the current syntax is probably reasonable; and the language itself is extremely well thought-out. Perhaps the long-term solution is to treat the JSON as an intermediary form and write bindings for individual languages to generate it.

However, as things stand, once stacks have more than a couple of resources in them, I find it hard to keep track of what is going on. So I’ve written a tool to vizualize templates. It reads template JSON and spits out Graphviz dot format which can be used to generate an image.

The examples in this post are from the sample templates provided by AWS. You can find generated images for all those samples here.

I’d love to hear from you if you think this is useful.

]]> <![CDATA[A thin layer over Enlive]]> 2013-02-09T19:10:00+00:00 http://benbc.github.com/blog/2013/02/09/a-thin-layer-over-enlive In my last post I showed a simple use of Enlive to create a web application with a common layout for all pages. I resisted the temptation to introduce any abstractions because I wanted to make it absolutely clear how to use the building blocks that Enlive provides.

But my fingers were itching the whole time to abstract away some of the wrinkles and I couldn’t let it rest until I’d had a play to see what it looks like. So here is a very thin layer over Enlive that manifests some of the structure that I saw.

Here’s the code we ended up with last time. There is a layout template that is used to wrap the two pages (index and show).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

(def things ["one" "two" "three" "four"])

(defn extract-body [html]
  (at html [#{:html :body}] unwrap))

(deftemplate layout "layout.html" [title content]
  [#{:title :h1}] (content title)
  [:div.content] (substitute (extract-body content)))

(defn show [things]
  (at (html-resource "show.html")
             [:li] (clone-for [thing things] (content thing))))

(def index (html-resource "index.html"))

(defroutes app
  (GET "/" [] (layout "Front page" index))
  (GET "/show" [] (layout "Show things" (show things))))

I’d like to find a way to clarify the page and layout concepts in the code. I’ll use macros to define them, so that I get the define-and-assign style effect which I think is appropriate for important entities like this.

Layouts first. There is some slightly tricky stuff here, stripping off unwanted <html> and <body> tags, which we can hide. Let’s assume that every layout will have one main piece of content, which we can define by convention will go into div.content; but we’ll allow each template to define other slots and substitutions for them.

So we want the layout definition for our example above to look like this.

1
2

(deflayout layout "layout.html" [title]
  [#{:title :h1}] (enlive/content title))

This defines a function (layout) which takes a single argument (title) and returns an Enlive template—another function that takes the page’s content and returns the rendered HTML. We’re effectively currying the layout function defined in the raw example because we want to specify title and content at different points.

Here is how we would use it, without changing anything else.

1
2
3

(defroutes app
  (GET "/" [] ((layout "Front page") index))
  (GET "/show" [] ((layout "Show things") (show things))))

And here’s what deflayout looks like.

1
2
3
4
5

(defmacro deflayout [name source args & forms]
  `(defn ~name ~args
     (enlive/template ~source [content#]
                      [:div.content] (enlive/substitute (extract-body content#))
                      ~@forms)))

And now pages. There are two different kinds of page: dynamic ones, like show, which take one or more parameters and transform their HTML and static ones, like index, with neither parameters nor transformations. We’d like to make defining and calling both types simple. We’d also like to specify the layout when defining the page and hide its application. Here’s what we’re aiming for.

1
2
3
4

(defpage show "show.html" (layout "Show things") [things]
  [:li] (enlive/clone-for [thing things] (enlive/content thing)))

(defpage index "index.html" (layout "Front page"))

And here’s the definition of defpage.

1
2
3
4
5
6
7
8

(defmacro defpage
  ([name source layout]
     `(def ~name
        (~layout (enlive/html-resource ~source))))
  ([name source layout args & forms]
     `(defn ~name ~args
        (~layout (enlive/at (enlive/html-resource ~source)
                            ~@forms)))))

This creates functions for parameterized pages and simple values for static ones to simplify calling them.

1
2
3
4

(defroutes app
  (GET "/" [] index)
  (GET "/show" [] (show things))
  (not-found "Not Found"))

You can see the complete code, in a working project, here.

This refactoring is obviously overkill for a tiny example like this, but something like it could be useful on a real project. It meets the two objectives, anyway: clarifying the important concepts and hiding some of the fiddly details.

]]> <![CDATA[A simple Enlive example]]> 2013-01-27T22:17:00+00:00 http://benbc.github.com/blog/2013/01/27/a-simple-enlive-example Enlive is a Clojure library for generating HTML that uses transformations instead of templates. Rather than starting with templates containing code which are then executed to produce the final output, it starts with plain HTML which is subjected to a series of transformations; the transformations are ordinary functions, targeted to the right part of the DOM by standard CSS selectors.

This approach allows a clean separation between template and code; it avoids creating a novel hybrid language—often crippled and always nasty—for the templates.

As you can tell, I am pretty sold on the benefits of selector-based templating, so I was excited to come across Enlive when I started playing with Clojure. Enlive is powerful and somewhat complex; there are two good tutorials but I found it hard to get straight in my head which bits were really necessary to do something simple.

So, having wrestled with this for a while (and having surface after a longish dive in the Enlive source code), I’ve put together a basic example, which I think takes a reasonable, minimal approach to using Enlive for a simple web application. This post walks through it, explaining how it fits together. It is not intended to be a complete introduction to what Enlive can do; for that, read the tutorials and documentation.

When using Enlive, it’s helpful to know that the library generally uses an internal data structure representation of HTML (called nodes in the documentation). Most functions and macros operate on and return nodes, or collections of them; a few return strings, which you need in order to render the HTML to the outside world (in my case when defining Compojure routes).

The simplest thing we can do is read in an HTML file from disk with html-resource.

1

(def index (html-resource "index.html"))

This returns nodes, which can be rendered into strings with emit*.

1
2

(defroutes app
  (GET "/" [] (emit* index)))

We would also like to be able to show dynamic content. With a transformation-based approach, we start with static HTML and apply transformations to it. We’re going to create a <ul>, with one <li> for each item in a list. The initial HTML shows the structure we want, with dummy content in the <li>.

1
2
3

<ul>
  <li><!-- item --></li>
</ul>

Again we read in the HTML using html-resource, but this time we use at to apply a transformation.

1
2
3

(defn show [things]
  (at (html-resource "show.html")
      [:li] (clone-for [thing things] (content thing))))

This targets the <li> element and applies a transformation which, for every element in the argument list things, clones the <li> and fills in its content.

Like html-resource, at returns nodes, so we can render the result with emit*.

1
2
3
4
5

(def things ["one" "two" "three" "four"])

(defroutes app
  (GET "/" [] (emit* index))
  (GET "/show" [] (emit* (show things))))

(Enlive has a defsnippet macro which we could use here, but it doesn’t work in quite the way that we need.)

For a real web application, we need a way of applying common layout and styling to the pages. With Enlive we can use a further transformation to insert the pages that we have so far into a common piece of wrapper HTML.

1
2
3
4
5
6
7
8
9
10
11
12

<html>
  <head>
    <style type="text/css">
      body { font-family: sans-serif }
    </style>
  </head>
  <body>
    <div class="content">
      Content goes here.
    </div>
  </body>
</html>

Enlive provides a macro deftemplate that we can use here. It wraps a couple of the things we’ve seen already (at and html-resource) as well as doing the function definition for us.

1
2

(deftemplate layout "layout.html" [content]
  [:div.content] (substitute content))

This replaces the contents of div.content with the HTML that we pass it. deftemplate also handles the call to emit* for us, so replace it with layout when defining our routes.

1
2
3

(defroutes app
  (GET "/" [] (layout index))
  (GET "/show" [] (layout (show things))))

We can improve layout to apply consistent titles to each page, by adding an argument and setting the content of <title> and <h1> tags.

1
2
3
4
5
6
7

(deftemplate layout "layout.html" [title content]
  [#{:title :h1}] (content title)
  [:div.content] (substitute content))

(defroutes app
  (GET "/" [] (layout "Front page" index))
  (GET "/show" [] (layout "Show things" (show things))))

And we add those elements, with placeholders, to the layout HTML.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

<html>
  <head>
    <title>Title goes here</title>
    <style type="text/css">
      body { font-family: sans-serif }
    </style>
  </head>
  <body>
    <h1>Title goes here</h1>
    <div class="content">
      Content goes here.
    </div>
  </body>
</html>

There is one wrinkle that I’ve glossed over up to this point. Enlive uses the wonderful TagSoup to parse HTML (so that you can use the same transformations on “wild” HTML that you’ve scraped from the web and which may not be well-formed). Because TagSoup assumes that it’s dealing with something potentially toxic, and tries to detoxify it, it adds <head> and <body> tags around any HTML that you give it. So the DOM returned by html-resource has these extra tags which end up wrapping the content in the middle of our webpage. We need to modify layout to strip these out.

Here is the complete code, which you can see in a working project here.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

(def things ["one" "two" "three" "four"])

(defn extract-body [html]
  (at html [#{:html :body}] unwrap))

(deftemplate layout "layout.html" [title content]
  [#{:title :h1}] (content title)
  [:div.content] (substitute (extract-body content)))

(defn show [things]
  (at (html-resource "show.html")
             [:li] (clone-for [thing things] (content thing))))

(def index (html-resource "index.html"))

(defroutes app
  (GET "/" [] (layout "Front page" index))
  (GET "/show" [] (layout "Show things" (show things))))
]]> <![CDATA[Why software development methodologies rock]]> 2012-08-01T20:24:00+01:00 http://benbc.github.com/blog/2012/08/01/why-software-development-methodologies-rock (This post was insprired by a sneak preview of a blog post by Jez Humble.)

Methodologies or other defined practices can have value in so far as they cause people to reflect on and criticise what they are doing. Often it’s the adoption of a new approach that brings benefits, rather than the approach itself.

This is why consultants are all convinced that their approach is the one-true-way. They tend to meet teams that have stopped reflecting and so are in trouble. They introduce change and see that everything magically starts to improve. A spot of attributional bias and they’re convinced.

The subjects of their advice, however, see that things get worse again after a while (because they’ve stopped reflecting). So they conclude that methodology X is all very well, but in practice it doesn’t work in the long term.

It takes a special kind of person and team to keep reflecting even when there is no change agent. People who can do this without any prompting are gold dust and don’t need a methodology. People who can nearly do it benefit from any methodology that they can be convinced by, because the existence of an ideal keeps them thinking about how they are deviating from it (and it’s the thinking that matters, not the extent of the deviation).

My experience has informed my opinion. My first job was for a small software shop that followed a rigorous waterfall approach: strong hierarchy, lots of documentation, estimation-by-loc-guessing, separate development phases. They were extremely successful and repeatedly delivered on-time and within-budget; I was miserable. My second job was as a consultant (with ThoughtWorks).

]]>