Engineering

Blacksmith is a data generation framework for Elixir which helps you to create sample data in tests. If you are a Ruby developer, you can think of it as an equivalent to factory_girl. I will explain a part of its functionality within the post. For further information about the package you should read its README as well as this blog post written by its creator. I’ve chosen this library for my code reading article because it makes use of OTP and metaprogramming while being short enough to grasp it completely (at the time of writing, the whole package consists of more than 200 LOC). Since no version was released yet, I will refer to a specific commit within its git repository.

Installation

If you want to try the code samples in this post yourself, you can grab the source code from Github; make sure to refer to the same commit as I do:

git clone git@github.com:batate/blacksmith.git
git checkout 5dbd9c83c4715b8234494ada7c113fab11dacb0f
mix deps.get
mix test
→ 9 tests, 0 failures

Code analysis: OTP Structure

I will start analysing the OTP structure of the package by looking into which processes are spawned. This helps me understand which parts are done concurrently. Erlang comes with a graphical tool called observer that allows to look into a running erlang system and displays its supervisor trees, processes and more. You can start it from iex:

iex -S mix

Interactive Elixir (1.0.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> :observer.start

In the Applications tab you can see that the blacksmith application started one process named “Elixir.Blacksmith.Sequence”.

The Processes gives more detailed information about the processes. Here we can see that the Sequence process is a gen_server process.

You can find the complete source code of the Sequence Server under lib/blacksmith/sequence.ex, the following gist shows an excerpt:

defmodule Blacksmith.Sequence do
  def start_link, do: Agent.start_link(&HashDict.new/0, name: __MODULE__)

  @doc "Generate the default sequence (:default)"
  def next, do: next(:default)

  @doc "Generate a named sequence"
  def next(name) do
    Agent.get_and_update __MODULE__, fn(seqs) ->
      current = HashDict.get(seqs, name, 0)
      next    = HashDict.put(seqs, name, current + 1)
      {current, next}
    end
  end
end

The start_link function starts an Agent process under the name Blacksmith.Sequence that holds an empty HashDict as its internal state. This is the process we saw in the observer, but as Agent is just an abstraction of GenServer, the Erlang observer displayed as a gen_server server process. The next/1 function takes a sequence name, returns the current value of the sequence and increases the stored value. As a convenience, a next/0 function is added which refers to a sequence named :default.

Knowing that the sequence runs under a named Agent process, we can use Agent.get/2 to look into the internal state of the process in iex:

iex -S mix

Interactive Elixir (1.0.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Blacksmith.Sequence.next
0
iex(2)> Blacksmith.Sequence.next(:default)
1
iex(3)> Blacksmith.Sequence.next(:foo)
0
iex(4)> Agent.get(Blacksmith.Sequence, &(&1))
#HashDict<[foo: 1, default: 2]>

Code Analysis: The core functionality

The core functionality of blacksmith is to build records with default values. In order to play with the basic usage, add a file “forge.exs” in the project’s root folder and compile it in iex:

defmodule Forge do
  use Blacksmith

  register :user,
    id: Sequence.next(:id),
    name: "José Valim"
end

iex -S mix

Interactive Elixir (1.0.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> c "forge.exs"
[Forge]
iex(2)> Forge.user
%{id: 2, name: "José Valim"}
iex(3)> Forge.user(name: "Bruce Tate")
%{id: 3, name: "Bruce Tate"}

Let’s have a look into how this is done internally. In order to do so, I will again use only an excerpt of the source code. I omitted quite a lot and also changed the implementation of new/4 as I don’t need some functionality for the example. You can find the complete code at lib/blacksmith.ex.

defmodule Blacksmith do
  defmacro __using__(_) do
    quote do
      import Blacksmith
      alias Blacksmith.Sequence

      @new_function &Blacksmith.new/4
    end
  end

  defmacro register(name, opts \\ [], fields) do
    quote do
      def unquote(name)(overrides \\ %{}, havings \\ %{}) do
        @new_function.(unquote(fields),
                       Dict.merge(overrides, havings),
                       __MODULE__,
                       unquote(opts))
      end
    end
  end

  def new(attributes, overrides, _module, _opts) do
    %{}
    |> Map.merge(to_map(attributes))
    |> Map.merge(to_map(overrides))
  end

  def to_map(list) when is_list(list),
    do: Enum.into(list, %{})
  def to_map(map) when is_map(map),
    do: map
end

The module consists of three parts:

• The using Macro enables the including module to call the register Macro directly in its source code and sets default value for the module attribute @new_function.
• The register Macro adds a method to the calling macro each time it is called.
• The new and to_map are normal functions of the Blacksmith module

The most important functionality happens in the register Macro. In order to analyze what it does exactly, Macro.to_string/2 is a great helper. This allows to look at the source code for an abstract syntax tree that is returned by the quote block of the macro. I will alter the source code of the macro and save the result of the block in a variable called ast and output the code representation before returning the ast:

defmacro register(name, opts \\ [], fields) do
  ast = quote do
    def unquote(name)(overrides \\ %{}, havings \\ %{}) do
      @new_function.(unquote(fields),
      Dict.merge(overrides, havings),
      __MODULE__,
      unquote(opts))
    end
  end
  IO.puts Macro.to_string(ast)
  ast
end

After recompiling the code, I can now see the result of the register :user call in our Forge:

iex -S mix

Interactive Elixir (1.0.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> c "forge.exs"
def(user(overrides \\ %{}, havings \\ %{})) do
  @new_function.([id: Sequence.next(:id), name: "José Valim"], Dict.merge(overrides, havings), __MODULE__, [])
end
[Forge]

The macro generates a user method within the Forge module which has the default values saved in its function body. You can also see that it calls the anonymous method that is stored in @new_function that was set to &Blacksmith.new/4 as a default. The reason for this indirection is that you could overwrite @new_function before calling register. The_Blacksmith.new_ now does the real work of building the record by merging the default values with the overriding values.

What I left out

As previously mentioned, I just used excerpts from the real source code, so feel free to dig into the other functionality:

• Saving the generated records in a repository
• Generating lists of records
• Using common data elements through the having macro (which would be worth a blog post of its own)

Implementation alternatives

Whenever you read someone else’s source code, you find things you would have done differently. The one thing I had expected to be implemented differently was the method generation. As an alternative to generating a method per register call, you could also generate different implementations of the same method, a pattern that is used in the Unicode handling of Elixir itself and makes use of pattern matching of the Erlang VM. The interface would then change:

# current implementation
Forge.user(overrides)
# alternative with pattern matching
Forge.new(:user, overrides)

Further readings

I hope you liked my approach of delving into the internals of blacksmith and learned as much as I did along the way. If you want to continue on this path, I suggest the following sources:

• Metaprogramming Elixir by Chris McCord explains the Elixir metaprogramming possibilities far more detailed than I did here.
• Xavier Shay’s blog series “code safari” was my inspiration of writing this kind of article.
• The ruby rogues also did some code reading and recorded it: http://youtu.be/lKmY_0uY86s.