ElixirConf US 2020

On September 3-4 the 2020 edition of ElixirConf US took place and six members of the software development team at Codegram had the chance to take part in it. As it was to expect with the COVID-19 pandemic still raging, the conference was completely remote. Since we are all based in Europe that was quite fortunate, as it would have not been feasible for all of us to attent the event on-site. All the talks were streamed live on a schedule based on the US CT time zone. That made it happen at a bit of a late time for us, but nothing that could not be overcome by shifting our schedules for a couple of days.

Elixir

Elixir is an interesting piece of technology mainly because of 2 reasons. The language itself tends to follow the functional paradigm, which in the long term has many advantages with regards to the readability and maintainability of the code. In addition, it runs on top of Erlang's OTP which is really good at both taking advantage of execution concurrency and helping developers built fault tolerant systems.

At Codegram, we have been usingElixirin some of our projects since as early as 2016, but it wasn't until lately that we started using it in our client projects. During these past few years its ecosystem has grown bigger and bigger, to the point that we're comfortable making the switch from Rails - especially now, with the amazing power that libraries such as LiveView put in our hands.

The hallway experience

Besides the talks themselves, there were indeed some activities for the attendants to get together and somewhat alleviate the lack of direct human interaction that one would otherwise get from a face-to-face experience. They were basically Zoom calls that had an allotted slot in the schedule, be it at lunch time or as an after party when all the talks were over.

To be fair, none of us attended any of these events. I believe there were two main reasons for this.

First of all, it was simply a matter of the events taking place on US CT time and us being based in Europe, on CET time, that is, 7 hours later. The lunch break did not clearly align with our dinner times and the after party would take place far too late (around 11PM) for us to be in the social mood.

But maybe a more significant reason for us not to be there was the fact that most of us were fairly new to the Elixir ecosystem and therefore not really familiar with its community, let alone the one across the pond. The natural shyness that one would have felt in an on-site setting was therefore exacerbated by the fact that this was a remote event. It is kind of difficult to pinpoint what would have helped encourage the members of our team to join those events. Maybe we could have simply put more from our part 🙂, but it is also true that it would have helped if the organisation was clearer on what to expect from them, or maybe if they had directed the attendands straight to them just at the end of the preceding talks.

We did have some sort of community experience, though. We were all connected through Codegram's Slack channels and kept an ongoing conversation on the talks as they were happening. This did indeed help to stay in the conference mood, so to speak, and dedicate the time and the attention that these talks deserve. Having simply watched their recordings individually on our own time would have not worked as well.

Some technical hiccups

The conference took place over the EventVivo platform. By just having a look at the plaform's site it's hard not to notice that it seems not to have had a long history in hosting remote events. In fact, ElixirConf 2020 may have even been the first time in which this plaform was used in production. And it showed a little bit.

For example, as soon as all the attendants connected to the platform at the expected time, the service went down with the dreaded 502 HTTP error (a timeout from the MUX downstream service, maybe?). A link for a Zoom call was then shared on the organization's Twitter account but, at is seems, it was not possible to prevent external users to connect to it. The start of the conference had thus to be postponed a couple of hours.

Later on, once the talks had started, some of us experienced frequent video freezes and if your stream was playing with a delay the talks would appear to shut off abruptly before the end of the actual talk.

Some notes on the talks

Keynote (Chris McCord)

If it wasn't for the fact that this was a remote conference, we would surely have heard the audience's jaws dropping as Chris McCord presented the latest milestone on the LiveView framework: Uploads! Interactive uploads have always been a hard problem to solve on the web (needing to resort to bloated JS libraries and external providers) - but thanks to LiveView and the Phoenix framework, that's now a thing of the past.

Thanks to the close feedback loop between the server and the client enabled by LiveView, state-of-the-art uploads including multi-file, progress indicator, drag-and-drop and validations are achievable with just a handful of lines. And what's most important: the abstractions behind it are really sound, so its behavior is really easy to customize to your needs.

Unfortunately, LiveView Uploads are still not publicly available, but Chris published a sneak peek not so long ago on Twitter. Exciting!

Keynote (José Valim)

As one would expect - José being the creator of the Elixir language - his keynote was about what's to come on the upcoming Elixir version (1.11). As he stated in several ocasions, this version isn't the most flashy one in terms of features but instead, the Elixir core developers wanted to focus on stability and developer experience.

The most remarkable improvements are:

• Better Erlang interop: 4 new log levels are made available to Elixir, and IEx will now show erlang documentation by using the outstanding h() helper.
• The compiler becomes more intelligent, and able to detect any bad usage of maps or structs within function bodies by inspecting the arguments - so compilation will fail if you're trying to access a key that doesn't exist. Useful!
• Application boundaries: We've been bitten by this in the past - an Elixir app can itself boot many other apps such as :crypto, erlang's cryptography library. If one forgets to include them as dependencies though, function calls such as :crypto.hash(...) would fail on runtime. Elixir will now warn you about this on compile time. Neat!
• Config runtime: Configuration mishaps (especially on production) are usual in Elixir, as configuration gets compiled down with the code, which causes all kinds of headaches if you're expecting your app's behavior to change after some ENV variable changed. Elixir comes with Releases built-in that palliates this, but it looks like the core team is looking into a more powerful approach, making things as runtime configuration swap possible.
• New APIs: Although the focus on this version wasn't new features, some new APIs were added such as is_struct guards, map.field guards and Calendar.strftime, among others.
• And last but not least, Elixir now comes with an integrated test coverage tool via a mix test.coverage command. Useful!

If you're curious, don't forget to check out the official changelog.

Contributing to Elixir and the ecosystem

Sometimes when we go to a conference we just expect to attend technical talks and we forget that non technical talks can be great and inspiring as well, that was the case with the talk given by Leandro Pereira, where he talked about how we can contribute to the Elixir ecosystem.

He told us how to write documentation or even how fixing typos is a good first step to start contributing. Adding tests to code with low coverage is another good way to start contributing. He also showed us that making mistakes is part of the process and we should not be afraid of asking for help.

Metaprogramming with Elixir

There were two talks about metaprogramming and macros that were very profitable for someone new to Elixir but with Ruby experience like many of us.

The first talk was Nicholas Henry's "The Upside Down Dimension of Elixir - An Introduction to Metaprogramming" and was a very clear explanation of the differences between these two worlds, and can be summarized with the following table:

Macros are defined using defmacro/2 and will receive and return code in the form of AST (Abstract syntax trees). These ASTs are represented by Elixir tuples with three elements — marker, metadata and children — that can contain more AST to create a tree structure. To ease the handling of this structure we have the quote/1 and unquote/1 macros. Here we have some examples of the use of quote to create ASTs from code and unquote to execute code with the variables defined at compile-time.

iex(1)> a = 1
1

iex(2)> quote do
...(2)>   a = 1
...(2)> end
{:=, [], [{:a, [], Elixir}, 1]}

iex(3)> quote do
...(3)>   unquote(a) * a
...(3)> end
{:*, [context: Elixir, import: Kernel], [1, {:a, [], Elixir}]}

When comparing to Ruby metaprogramming, the compiled nature of Elixir results on a clearer separation between both worlds and a much better performance. The only possible drawback could be some loss of flexibility, but I would need to work more with it to confirm this.

The other related talk was called "Deep dive into Elixir" by Jonathan Yeong. What initially seemed more like a "getting your feet wet" than a "deep dive" turned out to be one of the conference highlights. Well conducted, it drove us to the end of the rabbit hole until we understood how the pipe operator works.

This operator is implemented with a macro, which basically transform the input code into nested calls to the given functions.

Let's see how this works on very simple example with two nested calls to the functions func1 and func2.

var
|> func1
|> func2

To begin, the macro receives in AST format the left variable with the input to work with and the right variable with the tree of nested calls.

left = {:var, [], Elixir}

right = {:|>, [context: Elixir, import: Kernel],
          [{:func1, [], Elixir}, {:func2, [], Elixir}]
        }

Then it uses them to build and return an AST tree with the nested calls in inverted order, removing all the references to the |> operator.

{:func2, [], [
  {:func1, [], [
    {:var, [], Elixir}
   ]}
 ]}

Finally, the resulting AST tree is equivalent to the one that we would get if we write the following code.

func2(
  func1(
    var
  )
)

The talk left us inspired and with some questions:

• Could we write a version of the pipe operator that pipes to the last argumeent of the function call?
• Could we write a version of the operator where we manually specify where to use the argument?

Turns out, there are at least two (!) different libraries implementing our second question: magritte and CapturePipe. Still, we might try to develop our version for it, just to try it!

Domain-Driven Design with Elixir

Japa Swadia talked about Domain-Driven Design with Elixir, in this very interesting talk she explained how DDD is geared towards organization and structure rather than architecture, also, she told us the importance of finding clear boundaries between business logic. Japa showed us the wins of adopting DDD as faster iterations, easy collaboration, maintainable codebase, and better complexity management with clearly defined boundaries. Also, she showed that when the technical complexity of the application is higher than the business logic adopting DDD might not help. But in general, using DDD can give us a more reusable, maintainable, and readable codebase.

UintSet: enumerable, streamable, understandable

I did not know him (which does not say much), but Luciano Ramalho is a household name in the Python community, particularly in Brazil. In 2015 he published the successful Fluent Python (O'Reilly) and currently works at ThoughtWorks. As it was evident last week, his interests go beyond the language of the snake, and gave us an interesting talk on how to leverage the protocol mechanism in Elixir in order to create new and custom implementations.

The talk was centered around the Set data structure and started by noticing the apparent mismatch between its corresponding mathematical notion and the imperative orientation of the Von Neumann architechture of the machines that the vast majority of us develop for. Along these lines, he mentioned the stark contrast betwen the operations on sets that most standard libraries offer (e.g. member?, put, delete, size, ... ) and the common ones for their mathematical counterparts (e.g. intersection, union, difference, subset, ... ).

He then moved onto showing an example on how to create a new restricted set type of unsigned integers in Elixir, which uses an internal representation that make the latter list of operations more performant and streamlined. The module definition has a single Integer value for the internal representation of its members:

defmodule NaturalSet do
  # ...
	defstruct bits: 0
  # ...
end

The idea would be to interpret the binary representation of the internal integer value so that if the natural number at position n belongs to the set, then that digit will be a 1 and it will be a 0 otherwise. For example, let:

A = { 1, 3, 4, 6 } ⊆ ℕ

then the value of bits would be

1011010 = 90

Admittedly, this representation does initially make some of the set operations a bit more difficult to reason about, for example, in Elixir the addition of a new element to the set would be implemented as follows:

def put(%NaturalSet{bits: bits}, element) when is_integer(element) and element >= 0 do
 %NaturalSet{bits: 1 <<< element ||| bits}
end

As you can see, 1 <<< element shifts 1element positions to the left to represent its membership to the set and the result is applied the bitwise OR operation (|||) so that the previous ocurrences of 1 in bits are preserved.

However, this representation maks some of the expected operations less performant. For example, the convention in Elixir is for size/1 to return its value in constant time, but this is not possible in our case.

Nevertheless, as mentioned before, it does make some common set operations trivial, as they are reduced to simple bitwise transformations. For example:

def equal?(%NaturalSet{bits: b1}, %NaturalSet{bits: b2}) do
	b1 === b2
end

def intersection(%NaturalSet{bits: b1}, %NaturalSet{bits: b2}) do
	b1 &&& b2
end

def union(%NaturalSet{bits: b1}, %NaturalSet{bits: b2}) do
	b1 ^^^ b2
end

The second part of the talk then shifted to the exposition of what protocols are in Elixir and how this new set type could implement some of them: Collectable, Enumerable, Inspect, String.Chars, etc.

All in all, it was refreshing to see a talk that tackled the more abstract and fundamental parts of programming, as all too often they tend to be overlooked in favour of more pragmatic topics.

You can find the slides here, and they even include a photograph of Cantor! What is not to like? 🙂

It's Alive! Instrumenting Phoenix 1.5 with Telemetry and LiveDashboard

Sophie DeBenedetto explained how telemetry works in Elixir and the tools that Phoenix and Ecto offer to facilitate our work in this regard.

Telemetry and Observability

Observability is the ability to understand what's happening inside a system by examining its output. We can instrument our code, watch it deploy, and answer the question: "Is it doing what I expect?". We need some tools to inspect our project's behaviors, and here is when telemetry appears. Telemetry is an open-source suite of libraries that standardizes how apps on the BEAM are instrumented and monitored.

How does it work?

An event indicates that something happened, these events can have many handlers attached, performing a specific action when the event is published. And that's all, an event happens, Telemetry knows about it and does something.

Building a Telemetry pipeline

To build our Telemetry pipeline we have to:

1. Attach the event to the handler. Events are stored in ETS, along with a reference to their handler functions.
2. Execute the event. When an event happens, it looks up the event in ETS and calls the handler function.
3. Handle the event. Define a handler function to be invoked on event execution

It's pretty easy but it's not scalable. We have to write a lot of code if we have a lot of events.

Telemetry in Phoenix 1.5

Phoenix and Ecto execute Telemetry events out-of-the-box. Phoenix 1.5 ships with a Telemetry.Metrics module. Here we will define which Telemetry events we want to handle and how. The module measures events and reports them to a reporter. The reporter attaches events to handlers in ETS and handles events when they are executed. In Phoenix 1.5 this reporter is LiveDashboard.

1. We define our Telemetry metrics in the metrics module
2. We deliver these metrics to LiveDashboard
3. LiveDashboard stores the events in ETS
4. LiveDashboard handles the events when they are executed

With few work we have our Telemetry metrics rendered in live-time thanks to LiveDashboard!

Phoenix + Ecto's OOTB Telemetry Events

Finally, Phoenix and Ecto come with some predefined Telemetry events that help us observe the behavior of our project.

Using Phoenix out-of-the-box telemetry events we can observe router events, error events, socket events, channel events, etc.

Using Ecto out-of-the-box we can observe the metrics of all our queries.

It was an interesting talk to understand how metrics work. Thanks to Ecto and Phoenix we can have absolute control over what happens in our project without much effort. That's really awesome!

Last remarks

Attending Elixir Conf 2020 was such a rewarding team experience for us. Event if we missed some of the conference experience (travel, food, getting to know new people), the excitement of learning new stuff and sharing that with your partners was still there. We will surely consider attending more remote conferences in the future!

Despite being attributed to Armand Adroher, this blog post was coauthored with Edgar Latorre, Leo Diez, Arnau Besora and Josep Jaume, as well.

Cover photo by Daiga Ellaby on Unsplash