El otro día una amiga me preguntó que era un algoritmo y me acordé la definición que me dieron en la facultad: “es una secuencia de pasos a ejecutar en orden para resolver un problema”. Esta misma tiene una trampa que me gusta hacer y consiste en no mencionar la computadora, porque en el fondo una y otra cosa no tienen nada que ver. Los algoritmos existen desde mucho antes que las computadoras. Por ejemplo, hacer café es un algoritmo:

1. Calentar agua hasta que llegue a los 80 grados centígrados
2. Poner 22 gramos de café molido por taza de agua en la prensa
3. Verter el agua caliente en la misma
4. Esperar un minuto
5. Prensar
6. Servir

Multiplicar, como lo aprendemos en la primaria, también es un algoritmo y uno interesante ya que hay varios: existen muchas maneras de multiplicar dos números que llevan a la misma respuesta. Algunos son más fáciles, otros más veloces, o como hacían apenas 3 décadas antes en el colegio nuestrxs progenitores, usando tablas de logaritmos. Además, es un área de investigación activa: el año pasado inventaron uno nuevo.

Cocinar es algorítmico, una receta es seguir una secuencia de pasos y hasta podríamos aplicar técnicas algorítmicas a la cocina: imaginense que las mismas digan, por ejemplo, cómo crece la complejidad con respecto a la cantidad de personas, porque son distintos los problemas de hacer un asado que los de hacer un guiso para 20 personas. Y hasta tenemos loops: “revolver hasta que …” y condicionales: “si tenemos … entonces …”.

Otro, muy antiguo: se dice que en el siglo IX durante el califato abasí se encriptaban los registros de los impuestos, por eso quizás el filósofo musulman al-Kindi diseñó un algoritmo para descifrar mensajes encriptados donde las letras estaban intercambiadas.

Mecanización

Pero, es cierto, el verdadero potencial de los algoritmos se empieza a aprovechar con el advenimiento de las computadoras. A fin de cuentas, una computadora no es más que una máquina que puede ejecutar instrucciones ordenadas de manera programable. Generalmente, se dice que la primera computadora aparece en el siglo XIX: la máquina analítica. Pero me parece más interesante hablar sobre el primer programa. ¿Qué es un programa? Pues, es un algoritmo expresado de manera que una máquina –o persona– lo pueda entender.

Vayamos entonces a la historia del primer programa. El primer programa pensado para una máquina fue escrito por Ada Lovelace (a quien pusieron a estudiar matemática porque su padre, Lord Byron, era un romántico sin salvación) en 1843 para la computadora de su amigo, Charles Babbage, que estaba diseñando la susodicha máquina analítica. Lo que resolvía el programa de Lovelace era calcular un número de Bernoulli, muy útil en la época, ya que servía para hacer multiplicaciones entre números muy grandes. El programa es bastante complejo y hasta tiene algo que lo hace, más allá de toda duda, el primer “programa real”: un bug ². Desgraciadamente nunca lo pudieron probar ya que la máquina de Babbage jamás llegó a existir.

Lovelace, por otra parte, nunca resignó su linaje poético. En una carta a su madre le escribe enfurecida: “No me concedes poesía filosófica. ¡Invierte el orden! ¿Me darás filosofía poética, ciencia poética?”. Justamente se concebía como una pionera de la “ciencia poética”, y se dice que su amor por la poesía es en parte lo que la llevó a escribir el primer programa: un claro producto de la intersección entre el Romanticismo y la Revolución Industrial.

Con el tiempo, lo mecánico advino a lo electro-mecánico y finalmente terminamos en la computadora electrónica. Mientras la computadora de Babbage podía multiplicar dos números de 20 cifras en alrededor de 3 minutos, una computadora moderna puede hacerlo en una billonésima de segundo. Este crecimiento exponencial –ahora denominado Ley de Moore– fue observado por primera vez por Gordon Moore, uno de los fundadores de Intel, quien formuló que la cantidad de transistores ³ que entra en una computadora se duplicaría cada dos años. A efectos prácticos esto significó que diseñar algoritmos y software en general se volviese –para una clase grande de problemas– mucho más fácil: si tu programa andaba lento, podías esperar dos años para que, sencillamente por correrlo en una computadora nueva, funcione el doble de rápido.

Vale detenerse a reflexionar igual: el hecho de sacar cada dos años una computadora el doble de rápida que la anterior perpetúa uno de los efectos mas contradictorios del consumismo capitalista: la obsolescencia programada. La obsolescencia programada es una manera de aumentar las ventas de un producto diseñando el mismo de tal manera que sea obsoleto después de un tiempo artificial predeterminado. Para darnos una idea, el primer caso reconocido de obsolescencia programada es nada más y nada menos que las bombitas incandescentes: en 1925 se fundó un cartel de empresas productoras de bombitas que decidieron unánimemente limitar la vida útil de las mismas. Por lo tanto, mientras la Ley de Moore pudo significar que la “productividad” de las computadoras creció exponencialmente, también significó que millones de computadoras terminen en pilas de basura no degradable por siglos a venir.

Para bien o para mal, este proceso parece estar interrumpiéndose y ya no se espera que el crecimiento siga siendo exponencial; se estaría llegando a los límites físicos de cuánto se puede reducir un transistor.

Pero la algoritmización de la vida encuentra nuevas maneras de acelerar: la hiperconectividad de internet, la pervasividad de los smartphones, la acumulación incremental de datos personales y la subsecuente explotación: el algoritmo sigue produciendo, acaparando y no planea detenerse.

Explotación

¿Y de qué maneras los algoritmos modernos nos hacen la vida más fácil? El algoritmo de Google (PageRank), por ejemplo, nos simplifica la tarea de recordar sitios web y encontrar información en internet; EdgeRank de Facebook ordena y filtra inmensas cantidades de contenido de manera tal que nos llegue primero lo que más nos interesa; cientos de algoritmos trabajan en conjunto para que nos mandemos mensajes de texto casi instantáneos desde nuestros bolsillos.

Pero catalogar a estos algoritmos de benignos sería crudamente ingenuo. En su artículo del 2016, Cómo la Tecnología está Secuestrando Nuestra Mente, Tristan Harris desarrolla algunos de los puntos donde los algoritmos del capital nos afectan negativamente: nos hacen creer que tomamos decisiones cuando no es así, nos hacen adictos a la interacción tecnológica, potencian inseguridades sociales o físicas de tal manera que ya es tema de salud pública que las redes sociales aumentan la tasa de suicidios, y quien sabe cuantos mas.

Entonces me pregunto: si la revolución industrial –supuestamente– nos liberaba del trabajo manual, ¿los algoritmos nos deberían liberar del trabajo intelectual? Para pensar esto habría que investigar sobre qué líneas de fuga se plantan estos algoritmos. En el capitalismo estas líneas de fuga están generalmente enfocadas sobre un tipo de extractivismo económico trivial que gira alrededor de la acumulación de capital y esto es más que nunca verdad para los algoritmos.

De la misma manera que la automatización de las fábricas no liberó a la humanidad del trabajo manual, los algoritmos por sí solos no nos liberarán de las injusticias, la desigualdad, el hambre o la explotación de unos por parte de otros. Como dice Tiziana Terranova, “la automatización, desde el punto de vista del capital, debe siempre ser compensada […] con nuevas formas de controlar el tiempo y la energía así liberados. Debe producir pobreza y estrés donde debería existir riqueza y ocio. Debe hacer del trabajo directo la medida de valor aun cuando es evidente que la ciencia, la tecnología y la cooperación social constituyen la fuente de la riqueza producida” ⁴. Dada la realidad material del presente, sus contradicciones y la interminable desigualdad incremental que vivimos, a veces me resulta difícil entender el hype de mis colegas informaticxs por cualquier nuevo algoritmo o tecnología; siempre tan disociados del contexto social en el que se encuentran. Cuando Elon Musk anuncia que llevará 1 millón de personas a Marte antes del 2050, ¿a qué millón de personas se refiere?

Si dejásemos eso de lado, si pudiéramos olvidarnos por un momento del ímpetu capitalista de acumulación y explotación, ¿cómo serían los algoritmos desarrollados por y para personas y organizaciones sociales preocupadas por vivir en un mundo más justo para todxs?

Recuperación

En el libro Algorithms to Live By de alguna manera los autores intentan humanizar el uso práctico de algunos algoritmos, pero no es lo que uno esperaría. En vez de nuevas maneras emancipatorias de organización política y social, nos encontramos con casos aislados donde aplicar resultados estadísticos para problemas cotidianos puede ser más óptimo o eficaz. Como cuándo hay que dejar de buscar a la “pareja perfecta”, usando la regla de la secretaria que dice que en cualquier búsqueda lineal sobre un conjunto inmenso el óptimo existe en parar al haber “entrevistado” al 37% de los sujetos (si querían saber, aplicado a la búsqueda de pareja esto representa dejar de buscar a los 28 años). O que si se compran muchas verduras, la mejor manera de consumirlas es en el orden del vencimiento, excepto si algo está por vencer donde entonces se tiene que tirar “la que más comidas rinda”.

No me mueve tanto la aguja. Lo que me gustaría ver es si es posible tener algoritmos que resuelvan problemas sociales, estructurales y, por qué no, políticos.

Me viene a la mente el Proyecto Cybersyn: el plan maestro del gobierno de Salvador Allende para orquestar la economía del país, la misma ampliamente estatizada como parte de su proyecto socialista. Cybersyn hubiese consistido de una intranet entre empresas estatales que continuamente alimentaba un algoritmo que disparaba alertas y ayudaba a tomar decisiones económicas. El mismo nunca se puso en funcionamiento: el golpe de estado puso fin al sueño Chileno.

Volviendo al presente, en su charla para la CodeMesh 2019, por ejemplo, Veronica Dahl dice de pensar en una inteligencia artificial regenerativa y redistributiva. Imaginar algoritmos de machine learning que sirvan para, por ejemplo, levantar una alarma cuando una publicación científica utiliza una muestra demasiado chica o sesgada de la población. O algo más ambicioso: algoritmos que ayuden a los legisladores a tomar mejores decisiones.

Más allá de la discusión de si esto es factible o no –en la cantidad de años que el sistema judicial pueda tomar en reformarse ¿cuántas nuevas maneras de acumular habrá encontrado el capital?– comparto opinión con Terranova en que seguir desarrollando algoritmos abocados solamente a crear excedentes de tiempo y trabajo para ser apropiados por una minoría, “conduce así inevitablemente a la destrucción periódica y generalizada de la riqueza acumulada, en formas de agotamiento psíquico, catástrofe ambiental y destrucción física de la riqueza por medio de la guerra” ⁵.

¿Es realmente imposible pensar en algoritmos cuyo único propósito no sea acumular? No lo sé, pero creo que la búsqueda tiene el potencial de ser revolucionaria.

Llegué a este post sobre OSMnx y me gustó la idea, pero no me cerraba el gráfico.

Inspirado en eso y probablemente en estar leyendo Visual Explanations de Edward Tufte, se me ocurrió que tenia que haber una manera mas elegante de mostrar la dirección de las calles.

Justamente, las calles en sí son una visualización de su dirección, por lo que mostrando todas las calles “superpuestas” de alguna manera, deberían aparecer patrones.

Para esto, calculo el centroide para cada calle (por su nombre, lo que trae cierto error cuando hay nombres repetidos) y las traslado todas al origen, o sea el (0, 0).

Acá el notebook de jupyter.

Este es el resultado para las ciudades Argentinas:

Y para algunas ciudades del original:

Last year’s November I went to the amazing DuraznoConf, which I must emphasize was an awesome experience, and during my free time there I got to finally finish a project of mine which I had sitting around for too long: wave2blofeld.

Basically the Waldorf Blofeld, a lovely synthesizer I happen to own, has something called WaveTable synthesis which means it has literal wave tables (actually arrays though) that it iterates trough and other synth stuff. In the Blofeld, each wave table has 64 waves, and each one of these waves has 128 samples.

Now the fun part: you can load your own wave tables!

The catch is that there is no nice official way to do it 😬

So in order to maximize compatibility, and because why not, I decided to do it in C++ (or Rust, but the only MIDI lib I found was too beta for me).

wave2blofeld takes a 8192 samples WAV file and spits out to a MIDI file which you can transfer to your Blofeld using SysEx Librarian, for example.

Dependencies

Just run. Run as fast as you can (?)

Dependency management in C++ deserves it’s own dystopic novel. I tried a couple of solutions like Buckaroo but guess what happens when you search for midi.

Essentialy, you are on your own, so after losing more hours than I’d want I wrote a 28 lines Makefile which works like a charm. The thing is, since C++ package management is such a complicated issue, library writers usually end up doing their stuff as easy to build as possible, having no dependencies and sometimes even only header files!

Take for example this line from TCLAP’s home:

The library is implemented entirely in header files making it easy to use and distribute with other software.

Having to build libraries without depending on anything is a serious drawback.

Anyway, for this thing I ended up using:

• midifile for, ehm, MIDI files
• audiofile for you can guess what
• TCLAP for parsing command line options

Building

Last year I ended up having to learn CMake because my algorithms class teachers weren’t really good at it, so I went with it. It’s fairly easy to get started and possibly demential to learn in depth (the manual never ends). The whole thing is only a dozen lines long:

cmake_minimum_required(VERSION 3.0)
project(wave2blofeld)

set(CMAKE_CXX_STANDARD 11)
set(CMAKE_BUILD_TYPE Debug)

link_directories(deps/midifile-master/lib)

include_directories(deps/midifile-master/include)
include_directories(deps/audiofile-master)
include_directories(deps/tclap-1.2.1/include)

file(GLOB SOURCES "src/*.cpp" "deps/audiofile-master/*.cpp")

add_executable(wave2blofeld ${SOURCES})
target_link_libraries(wave2blofeld midifile)

It definitely beats having to write a Makefile 😂. Also, it is kind of readable.

With that out of the way, building is as easy as creating a build folder and inside it running cmake .. followed by make.

Actually programming

Building a MIDI file is not complicated, but a SysEx file … that took me a while.

Basically MIDI has this System Extended message format, which manufacturers can use for anything they like. Of course this means it’s all done in binary.

Thankfully for reference I had Jonas Norling’s page which kindly offers the binary format explained and a reference implementation.

For starters, the Blofeld wants 128 samples of big-endian signed 21-bit integers each, that is integers between -1048575 and 1048575. audiofile loads samples as floating point numbers between -1 and 1, so each loop we normalize every sample and then load it in 7-bit triplets [Edit: Jonas’ post says “21-byte little-endian” numbers, which is weird because 21-bytes is a lot and anyway little-endian didn’t work for me, so I guess that’s a typo].

    for (int i = 0; i < 128; ++i){
        mm[8 + 3*i    ] = (samples[i + wave*128] >> 14) & 0x7f;
        mm[8 + 3*i + 1] = (samples[i + wave*128] >>  7) & 0x7f;
        mm[8 + 3*i + 2] = (samples[i + wave*128]      ) & 0x7f;
    }

Then it’s a matter of following the spec, and building a 410 byte array which is the SysEx message. We need to send a message for each of the 64 waves of the wave table.

Finally, a touch of iterators for adding every byte of the wave, which makes a kind of checksum:

    int checksum = std::accumulate(
        mm.begin()+7, mm.begin()+407, 0);

and done

    mf.addEvent(0, 0, mm);

Also don’t forget, you need a lamdba to make it really C++11:

bool isValidName(std::string& s){
    return std::all_of(s.begin(), s.end(),
        [](char& c){
            return 0x20 <= c and c <= 0x7f;
        });
}

Command line options

This was ok. It’s not Rust’s clap or Haskell’s optparse-applicative but it works. In short you need to build arguments like

    TCLAP::ValueArg<unsigned int>
        slot("s", "slot", "Wavetable to write to. Between 80 and 118.", true, 0, "slot");
    cmd.add(slot);

and the library’s “template magic” does some validation.

Notice how you need to add the argument to the TCLAP::CmdLine cmd object unless it’s a TCLAP::SwitchArg in which case you pass cmd as an argument. Why? Because software 💦.

Closing

In the end, it works like a charm. I also have plans to somehow add this functionality to WaveEdit which is super cool.

For now, you need to export the wave table from WaveEdit, which exports them as 64 waves of 256 samples each (not 128), and convert them using wave2blofeld’s -d switch which will subsample each sample. Not the best solution but hey it works.

Some time ago I decided to make something in Rust so my snobbish commentaries regarding the language would, at least, be honest.

Truthful to my style, and because it’s usually the best way to try a language, I decided to do something industrialistic, something that takes command line arguments, has error messages and is efficient. I find that this kind of project leaves you with a bigger picture of the language, community and libraries than doing just an exercise on it’s features and ideas.

The project I’ll go trough in this blog post is a pixel sorter. It takes an image like this

and turns it into this

In this example, the pixels where sorted by redness which is the default, but it can sort them by many dimensions: red, green, blue, alpha, hue, saturation and lightness.

The code is over at GitHub, so feel free to criticize/improve it.

Cargo is love

Having worked with C++ a lot this year I think now I can really appreciate how great cargo is, and rustup too. Dependency handling, which can take literally between a day and a week in a new C++ project, is solved with 3 commands. Just wanted to say that, and cheers to the cargo team!

Also, for reference, this was done in Rust nightly 1.24.

First impressions

First of all, let me say I have but the deepest admiration to the team behind Rust. The language is barely 7 years old but it already feels extremely robust, and can, in occasion, feel really elegant.

On the other hand, getting my head around lifetimes was, I admit, tricky, but even worse were traits! I spent days just browsing trough the image crate’s documentation, trying to get my head around all the possible ways to access the underlying pixel data.

Things like

impl<P, Container> GenericImage for ImageBuffer<P, Container>
where
    P: Pixel + 'static,
    Container: Deref<Target = [P::Subpixel]> + DerefMut,
    P::Subpixel: 'static,
  type Pixel = P;

would confuse me for a long time. Specially because of the mix between types, traits and lifetime parameters that occur.

And I have a decent Haskell background, I should be ready for this kind of things …

By the way, that is basically saying that, an ImageBuffer parameterized over such P and Container types, implements the GenericImage trait and thus has all the trait’s methods.

On the other hand, this crate does a lot of things, and it manages to do that in an orderly, pseudo type-safe fashion.

From the top

Reading the main function in main.rs gives us a basic idea of the full program:

1. parse some options
2. try to open and decode the image file
3. sort it
4. guess the format of the output file (which could be different from the input file)
5. create the empty file
6. write to it

Every step except for 3 can fail, and in that case it prints the error and exits.

Pattern matching does look cool too, and it let’s you declare and define a variable on a conditional (something which C++ makes really complicated):

let img = match image::open(&opts.inpath) {
    Ok(f) => f,
    Err(err) => {
        eprintln!("{}", err);
        process::exit(1);
    }
};

Yes, I know this is the canonical either monad example, but Rust makes it more elegant to do it like this (for me).

Option parsing

This is done in options.rs and is extremely straight forward. All the logic is expressed in one single expression:

let matches = App::new(env!("CARGO_PKG_NAME"))
    .version(env!("CARGO_PKG_VERSION"))
    .about(env!("CARGO_PKG_DESCRIPTION"))
    .arg(Arg::with_name("mode")
        .help("Sort dimension")
        .long("mode")
        .short("m")
        .takes_value(true)
        .default_value("red")
        .possible_values(&["red", "green", "blue", "alpha",
                           "hue", "sat", "lig"]))
    .arg(Arg::with_name("INFILE")
        .help("Input image")
        .required(true))
    .arg(Arg::with_name("OUTFILE")
        .help("Output image")
        .required(true))
    .get_matches_from(args);

I can’t imagine it being simpler or clearer, props to the team behind clap.

Also, the env! macros bring information over from the cargo environment, again an excellent feature of the build system.

Almost sorting

The sorting function’s type signature looks like

fn sort_pixels(img: &image::DynamicImage, mode: &options::Mode) -> image::RgbaImage

The program starts by choosing which function to use when sorting the pixels. This is (almost) very simple:

let key_fn = match *mode {
    options::Mode::Red => sorters::get_red,
    options::Mode::Green => sorters::get_green,
    options::Mode::Blue => sorters::get_blue,
    options::Mode::Alpha => sorters::get_alpha,
    options::Mode::Hue => sorters::get_hue,
    options::Mode::Saturation => sorters::get_sat,
    options::Mode::Lightness => sorters::get_lig,
};

Where the sorters namespace has functions of the type (p: &&image::Rgba<u8>) -> u8.

Now, what’s up with *mode? This is actually quite interesting. We are matching on a reference, thus we can either match on references (for example, &options::Mode::Red) or dereference it. This language wart isn’t a big deal, but is nevertheless being looked at over at RFC 2005.

Fun thing: the compiler error actually takes you there. Nice detail.

error: non-reference pattern used to match a reference (see issue #42640)

Actually sorting

This is where I spent most of the time, and still couldn’t get it to do exactly what I wanted to do, but it works.

Here is the actual sorting code:

let buf2 = buf.clone();
let mut sorted_pixels: Vec<_> = buf2.pixels().collect();
sorted_pixels.sort_by_key(key_fn);

for (i, pixel) in buf.pixels_mut().enumerate() {
    *pixel = *sorted_pixels[i];
}

Here buf is an RgbImage = ImageBuffer<Rgb<u8>, Vec<u8>>. As hard as I tried, I couldn’t get to sort that without copying the data (at least I assume it is copying? Could clone() be optimized away?). And I can’t just access the underlying Vec<u8> because that’s flattened to it’s channels, elements aren’t pixels but single colors.

As it is now, it first copies everything into a temporary buf2, sorts that and then iterates over the original buffer, assigning each position the sorted result.

I think it is possible to avoid the copy, but I already spent too much time trying so if you kind reader know how, please contact me.

Tests?

Testing in cargo is okay. It’s cool to have testing directly built on the build tool, but it’s a little awkward to use and extremely dependent on file and folder names.

For example you need to include the test module in main:

#[cfg(test)]
mod tests;

Nevertheless, it works like a charm.

Wrapping up

For me, the most amazing bit was the tooling. I haven’t tried the debugger (I’ve no idea about this, but LLDB, the llvm debugger, is an excellent tool and I guess a plug-in would work like a charm) but cargo, docs.rs and rustup make it really simple and ergonomic to work with other people’s code, which for me is a crucial aspect of modern software development.

Safety was also not a problem. I had zero out-of-range-access errors but the code is kind of high level, which makes it not so surprising.

Build times where quite short too: while an order of magnitude slower than C++, think about this: This blog is made in Haskell, and I had to rebuild it from scratch when starting to write this post because I had deleted the ~/.stack/ folder … well, it just finished building right now.

That’s almost 2 hours.

Yesterday I finally decided to port our D3 v3 viz code to the latest D3 v4, and it was a really pleasant experience actually! This came as a nice surprise, since most JavaScript libraries don’t offer a nice upgrading experience, in my opinion (react-router I’m looking at you).

Also, this semester me and some class mates did a lot of viz work for our Numerical Methods class which resulted in some really nice graphics.

Anyway, during this time it ocurred to me how silly it would be to do a pseudo-statistical map on a polyhedral projection. So, finally with some time available, I decided to do another voronoi map!

Let me be clear: this is the least statistically significant map that can be drawn. Don’t show it to your friends.

But also, it looks amazing! Show it to your boss!

(For example, some Antartida zones fall inside some California city’s polygon, so yeah, nope)

How?

The temperature data is from Kaggle and the planet from our beloved Natural Earth.

First of all, the data is grouped by year so we have the average temperature for each city for a given year. For this map I chose 1992. Secondly, we take each city and project it’s latitude and longitude using d3.geoPolyhedralWaterman().

Finally, we separate the space using a voronoi diagram so each city is given a polygon that contains all the points it is closest to.

Wanna hear awesome? Initially I used d3.geoEquirectangular() for tying everything up, only at the end I switched to the butterfly and it worked at once!

Also I’m using Susie Lu’s excellent d3-legend. The legend’s entire code is:

let legend = d3.legendColor()
    .shapeWidth(30)
    .shapePadding(0)
    .labelFormat(d => d3.format('.0f')(d) + ' ºC')
    .labelAlign('start')
    .cells(8)
    .orient('vertical')
    .ascending(true)
    .scale(scale);

let legend_y = height / 2;
svg.append('g')
    .attr('class', 'legend')
    .attr('transform', 'translate(10, ' + legend_y + ')')
    .call(legend);

Another interesting bit is the clip path. Since the Waterman projection is not continuous, we have to cut some pieces of the map away. For that we use two things: that d3.geoPath() and the projection know how to render a {type: 'Sphere'} GeoJSON datum, and an svg clipPath in a <defs> element (which we also use to render the limit of the world). The full code for the clipping and limit drawing is:

var defs = svg.append('defs');

defs.append('path')
    .datum({type: 'Sphere'})
    .attr('id', 'sphere')
    .attr('d', path);

defs.append('clipPath')
    .attr('id', 'clip')
  .append('use')
    .attr('xlink:href', '#sphere');

svg.append('use')
    .attr('class', 'limit')
    .attr('xlink:href', '#sphere');

Followed by .attr('clip-path', 'url(#clip)') on the <path> elements we want to clip. So simple!

Anyway

Another horribly bad thing going on in this map: d3.voronoi() is two dimensional, it’s not made for spherical coordinates. This is greatly discussed here. I have over 3500 cities here so using the O(n^2) algorithm mentioned there is just not possible (my tab crashed instantly).

I hope you liked it or found it useful. The code is over at GitHub, like always.

Cheers!

A couple of weeks ago this blog post over at the Hypothesis blog caught my eye. I had heard of TLA+ before, but the simplicity of the linked solution surprised me.

And the Python solution was very nice too, I totally have to bring Hypothesis to the office someday.

Anyway, some days later that same week, @jystic announced the release of Hedgehog, a property based testing library, much like QuickCheck but with a more modern approach.

Look, I’d never used QuickCheck or property testing for a real project before, so I was kind of surprised when I recognized some of the advantages @jystic said his library had. Specifically:

• No Arbitrary instances.
• • The amount of boilerplate needed to avoid orphan instances is tedious in general.
• Showing the values and code where a property fails.
• • The first frictious moment I had when using QuickCheck was exactly because of this.

So it was set. Let’s try Hedgehog with the Water Jug problem.

The problem

This is better explained in the Hypothesis post, but let’s recapitulate: you have 2 water jugs that you can either fill or empty. One has 5 litter capacity and the other 3. You win by leaving 4 litters in the big one.

The code

I didn’t want many dependencies, so no state monads or funny folds. Just a good old finite state machine. The fsm function will take a state and a step, and return a new state.

We start by importing,

module Jugs where

import           Hedgehog
import           Hedgehog.Internal.Property (TestLimit(..))
import qualified Hedgehog.Gen as Gen
import qualified Hedgehog.Range as Range

and then defining our type for the problem:

data Step = FillBig
          | FillSmall
          | EmptyBig
          | EmptySmall
          | SmallIntoBig
          | BigIntoSmall
          deriving (Show, Eq, Enum)

Those are all the different kinds of steps we’ll have.

We will also need state to keep track of how much water our jugs have,

data State = State
  { bigJug :: Int
  , smallJug :: Int
  } deriving (Show, Eq)

which by the way start out empty:

initial :: State
initial = State { bigJug = 0, smallJug = 0}

Now, let’s define what each step does:

fsm :: State -> Step -> State
fsm s FillBig = s { bigJug = 5 }
fsm s FillSmall = s { smallJug = 3 }
fsm s EmptyBig = s { bigJug = 0 }
fsm s EmptySmall = s { smallJug = 0 }
fsm (State big small) SmallIntoBig =
    let big' = min 5 (big + small) in
    State { bigJug = big'
          , smallJug = small - (big' - big) }
fsm (State big small) BigIntoSmall =
    let small' = min 3 (big + small) in
    State { bigJug = big - (small' - small)
          , smallJug = small' }

That was easy. Let’s also define a function that executes a list of steps from the initial state:

execute :: [Step] -> State
execute = foldl fsm initial

Finally, the Hedgehog bits. We want to say: give me a random list of steps that has length between 0 and 20. We do it like so:

steps :: Monad m => Gen m [Step]
steps = Gen.list (Range.linear 0 20) (Gen.enum FillBig BigIntoSmall)

Also we define our property, which we want to see proven false,

prop_solution :: Property
prop_solution = withTests (TestLimit 5000) . property $ do
    s <- forAll steps
    let (State big small) = execute s
    assert $ big /= 4

also setting the TestLimit to 5000, so it doesn’t give up too early.

The solution

We can load this file using stack ghci --package=hedgehog-0.1, after adding hedgehog and any other missing deps to the global stack project (that is ~/.stack/global-project/stack.yaml). Finally, we get:

*Jugs λ> check prop_solution
  ✗ <interactive> failed after 3290 tests and 5 shrinks.

       ┏━━ Jugs.hs ━━━
    44 ┃ prop_solution :: Property
    45 ┃ prop_solution = withTests (TestLimit 5000) . property $ do
    46 ┃     s <- forAll steps
       ┃     │ [ FillBig
       ┃     │ , BigIntoSmall
       ┃     │ , EmptySmall
       ┃     │ , BigIntoSmall
       ┃     │ , FillBig
       ┃     │ , BigIntoSmall
       ┃     │ ]
    47 ┃     let (State big small) = execute s
    48 ┃     assert $ big /= 4
       ┃     ^^^^^^^^^^^^^^^^^

Isn’t that pretty? It even has colors on my terminal.

Also, the solution is exactly the same the Hypothesis post had! (I saw other solutions, some even not optimal, but this one was the most common).

I think I’m gonna like Hedgehog.

In the meantime, I’ll see about implementing the generic solution some day.

Dr. Who is a hell of a good show, it matches the right amount of sci-fi with a young heart and an amazing production team.

I’ve been trying to watch season 9 since it came out, but somehow I don’t feel the show as I did before, so I thought, is it me or did the quality change? Let’s plot it and see!

So, first I did some bash-ing of the episodes by season:

for x in {1..9}; do
    wget "http://www.imdb.com/title/tt0436992/episodes?season=$x";
done

Then, I wanted to try scalpel, a Haskell scraper. It is a really cool thing, for example:

title :: Scraper Text Text
title = attr "title" $ "a" @: ["itemprop" @= "name"]

matches an “a” tag with the “itemprop” attribute set to “name”, and the extracts the “title” attribute value. Here is the full gist.

Lastly I added some D3js pain (sorry JavaScript, you are awful) and I got the plot.

Now, a couple of things grab my attention:

Linear regression is boring

I spent around an hour plotting the linear regression (it’s the light-gray line), but it turned out to be almost constant!

This kind-of-means that I was wrong, the show is neither “better” or “worse” (for whatever that means in the context of IMDB ratings).

Actually the slope of the regression is -0.008453710369452755 so yeah, not enough for my elitist palate.

Jam the lines

The lines cross the mean (which is exactly 8, by the way) a lot of times. Episode ratings alternate a lot between almost-7★ and not-quite-9★. (If you are curious, the median gets crossed 40 times.)

For example, episode 9-9 (Sleep No More) rates at 6.2★, while the next one (Face The Raven) jumps all the way up to 8.8★! It’s a difference of 2.6 stars, the largest one in consecutive episodes. Even more, the previous one, episode 9-8 (The Zygon Inversion) is an 8.7★.

Make your own assumptions (because I won’t make a single one, seriously, it doesn’t mean anything).

Season finales

Every season finale scores better than the first episode of the next season, which is kinda obvious considering Dr. Who usually ends seasons in a time-galatic war between pan-dimensional hamburgers and robo-bears.

But wait a minute, except for season 9!

There it is, I just made a hiatus on an exceptional low note.

Question answered.

Good bye.

In my eternal quest to learn Haskell I decided to implement a game in ncurses.

The code is over at GitHub if anyone wants to check it out.

The game is “Ultimate Tic Tac Toe”, a variant on classic 3x3 Tic Tac Toe where inside each of the 9 cells there is another, smaller cell, where actual gameplay takes place. But there is a catch: the sub-board where the player plays is decided by the playing position of the last player’s move!

I came across this game a while ago in this blog post. The link also contains a better explanation of the rules, which you should read for any of this to make sense.

By the way, the main libraries I chose for this project are:

• UI.NCurses for good old graphics
• Lens.Simple for lensing
• Data.Array as the main data structure for the board

Some types for great programming

This is the main type:

type Game a = StateT GameState Curses a

Which means Game a is a Monad that carries a state of type GameState, another Monad inside named Curses, and returns some a.

Convenient fact: Curses is an instance of MonadIO which means we can use liftIO!

Then we have the game state:

data GameState = GameState
    { _gPlayer :: Player
    , _gBoardState :: BoardState (BoardState (Maybe Player))
    , _gMode :: Mode
    , _gQuit :: Bool
    } deriving Show

This record holds most of the game mutable state, namely:

• who is currently playing
• the state of the board
• the game mode
• if they want to quit

The game mode is a simple sum type data Mode = Free | Fixed. Free means the player can choose his/her next sub-board (for example in the first turn), where Fixed means they can’t (for example after a valid move from the other player).

Now, notice that funny nested BoardState? Check out the type:

data BoardState t = BoardState
    { _bsCells :: Array Position t
    , _bsPosition :: Position
    , _bsWinner :: Maybe Winner
    } deriving Show

A board is three things:

• a position (where the player currently is)
• a winner, maybe
• an array indexed by Positions populated by some thing t

Thus, the big board is just a board with an array of boards, where each one has inside an array of Maybe Player, because either it’s an empty cell or it has a player’s move.

All this arrays are indexed by the Position type:

data Position = Position Vertical Horizontal
              deriving (Show, Eq, Ord, Ix)

data Vertical = T | M | B deriving (Show, Enum, Eq, Ord, Ix)

data Horizontal = L | C | R deriving (Show, Enum, Eq, Ord, Ix)

Where each letter stands for Top, Middle, Bottom (for Vertical) and Left, Center, Right (for Horizontal).

This means we can have a 3x3 array filled with ones as easily as:

someAray :: Array Position Int
someArray = listArray (Position T L, Position B R) (repeat 1)

which you can index

someArray ! (Position T R)

And of course, deriving Ix knows how to produce all the Position pairs! Isn’t that just lovely?

Finally just make some lenses:

$(makeLenses ''GameState)
$(makeLenses ''BoardState)

I also took the liberty to add a lens for arrays (my first custom lens!):

ax :: Ix i => i -> Lens (Array i a) (Array i a) a a
ax i = lens getter setter
  where
    getter = (! i)
    setter = (\arr v -> arr // [(i, v)])

Imperative programming in the key of Monad

How does this look on action? Well, quite sexy I’d say. This is what happens when a player presses spacebar on a sub-board’s cell:

-- if the sub-board's cell is occupied return Nothing,
-- otherwise return the current sub-Position
actionPlayer :: Game (Maybe Position)
actionPlayer = do
    -- `use` applies a lens to the state, giving back it's contents
    pl <- use gPlayer

    -- of course, lenses compose
    pos <- use (gBoardState . bsPosition)

    -- `zoom` runs a State computation inside a piece of our bigger state,
    -- in this case, inside the sub-board at `pos`

    -- `bsAx p` is just `bsCells . ax p`
    zoom (gBoardState . bsAx pos) $ do

        pos' <- use bsPosition

        -- LambdaCase is surprisingly fun for State monads,
        -- you can get a field of the state and pattern match
        -- on it in a single line, just like in the imperative world
        -- (except the imperative world doesn't have pattern matching)
        use (bsAx pos') >>= \case

            -- sow now we are inside a cell in a sub-board and match:
            -- if the spot is already occupied, return Nothing
            Just _ -> return Nothing

            -- if the spot is free, assign the current player to it
            -- and return this position
            Nothing -> do
                bsAx pos' .= Just pl
                return $ Just pos'

Two things stand out:

This code looks very Python/Ruby/Javascript like. The syntax is different, but there is less syntax too. I mean, we are super used to reading stuff like myObj['key'][thing].lol, but that’s just syntax. On the other hand, things like >>= or gBoardState . bsPosition are functions. You can check their types, you can pass them around and put them in a list.

(well yeah, <- is syntax too, but one that applies to every Monad ever not just dictionaries)

The second thing is that zoom is great. With it you force a do block to have access to only a part of the entire state, which of course makes your code simpler to reason about and generally safer!

Drawing with ncurses was also pretty simple. For example, here we draw all the board’s crosses:

drawCrosses :: GameState -> Colors -> Update ()
drawCrosses gs colors = do
    -- main cross
    drawCross 7 Nothing (0, 0)

    -- small crosses
    let offsets = [1, 1 + 8, 1 + 8 + 8]
        coords = (,) <$> offsets <*> offsets
        poss = range (Position T L, Position B R)
        winner p = gs ^. gBoardState . bsAx p . bsWinner
        color_ids = map (winner >=> return . colors . color) poss

    mapM_ (uncurry $ drawCross 1) $ zip color_ids coords

Where drawCross takes the size of a cell, a Maybe ColorID and a coordinate offset, and just draws the horizontal and vertical lines.

Since smaller boards get colored when a player wins, we have to get their winners and colors first.

Also (,) <$> offsets <*> offsets is funny. It builds a list of pairs of all combinations of offsets. Basically \( \{ (x, y) : x \in S , y \in S \} \) with \( S = \{ 1, 1+8, 1+8+8 \} \) because each big-board cell is 7 characters wide plus one for the line.

The main loop

It looks like:

-- I didn't want to add the windows and colors to the state nor use a Reader,
-- so I just used good old arguments
mainLoop :: Window -> Window -> Colors -> Game (Maybe Winner)
mainLoop w1 w2 colors = do

    -- we draw everything
    drawAll w1 w2 colors

    -- then, depending on the input
    parseInput w1 >>= \case
        -- if a movement key, move accordingly.
        -- remember: movement is just an update on the Position in the state
        -- this when rendered moves the cursor around
        Movement m -> movePlayer m

        -- if they pressed spacebar then
        Select -> use gMode >>= \case

            -- free mode: just lock into the sub-board
            Free -> do
                p <- use (gBoardState . bsPosition)
                use (gBoardState . bsAx p . bsWinner) >>= \case
                    -- unless board is already closed (has a winner)
                    Just _ -> return ()

                    -- board is open, enter
                    Nothing -> gMode .= Fixed

            -- fixed mode: lots of things
            Fixed -> actionPlayer >>= \case
                -- illegal action, do noting
                Nothing -> return ()

                -- legal action, `played_p` is where they played
                Just played_p -> do

                    -- calculate sub-board and big-board winners
                    p <- use (gBoardState . bsPosition)
                    gBoardState . bsAx p . bsWinner <~ innerWinner played_p

                    gBoardState . bsWinner <~ outerWinner p

                    -- switch players
                    gPlayer %= \x -> if x == X then O else X

                    -- move to next board
                    -- `%=` updates a field of the state with a function
                    gBoardState . bsPosition .= played_p

                    -- enter free mode if closed
                    use (gBoardState . bsAx played_p . bsWinner) >>= \case
                        Nothing -> return ()
                        Just _ -> gMode .= Free

        -- finally if quitted update the state
        Quit -> gQuit .= True

    use gQuit >>= \case
        -- now, if they quitted return
        True -> return Nothing

        -- otherwise, check if we have a winner and return that, or just loop
        False -> use (gBoardState . bsWinner) >>= \case
            Nothing -> mainLoop w1 w2 colors
            winner -> return winner

Well that’s pretty big, but if you consider that’s almost everything you need for the main loop, it’s quite interesting.

Also, see that last two lines? If you switch them around you introduce a bug, but don’t fear, GHC will saves us (with -Wall enabled, which you should always do) saying:

Warning:
    Pattern match(es) are overlapped
        In a case alternative: Nothing -> ...

I think that’s super neat.

Wrapping up

This was not my first time with a monad transformer, nor with lenses, but I did learn some things. For example, I could read some lens’ errors!

While I can use lenses, the abstraction is quite … abstract. But this time I think I got a step closer to understanding it.

Testing was quite easy too. Since I could just spit the GameState to stderr (and pipe it to a file), I would start the game with a board full of moves! All of this for free deriving Show and Read instances.

I was also about to build a server/client mode for online multiplayer, but I thought: “if anyone is ever gonna play this online, they surely know how to screen -x”.

In the end, doing something actually playable is always fun :D

While working on discogs2pg, a tool for importing Discogs’ data dumps into Postgre efficiently, I ran into a space leak.

So, in the beginning, things worked nice and in constant memory.

I opened the XML as a Lazy.Bytestring, Text.XML.Expat.Tree took that and gave me a lazy tree of nodes, I made a list of records with those and finally, they were copied by Database.PostgreSQL.Simple.Copy into de database.

The parsing and storing of the first XML file was done, so I move on to the next one, simplifying and refactoring as needed, all while thinking “oh my Haskell is great, so so good”.

The problem

Suddenly, while trying the thing, I realized something was eating memory. My first suspect was firefox, but htop quickly verified it was the parser.

I’m using stack, which is great by the way, for the first time in a real project, and enabling profiling was super easy.

First I ran stack build --executable-profiling --library-profiling and it cried about base missing. This was because I didn’t have the profiling base library installed, something apt fixed right away. Then I started trying different flags, since profiling in haskell is not really well documented, even though it’s really well supported.

I did

$ discogs2pg artists.xml +RTS -hy
$ hp2ps -c discogs.hp
$ evince discogs2pg.ps

and saw this, which may well be the epitomical example of a space leak:

So I googled ARR_WORDS and found it has to do with ByteStrings (I use a lot of those here because performance). The thing is, I have no idea what they have to do with the space leak because I fixed it with a little change that doesn’t involve them.

But first, let me introduce you to the program.

The program

The program ends in the store function which takes a list of things that implement Storable, escapes them (generates some ByteString.Builder) and makes them ByteString for Postgre’s COPY. The class looks like this:

class Storable a where
    getName :: [a] -> String
    getTables :: [a] -> [TableInfo]
    toRows :: a -> [Builder]
    avoid :: a -> Maybe String

data TableInfo = TableInfo
  { tableName :: String
  , tableColumns :: [String] }

Ignore the weird [a] -> Stuff, that’s gonna change in the future. That just says “a collection of ‘a’ has this attributes”.

So, each Storable can become a [Builder] (one for each table) and a list of them has a [TableInfo] (same size as the [Builder]), which is just a String (the table name) and a [String] (the table columns). This is used in store like this:

store :: (Show a, Storable a) => ConnectionInfo -> [a] -> IO ()
store conf values = do
    -- open a connection per table, TRUNCATE the table
    -- and start the COPY ... FROM STDIN
    conns <- forM (getTables values) $ \table -> do
        conn <- connectPostgreSQL conf
        begin conn
        _ <- execute_ conn $ fromString ("TRUNCATE " <> tableName table)
        copy_ conn $ toQuery table
        return conn

    -- then for each value
    forM_ values $ \val ->
        -- we get the [Builder]
        let rows = toRows val
        -- zip them with the connections list
        forM_ (zip conns rows) $ \(c, r) ->
            -- make them ByteString and send it to Postgre
            mapM_ (putCopyData c) (toChunks $ toLazyByteString r)

    -- finally, putCopyEnd for each connection, printing the table name
    -- in the process and the count of copied elements
    forM_ (zip (getTables values) $ conns) $ \(table, conn) -> do
        n <- putCopyEnd conn
        commit conn
        putStrLn $ tableName table <> " = " <> show n

Pretty imperative looking, but it works like a charm. Except for the space leak.

Now, the first thing I tried was refactoring the inner loop, avoiding currying and composition but that didn’t work.

I googled a while but nothing.

I was getting ready for making all my records strict, which I knew was not the problem since it worked perfectly a day ago, and then fortunately while looking around, I got rid of the duplicate (getTables values) which was just bothering me … and holy shit that was it!

The fix

The fix is so simple and misterious it’s not even funny:

store :: (Show a, Storable a) => ConnectionInfo -> [a] -> IO ()
store conf values = do
    let tables = getTables values -- here

    conns <- forM tables $ \table -> do
        ...

    ...

    forM_ (zip tables conns) $ \(table, conn) -> do
        ...

Which generates this pretty graph:

Somehow a list of String was causing a 60MB space leak of ARR_WORDS.

Damn.

I love Haskell.

UPDATE: The explanation

Over at reddit pycube offers what seems to be a very reasonable explanation (link).

Basically, even though getTables doesn’t depend on values, GHC doesn’t really know this, all it can see is getTables values.

So in the leaky version, it doesn’t GC the things inside values because it thinks it’s gonna need them later for the last getTables! Meanwhile in the fixed version, nothing prevents those things from being GCed.

That’s also why the profile says the leak is from ARR_WORDS, because what was leaking was values.

Thanks to pycube and the folks over at reddit for helping with this!

Se me dio por hacer un mapa de los estadios de los equipos de Primera División del fútbol argentino a la fecha. Osea, tomando cada estadio como un punto en el plano, trazar el diagrama voronoi que se forma.

Además de ser completamente irrelevante, es un lindo ejercicio para practicar D3.

Las shapes

Primero necesitamos dibujar la Argentina, para lo que necesitamos los shapefiles. Mi fuente favorita es Natural Earth, específicamente la versión Admin 0 países (para no entrar en discusiones de soberanía).

Para procesar shapefiles mi herramienta es ogr2ogr, parte del conjunto de herramientas de gdal-bin. No es fácil o simple de usar, pero es muy poderosa y sigue de cerca la especificación.

Mi workflow con shapefiles es muy simple:

1. Veo que trae el mismo por arriba con ogrinfo -geom=no ne_10m_admin_0_countries.shp ne_10m_admin_0_countries (-geom=no es crucial para poder ver algo además de muchas coordenadas).
2. Pasándolo por alguna herramienta de CLI, identifico las features que busco, en este caso grep -i arg.
3. Uso ogr2ogr para pasarlo a GeoJSON, específicamente ogr2ogr -f GeoJSON argentina.json ne_10m_admin_0_countries.shp ne_10m_admin_0_countries -where 'SOV_A3 = "ARG"'.

El javascript

El código se puede ver en el repo de este blog

Mike Bostock tiene un formato llamado TopoJSON que es lo que vamos a usar (porque es mas liviano mas que nada, no trae mas ventajas para esta visualización).

Primero que nada tenemos que elegir una proyección:

var proj = d3.geo.albers()
    .center([0, -36])
    .rotate([65, 0, 0])
    .parallels([-20, -56])
    .scale(1700)
    .translate([width / 2, height_full / 2]);

Albers es muy linda y clásica, por lo que no hay mucha vuelta que darle, aunque tardé un rato en encontrar una combinación que me diera el país derecho. Se ve que lo mejor es darle un center vertical y buscar la feature con rotate que hacer todo en uno.

Luego tenemos que agarrar la frontera de la Argentina y transformarla en un path de SVG y en un array de puntos. El primero para dibujarlo, y el segundo para recortar la geometría voronoi.

var border = topojson
    .feature(arg, arg.objects.argentina);

var path = d3.geo.path()
    .pointRadius(2)
    .projection(proj);

svg.append('path')
    .datum(border)
    .style({
        'stroke': 'black',
        'fill': '#eee'
    })
    .attr('class', 'argentina')
    .attr('d', path);

var voronoi = d3.geom.voronoi();

var points = equipos.features.map(function(d){
    return proj(d.geometry.coordinates);
});

var exterior = projectLineString(border, proj);

var voronois = voronoi(points).map(function(d){
    return d3.geom.polygon(d).clip(exterior.slice());
});

El resto son detalle estilísticos.

Además, en el código final, puse todo en una función parametrizada por la proyección, para poder dibujar el segunda mapa con “zoom”.

El mapa

y con detalle en Buenos Aires:

Me gustaría dibujar los partidos y provincias en otro contraste pero no pude hacer que quede lindo.

D3 es una de mis librerías favoritas, la uso para hacer tanto para SVG como Canvas y HTML. Es muy versátil y super eficiente, aunque a la documentación a veces le falta (pero bueno, expone mil funciones).

Además, al feeling funcional no hay con que darle.