{"id":4202,"date":"2018-09-01T16:52:34","date_gmt":"2018-09-01T14:52:34","guid":{"rendered":"https:\/\/blog.mi.hdm-stuttgart.de\/?p=4202"},"modified":"2023-06-09T12:01:37","modified_gmt":"2023-06-09T10:01:37","slug":"parsing-all-open-source-elm-code","status":"publish","type":"post","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2018\/09\/01\/parsing-all-open-source-elm-code\/","title":{"rendered":"Parsing all Open Source Elm Code"},"content":{"rendered":"

This project was originally inspired by a talk<\/a> Felipe Hoffa gave at the Github Universe conference last year. He talked about how we can analyse the code hosted on Github at a large scale to learn interesting things. I’m always excited about learning new programming languages, at the moment my favourite new langue is Elm<\/a>, a small functional programming language for building web applications. After watching the talk I thought it would be nice to do this kind of analysis on all the public Elm code hosted on Github.
\n<\/p>\n

Initial Idea<\/h2>\n

The big question question is: Why isn’t it already possible to learn interesting things about the code hosted on Github? The problem is that Github works on the text level. All the code is just a huge collection of text files. It would be much more useful if we could operate on the structure which the text represents. Instead of searching for all files which contain the string “List.map” we could precisely search for all source files which actually contain a reference to the List.map function.<\/p>\n

\"\"<\/a><\/p>\n

If we want to scale this approach to all the Elm files hosted on Github we need a few steps:<\/p>\n

    \n
  1. Find all Elm repos<\/li>\n
  2. Parse all the Elm files in each repos and extract the references<\/li>\n
  3. Store the references and files in the db so they can be querried later<\/li>\n<\/ol>\n

    In my implementation I’ve limited myself to just storing which symbols a file defines and which symbols it references instead of storing the whole syntax tree of each file in the db. The resulting graph structure is represented in the graph below.<\/p>\n

    \"\"I’m a notorious procrastinater (perfectly proven by the fact that I’m writting this blogpost on the day of the deadline), therefore I decided to submit my idea as a talk proposal for Elm Europe 2018<\/a> to give myself some accountability. I guess you could call this approach “talk driven development”. I was lucky and my talk got accepted by the conference.<\/p>\n

    Prototype for the conference talk<\/h2>\n

    The focus of the prototype was to quickly get some results and see for which use-cases such a graph could be useful. I wrote a Node.js script which I ran on my local machine to import the data into a Neo4j Graph Database.<\/p>\n

    Example of the resulting dependency graph of the elm-todomvc<\/a> repo:<\/p>\n


    \n\"\"<\/a><\/p>\n

    In my talk<\/a>, I’ve demonstrated some examples how you can use the graph<\/p>\n