Neubot report: October - December 2014

@bassosimoneMerry Neubot

Most of the development work in the last three months focused on libight, a library for running network experiments on the mobile platform developed in cooperation with Arturo Filastò (Tor Project / Hermes Center). This library is developed to implement the mobile applications of Neubot and OONI.

I worked on libight during the 8-14 December hackfest in Rome in which we have implemented the basic library functionality (stream sockets, DNS and HTTP). We have now three prototypal OONI tests (DNS injection, TCP connect and HTTP invalid request line) that produce a YAML file as result. And we have sketched up a plan for version of libight.

Colosseum by night
Colosseum by night—Picture by me (CC BY 3.0)

Meanwhile, Alessandro has been working to implement a utp engine for libight. Once libight is stable and the utp engine is merged, the plan is to implement a utp test for Neubot using libight. Probably, given the work to run Neubot tests as independent processes (see below), this will be implementing by packaging an executable linking libight with Neubot.

I was also invited at the 3rd Workshop on Internet Censorship Measurements—organized by Meredith Whittaker of M-Lab and held in New York City the 29-30 October 2014—to discuss issues at the intersection between traffic shaping and Internet censorship.

New York City seen from Central Park
New York City seen from Central Park—Picture by me (CC BY 3.0)

I have also worked on the draft of the journal paper describing Neubot, integrating changes proposed by Juan Carlos and further polishing and refining the text.


Neubot update 2014 Q3


So, here's an update concerning our work on Neubot in the last few months.

I finished editing my doctoral thesis on Neubot, the final version of which was delivered to Politecnico di Torino on July 18, 2014. We are currently working on a journal paper partly based on such thesis. Considering the typical rules of the publication industry, it is safer to publish my thesis on this website only after we're done with the journal paper.

I presented my doctoral thesis in Nexa Wednesday #66 (the discussion was all held in Italian, see below for the video).

After my thesis was complete, @demartin and I spent a lot of time working on the above-mentioned journal paper that, by the way, is codenamed "Neubot, the full story." At the end of the quarter, the paper's draft was complete and we were further revising and polishing it. We look forward to submit it by the end of the year.

Regarding development, I opened two development branches.


A new, old visualization of Neubot data

Academic paper@bassosimone

My coauthors, and friends, Enrico Masala and Antonio Servetti wrote a nice service that shows useful Neubot statistics, updated daily.

I call their service "a new, old visualization" because they wrote it in the fall of 2013, and I was very slow to catch up, sorry :).

With this post I just want to say them thank you publicly.

Because I find it very useful, I listed their service also in the Data section of this website.

Here you can see a couple of screenshots of this service (from which one can also have an idea of the number of tests run daily by Neubot):


New Publication on Neuviz for the "Mondo Digitale" journal

@giuseppe_futiaAcademic paper

With this blog post I am pleased to announce that the paper entitled "The NeuViz Data Visualization Tool for Visualizing Internet-Measurements Data" has been published on "Mondo Digitale", the official journal of AICA - Associazione Italiana per l'Informatica ed il Calcolo Automatico.

In the paper, written by Enrico Zimuel, Simone Basso, Juan Carlos De Martin, and me, we describe NeuViz (Neubot Visualizer), an architecture to analyze and visualize Neubot data to identify cases (to be investigated with more specific tests) in which a protocol seems discriminated.


NeuViz presentation: one of the best of 2013 for Visualoop

@giuseppe_futia, one of the most popular websites on information design and data visualization, listed the Congresso AICA 2013 NeuViz presentation among the "the most useful [data visualization] resources [made] available" on SlideShare, Scribd, and similar services since April 2013.

We are very proud that our slides have been listed together with the presentations of Alberto Cairo, Nathan Yau, and Andy Kirk, who are some of the leading international experts in data visualization.


From Big Dive to Twisted

@giuseppe_futiaBig Dive 2 logo

In June 2013 I attended the Big Dive training program, organized by the Top-IX Consortium, that aims to “boost the technical skills needed to dive into the big data universe and to extract value”.

During the course I learnt: TurboGears, a framework for developing Web applications; MongoDB, a NoSQL data storage solution; the basics of network science and statistics. Also I studied MapReduce techniques and improved my data visualization skills, by learning the D3.js JavaScript library.

This wealth of knowledge has been crucial to realize with Enrico Zimuel the Big Dive final project (entitled "Gramsci devoted"), in which we developed in a few days a prototype of a Web visualization tool to explore the network measurement experiments performed by Neubot.

Here are the photos taken during the course and the final presentation:

Within the Big Dive final project I was initially involved in the front-end development of the Web interface. Once I finished the course, in the second work phase on our prototype I studied Twisted, an “event-driven networking engine written in Python”, and implemented the prototype code to create a REST API module to retrieve Neubot data, which can be exploited by other applications, as well as from our Web interface.

As explained in a previous post, the Big Dive experience led to the development of NeuViz, a prototype data processing and visualization framework to explore network measurement experiments, and to the publication entitled "Visualizing Internet-Measurements Data for Research Purposes: the NeuViz Data Visualization Tool", in occasion of the 2013 AICA Congress.

The slides of the Big Dive final project are available on Slide Share.


Neubot/ released

@neubotNeubot icon

Version is the latest generally-available stable release of Neubot, distributed for Ubuntu >= 12.04, MacOSX >= 10.6, Windows >= XP SP3 and FreeBSD, as well as in source format.

Here's what changed in less than 140 characters: new MPEG DASH test, support for tests as plugins, improved results.html, Lisp-like mini-language to gen. plots, fixes.

Install, upgrade instructions at <>. Read on for more details.


[slides] The NeuViz Data Visualization Tool

@giuseppe_futiaAcademic paper

Hello everyone, my name is Giuseppe Futia and this is my first contribution to the Neubot website.

I started to work on the Neubot project during the Big Dive course organized by the Top-IX consortium, where I developed with Enrico Zimuel the prototype of NeuViz (the Neubot Visualizer), a data processing and visualization tool to explore network measurement experiments.

With further development of this work, we presented the paper "Visualizing Internet-Measurements Data for Research Purposes: the NeuViz Data Visualization Tool", written by Enrico Zimuel, Simone Basso, Juan Carlos De Martin, and me, at the Congresso Nazionale AICA 2013.

Here are the slides of the presentation.

In the next post I'll give you more details about my experience during the Big Dive course.


A preview of Neubot - part 1

Academic paper@bassosimone

With this blog post (and with the follow up posts) I describe what's new in the next version of Neubot, The testing of the new version is still in progress, but I am confident that I will release it next week, probably on Tuesday.

In this first post I describe the two most relevant changes since the previous stable version,, that is: the new DASH test and the support for tests as plugins. In the follow up posts I will describe more changes since

Update — Because I have now released Neubot, I will not write the second part of this blog post. Please, refer to the release announce for the whole list of changes:


New Publications: CCR, BiDaTA, and AICA

Academic paper@bassosimone

In this blog post I describe the three Neubot-related publications to which I contributed in the last six months. This post comes after a six-months-long silence period in which I did not published any update on the Neubot website, and in which I dedicated more than 75% of my time to research and teaching.

Of course, in the past six months I also did some development, however I didn't find enough time to properly test and release a new stable version of Neubot. Anyway the content of the next release of Neubot will be the subject of my next blog post, now let's concentrate on the research papers to which I contributed recently.

CCR July 2013 issue. The paper "Strengthening measurements from the edges: application-level packet loss rate estimation", writen by Michela Meo, Juan Carlos De Martin, and me, was published on the July 2013 issue of the ACM SIGCOMM Computer Communication Review.

The paper compares two models for estimating the TCP packet loss rate at the application level: the Inverse-Mathis model developed in our previous work, and the new, more advanced Likely-Rexmit model. We used Neubot's raw test to gather the data that we used to validate and compare the Inverse-Mathis and the Likely-Rexmit models.



Subscribe to Neubot RSS