Homework‎ > ‎

HW 1: Using a data visualization DSL and Meta-programming


This homework will familiarize you with a domain-specific language (DSL) developed in JavaScript.  This language is d3, a language for data visualizations implemented as a JavaScript library.  

You will also learn simple meta-programming, i.e., writing a program that modifies an existing program.  You will use Tampermonkey to modify a web application loaded into your browser.

You will combine d3 and Tampermonkey to place Facebook users on a map.  This experience will teach you how you can easily do other cool things with web pages.  

Along the way, you will also learn the basics of JavaScript, web programming, regular expressions, which will be needed in later assignments. 

For this homework, you will work alone.  You can consult d3 and other technologies with your class mates but you need to implement your own solution.

Due Date

Monday Jan 28, 2013, 11:59 PM

The Problem

Have you ever wondered where your friends live across the world? Could you use Facebook to visualize their locations?  Similarly, could you visualize the geographical distribution of Facebook users commenting on a controversial post? 

You probably know that many Facebook users reveal where they live. We will write a script that collects this information and draws their locations on the world map, which it will insert on the top of the Facebook page:


Recommended Steps

We recommend that you develop the solution in three consecutive steps, described below. You will not submit the intermediate solutions but having solutions to Steps 1 and 2 in hand will make the last step easy because you will simply integrate the two.
 

Before You Start (Important Prerequisites)

We will be testing your homework projects using Chrome 24 (not Chromium). Since different versions of Chrome behave differently, make sure you are using Chrome Version 24 to avoid any compatibility issues. 

Part I: Put Berkeley on the Map

In this step, you are asked to use d3 to render a world map and draw Berkeley on it. In addition to d3, you will learn how to use a geocoding API, which translates a location name to its coordinates. In this program, "Berkeley, CA" will be hard-coded in your program; in Part 3, you will replace this string constant with locations scraped from Facebook pages.  

In part I, your JavaScript program will produce this output:

 

STEP 1 Read a tutorial

We recommend that you start by reading this very nice d3 tutorial, which will also introduce you to basic concepts of web programming such as the DOM and JavaScript.  The follwoing sections are especially useful:
    • Section 2 -- Fundamental web programming including HTML, DOM, CSS, JavaScript, and SVG. 
    • Section 3 -- Set up. You do not need to download D3 just simply link to the latest release by put <script src="http://d3js.org/d3.v3.min.js"></script> inside the header (between <head> and </head>).
    • Section 4 to 7 -- These should give enough information to get started.
It does not hurt to read the whole tutorial.  Paying special attention to drawing bar charts may pay off when you need to create simple visualizations for debugging of your compiler in later cs164 projects.


STEP 2 Test yourself

Before you proceed, test your understanding of d3 data binding with this exercise.  Specifically, we want to help you understand the semantics of dataenter, and exit functions, which are a perfect example of programming constructs that DSLs introduce to simplify hairy programming tasks.  Read this tutorial http://mbostock.github.com/d3/tutorial/circle.html and then create a new HTML file with this content:

<!DOCTYPE html>
<html lang="en">

  <head>
    <meta charset="utf-8">
    <title>CS164: Where Do People Live?</title>
    <script src="http://d3js.org/d3.v3.min.js"></script>
  </head>

  <body>
    <p id="Prof. Bodik">Ras</p>
    <p id="Ali">Ali</p>
    <p id="Mangpo">Mangpo</p>

    <script>
      //var people = d3.select("body").selectAll("p").data([ "Slovakia", "Turkey", "Thailand", "Berkeley"]);
      
      //people.text(function(d) { return this.id + " is from " + d +"."; });
      //people.enter().append("p").text(function(d) { return "I am from " + d +"."; }).attr("id","I");
      
      //dataset = [ "Berkeley", "Berkeley", "Berkeley"];
      //people = d3.select("body").selectAll("p").data(dataset);
      
      //people.text(function(d) { return this.id + " lives in " + d +"."; });
      //people.exit().remove();
    </script>
  </body>

</html>

In <script>, comment out line by line starting from the beginning, think what the program will do and run the program, to verify your understanding. Make sure you know answers to the following questions:
  1. What does enter() and exit() do? Specifically, on what DOM objects do these calls have an effect?
  2. When does data binding between a DOM element and a data element happen? When you call .data(), when the dataset is changed, or potentially at some other time?
  3. Is there any visual change when data binding happen? Never, sometimes, always? Why?

STEP 3 Create index.html and start a local web server

Create index.html in your working directory.  Add some html content to the file.  

References that you may need: What is HTML?

Chrome enforces strict permissions for reading file out of the local file system. Therefore, in Part I, you will need to run your own local sever.  In the directory where you created your index.html, start your own local web server by running Python's built-in server:
python -m SimpleHTTPServer 8888
Once the server is running, direct your browser to http://localhost:8888/. The browser will display your index.html file.


STEP 4 Draw the world map

First, we need world map data (shapes of countries and such). You can use the TopoJSON data at http://www.cs.berkeley.edu/~bodik/world.json or you can download this file to your working directory so that your server can serve it for you as http://localhost:8888/world.json. 

Next, in index.html, use d3 to draw the world map.

Mapping resources:
  • A really useful mapping tutorial.  You can use its code pretty much as is, as long as you change UK subunits to world countries.
  • Use a search engine using terms d3.json, d3.geo, and TopoJSON.
  • Play with projections.
  • Another useful example. Note that we want a contiguous map, and a "country", not "state" for the attribute class.

STEP 5 Draw Berkeley on the map
Use Google Geocoding to find latitude and longitude of Berkeley. We recommend that you use the JSON output format. Next, convert latitude and longitude to pixel coordinates on the map, using the D3 projection object. Finally, draw a red circle on these pixel coordinates.


STEP 6 Fix the asynchrony bug
Our early prototype had a bug that manifested itself as the non-deterministic behavior shown in these screen shots. You may have the same bug. To try to reproduce the bug, refresh the browser, rerunning the program, and observe whether the circle is sometimes on top of the map, and sometimes under the map.

    
        Berkeley over the map                    Berkeley under the map
What causes this behavior? Can you rewrite the program to ensure that this behavior does not happen in your program no matter how the servers answer your requests?

Hint: read the d3 tutorial on the lack of z-indices in SVG.


Part II: Scraping Friends' Locations From Facebook

In this Part, you are asked to extract locations of Facebook users whose name appears on the web page and print these locations on the browser console. In Part III, you will feed these locations to the map visualizer you have built in Part I.

In more detail, we are interested in locations of users involved in Facebook activities such as postings on walls, picture posts, comments and likes on posts, etc. To extract locations, you follow to the user pages by clicking on their names; when on their pages, you can see the text "Lives in ..." if they share the information.

STEP 1 Work with Tampermonkey

Tampermonkey is a Chrome extension that allows users to extend the functionality of web pages. A programmer writes a Tampermonkey script in the Javascript language. After a web page has loaded, the script is run. Tampermonkey scripts have the ability to read and modify the loaded web page as well as query other web sites, such as call the Google geocoding API.

What to do?  Install Tampermonkey in your browser and familiarize yourself with Tampermonkey configuration.

Resources:

  • http://hibbard.eu/tampermonkey-tutorial/ might help you get started with Tampermonkey 
  • Dive into Greasemonkey  is a good book showing classic examples on how to use Greasemonkey. Greasemonkey is very similar Tampermonkey, although it works Firefox, not Chrome. You can still refer to this book as a good introduction to modifying web pages by removing or adding DOM elements.
Additional recommendation: learn browser debugging tools

You will use Chrome Developer Tools often so make them your friend. Chrome Developer Tool comes with your Chrome browser. The most basic use is printing messages to Developer Tools' console, with the  following command, which you can insert into your Tampermonkey script:

console.log(string) // prints out the string to the console

STEP 2 Extract links to friends' pages

The web page is represented by a DOM tree.  The links to your friends' pages can be easily extracted in a Tampermonkey script by selecting suitable DOM elements.  You will do so by writing a DOM query.  You can use d3 methods to write this selection query.  This tutorial might be useful. 

Use Chrome Developer Tools to help your figure out what DOM queries are needed to find the set of DOM nodes that contain the links.  

Hint: Link to your friends' page using the Inspect Element command of the Chrome debugger. 

Notice that the DOM node has "actorDescription actorName" class. You should use this class information to obtain the link to the poster's page.  You will need to do something similar for commenters, likers, etc.

STEP 3  Obtain the content of friends' pages

Invoke XMLHttpRequest to query data from the link you found in the previous step.

Hint: You might want to set withCredentials to trueThis will send your cookies to Facebook, allowing you to view your friends pages as if you were logged in.

STEP 4 Extract location where your friend lives

Use a JavaScript regular expression on the HTML text returned in Step 3.

STEP 5 Print the location to the developer tool's console.

Verify that the string only contains the city, state, and/or country. It should not contain any irrelevant string, such as a fragment of HTML. Using console.log is a very handy way to debug your program.


Part III: Facebook Map

Now, we are ready to put everything together by feeding output of Part II into Part I.  

STEP 1 First, you will insert the svg element with the map as the first item in the body of Facebook News Feed page, so that it appears visually on top of the page.  You can use d3 methods to modify the Facebook DOM.

Note:  Since you are running Tampermonkey script in the facebook.com domain, you will not be able to access the world map data from you local file system; you need to query the data directly from http://www.cs.berkeley.edu/~bodik/world.json. The methods d3.json and XMLHttpRequest will no longer work  because they violate the same-origin policy. Luckily, Tampermonkey and Greasemonkey provide GM_xmlhttpRequest that you can get around the policy.

STEP 2  After successfully drawing the map on Facebook, you will extract as many friends' pages as possible from your news feed. Find out where they live and draw their locations on the map!

Here you may run into a problem that when your Tampermonkey script is executed, the DOM does not yet contain the names of your friends.  This is because the facebook JS scripts load these dynamically, and often take their sweet time, even if it may seem to you that the page content loads instantaneously

Hint: Registering a DOM event handler

Often we want something to happen after a specific event occurs. The event could be a mouse click, mouse hover or a particular DOM element being modified. This is done through event handlers. Every element in the DOM has a set of predefined events. One can add a handler, i.e., a function that will be executed after a specific event, using the addEventListener function. Below is an example.


var afterLoad = function() {
  alert('Page loaded');  
};

window.addEventListener('load', afterLoad, true);


Links to useful sites on this topic can be found at:
http://developer.mozilla.org/en/DOM/element.addEventListener
http://developer.mozilla.org/en/DOM/DOM_event_reference

Hint: Use a timing events to run your script after the DOM has been completed

Sometimes we also want something to happen at a specific time or time interval. You might consider using JavaScript setTimeout and setInterval. Please refer to http://www.w3schools.com/js/js_timing.asp.

STEP 3: Solving remaining challenges

Once we have your script working. Let’s see how you can make your script better! This part is mandatory.

1) When you scroll down the new feed page, more data is loaded. There are more people that you can map on your facebook map. Make sure your script can handle this!

2) If your facebook runs slower and slower after your script is running for a while, you might encounter this problem. Your script might obtain same places from same people multiple times (especially. if you use setInterval or eventListener). You do not want to slow down your Facebook page by drawing a hundred circle on top of each other. Fix this!


Deliverables

For this project you will turn in the following:

  • Source code for Tampermonkey script
  • 60 second (preferably shorter) video of your script working. Go to "UC Berkeley" Facebook page and start recording the video. (We chose "UC Berkeley" rather than your Facebook page because we do not want to intrude on your personal life. Make sure that your script can extract links from UC Berkeley page to people's pages.) The video should show places being added to the map. Scroll down the page to reveal more posts, and then go back up to show that more places have been added to the map. The video should also show your name at the top right corner of the Facebook page (or somewhere else on the screen, should you not want to log in to Facebook).
  • Shorter than one page write-up of the project answering the project questions (see below). The questions the we ask along the steps are just thought exercises for you. You do not have to submit answers to those questions.

Writeup

Write two short paragraphs, each up to 70 words.  Yes, that's very little text, so start with a longer draft and then revise it to communicate only your best thoughts and so concisely.

 

1st paragraph: Where did your time go?  Reflect on your experience with developing Facebook Map app and identify the "developer time bottleneck". 

 

  • We want to know which step of the development took you the longest time, weighted by your frustration level. 
  • Analyze and clearly describe the reason(s) why this step accounted for more time than you think it should in an "ideal programmer's world".
  • In your reflection, include all steps of the development, starting from understanding the specification (ie. the HW1 handout), installing Tampermonkey, reading about related materials and tutorials, and testing the app.

 

2nd paragraph: Suggest how to remove the frustration or accelerate the identified bottleneck.  Propose a tool, a development methodology, a library, a language extension, or some other innovation that  would have simplified your programmer task.

  • Your innovation does not need to be a tool or a language.  It could be a methodology (eg. a new way of providing specs or testing meta-programming apps).
  • Describe what the innovation does and how the programmer uses it.
  • Describe how it solves (or sidesteps) the problems you identified in the first paragraph. 

Screen Recording

Screen recordings can be made with the following software, depending on what system you are using.

    OS X: You can use a free trial version of iShowU HD or create a screen recording with Quicktime X.

    Windows: You can use the free Cam Studio.

    Linux: You can used recordMyDesktop, but You might have to convert the video file afterwards.

Submission

Using your instructional account, submit the following three files:

  1. hw1.js                                     <-  Script source-code
  2. hw1.avi or hw1.ogv or hw1.mov  <-  Video
  3. hw1.txt                                   <-  Write-up
with the command submit hw1
Č
ċ
starter.user.js
(2k)
Shaon Barman,
Jan 19, 2012, 12:54 PM
Comments