Lane Tech HS - PCL - Data Visualization

Inspiration

The idea of this project is to have multiple interpretations of data visualized so that a user can toggle between seeing one or other.

In this case, I've chosen the change in crimes in my ward and change in COVID-19 cases in my city as a whole. By comparing yesterday's and today's crimes/cases, I get two interpretations of the data that can recommend me whether to go outside the area of my yard or not.

• At first, I wanted two servos to separately show the data for each, but I ddi not have two at my disposal.
• By having a button toggle between crimes and cases, I can still weigh the two recommendations of data differently, depending on the day (ex: COVID-19 is probably going to be more important for me on most days).

Date Methods

The following method retrieves the string format of a date (YYYY-MM-DD) given a shift in date from today:

String getDate(int shift) {
    int daysInMonth[] = {31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31};
    
    int year = Time.year();
    int month = Time.month();
    int day = Time.day();
    
    while (shift < 0) {
        day--;
        shift++;
        
        if (day == 0) {
            month--;
            
            if (month == 0) { 
                month = 12;
                year--;
            } else if (month == 2) {
                if (year % 4 == 0) {
                    daysInMonth[month - 1]++;
                    if (year % 100 == 0) {
                        daysInMonth[month - 1]--;
                        if (year % 400 == 0) {
                            daysInMonth[month - 1]++;
                        }
                    }
                }
            }
            
            day = daysInMonth[month - 1];
        }
    }
    
    while (shift > 0) {
        day++;
        shift--;
        
        if (day == daysInMonth[month - 1] + 1) {
            month++;
            
            if (month == 13) { 
                month = 1;
                year++;
            }
                
            day = daysInMonth[month - 1];
        }
    }

    if (day < 10 && month < 10) return String(year) + "-0" + String(month) + "-0" + String(day);
    else if (day < 10) return String(year) + "-"  + String(month) + "-0" + String(day);
    else if (month < 10) return String(year) + "-0" + String(month) + "-"  + String(day);
    else return String(year) + "-"  + String(month) + "-"  + String(day);
}

For example, if "shift" is -1, the date retrieved is yesterday's; if "shift" is 1, the date retrieved is tomorrow's. This was a method that I found relatively simple to make myself and that was useful for narrowing down particular data that I wanted.

Additionally, this method is used for finding a date of the same format (YYYY-MM-DD) after the constant JSON value we want in a string representation of a part of the data:

String findDate(String data) {
    int index = data.indexOf(value);
    int startOfDate = index + value.length() + 2;
    int endOfDate = startOfDate + dateFormat.length();
    String date = data.substring(startOfDate, endOfDate);
    return date;
}

Crime Data API Setup

The JSON file for the API used to retrieve crime data (in Chicago) is https://data.cityofchicago.org/resource/crimes.json.

The first limitation of this API is that the JSON file only holds the 1000 most recent crimes. This can be solved by using SQL injection into the query parameters of the URL. We can get a series of JSON objects only in a certain ward after a certain date (a much smaller and more defined set of data) by inserting a particular date and a particular ward number of the city in this URL:

https://data.cityofchicago.org/resource/crimes.json?$where=updated_on%3E%{{{INSERT_DATE_HERE}}}%27%20AND%20ward=%27{{{INSERT_WARD_HERE}}}%27

If we keep the ward number constant and utilize the Particle.publish function to feed the date we need into the webhook, we can retrieve the crime data as such:

The final limitations of this data deal with how it is updated by the city of Chicago:

• Crime reports are added as JSON objects to the API approximately 7 days after they occur (or greater than if they are found to be crimes later). As such, I used the "updated_on" JSON value of each object instead of the "date" value (which still gives us an idea of the crimes around that date since when the city determines information about a crime is also important).
• The crime reports are added as data to the API around 4PM CST. Since I want to be able to check my Argon's data visualizations in the morning, the date I used as "yesterday" is actually 3 days ago and the date I used as "today" is actually 2 days ago. These are still relatively current dates that can indicate how bad or worsening the situation is today.

Using this code to test the webhook, we see that the Argon correctly comes up with the crimes on the before-date and after-date after processing all the webhook's response events:

void setup() {
    Time.zone(-6);
       
    dateBefore = getDate(-3);
    dateAfter = getDate(-2);
    
    Particle.subscribe("hook-response/CityOfChicagoCrimeData", crimeHandler, MY_DEVICES);
    
    // The dateBefore parameter means we get all the crimes for both dates we need (as it checks for after or on that date)
    Particle.publish("CityOfChicagoCrimeData", dateBefore, PRIVATE);
}

void crimeHandler(const char *event, const char *data) {
    String date = findDate(data);

    if (date == dateBefore) crimesBefore++;
    else if (date == dateAfter) crimesAfter++;

    Serial.println("\n\n");
    Serial.println(event);
    Serial.println(data);
    Serial.println(date);
    Serial.println(crimesBefore);
    Serial.println(crimesAfter);
    Serial.println("\n\n");
}

COVID-19 Case API Setup

The JSON file for the API used to retrieve COVID-19 data (in Chicago) is https://data.cityofchicago.org/resource/naz8-j4nc.json.

Each JSON object in this API is a single date with COVID-19 case, death, age-aggregated, etc. instead of a single case report (like with the crime API). We can use SQL injection into the query parameters of the URL again to get a single JSON object for the data report on a particular date as such:

https://data.cityofchicago.org/resource/naz8-j4nc.json$where=lab_report_date%3D%27{{{INSERT_DATE_HERE}}}%27

If we utilize the Particle.publish function to feed the date we need into the webhook, we can retrieve the crime data as such:

This data is easier to analyze, so we used Particle's response template parser (seen in the standard webhook builder above) to get what we need (the date and the number of cases on that date). This means we work with a single webhook response event each time we publish the event to call it.

The main two limitations of this data deal with how it is updated by the city of Chicago:

• The use of dynamic updates means that case numbers on a date change even after they are reported on that date. This is a reason is it is best to not analyze the data for today or even for yesterday.
• The COVID-19 report for that date is added as data to the API around 4PM CST. Since I want to be able to check my Argon's data visualizations in the morning (and because of dynamic updates), the date I used as "yesterday" is actually 3 days ago and the date I used as "today" is actually 2 days ago. These are still relatively current dates that can indicate how bad or worsening the situation is today.

Using this code to test the webhook, we see that the Argon correctly comes up with the crimes on the before-date and after-date after processing both webhook responses:

int casesBefore = 0;
bool beforeDone = false;
int casesAfter = 0;
bool afterDone = false;

void setup() {
    Time.zone(-6);
    
    dateBefore = getDate(-3);
    dateAfter = getDate(-2);
    
    Particle.subscribe("hook-response/CityOfChicagoCaseData", caseHandler, MY_DEVICES);
    
    // The date parameters means we get all the cases only on that date
    Particle.publish("CityOfChicagoCaseData", dateBefore, PRIVATE);
    Particle.publish("CityOfChicagoCaseData", dateAfter, PRIVATE);
}

void caseHandler(const char *event, const char *data) {
    String strData = String(data);
    
    String date = strData.substring(0, dateFormat.length());
    strData = strData.substring(dateFormat.length());
    strData = strData.substring(timeFormat.length() + 1);
    int cases = atoi(strData);
    
    if (date == dateBefore) {
        casesBefore = cases;
        beforeDone = true;
    } else if (date == dateAfter) {
        casesAfter = cases;
        afterDone = true;
    }
      
    Serial.println("\n\n");
    Serial.println(event);
    Serial.println(data);
    Serial.println(date);
    Serial.println(cases);
    Serial.println("\n\n");
}

Servo and Button Setup

In order to display both data visualizations with the one micro-servo I have available, I made use of a button switch to toggle between the two:

This was the basic setup that I wanted to have with the scale and test before:

We can test this setup with this code (which provides for a debouncing button in the loop function so that one press is registered as a single toggle):

Servo servo;

const int servoPin = D6;
const int buttonPin = D5;

int reading;
int buttonState = HIGH;
int lastButtonState = HIGH;

unsigned long lastDebounceTime = 0;
unsigned long debounceDelay = 50;

int crimesExample = 0;
int casesExample = 180;
bool forCases = true;

void setup() {
    servo.attach(servoPin);
    pinMode(buttonPin, INPUT_PULLUP);
    
    if (forCases) servo.write(casesExample);
    else if (!forCases) servo.write(crimesExample);
}

void loop() {
    reading = digitalRead(buttonPin);

  	if (reading != lastButtonState) lastDebounceTime = millis(); 
  	
   	if ((millis() - lastDebounceTime) > debounceDelay) {
      	    if (reading != buttonState) { 
          	buttonState = reading;
     		if (buttonState == LOW) {
                    forCases = !forCases;
                
                    if (forCases) servo.write(casesExample);
                    else if (!forCases) servo.write(crimesExample);
     		}
            }
   	}
  	
  	lastButtonState = reading;
  	
  	Serial.println("Button: " + String(buttonState) + "; Boolean: " + String(forCases));
}

Making the Scale

I had the idea of a custom, odometer-like color scale to visualize the data and how much better or worse the crime situation and COVID-19 situation is getting day-by-day.

My process in making the scale was:

• First, to make a prototype, so that I could get a general idea of the shape of the hole.
• Next, to use an exacto knife to make another hole on another piece, utilizing the prototype to lay where the boundaries are for the servo pointer and make the hole as compact as possible.
• Finally, to color the scale based on a scale based on the written angles of the servo, from 0° (green, negative change, decrease in crimes or cases) to 180° (red, positive change, increase in crimes or cases).

While it is very simple, I thought the scale would be a good way to visualize the data (change from one day to the other) for crimes and COVID-19 cases by literally seeing how improving or worsening the situation is getting for each.

Putting it all Together

To put everything together, we combine all three instances of code seen in the above parts of the project; this is all in the code section of this project.

• Additionally, I added conditionals within the crime webhook response handler and case webhook response handler to update the variable for change from the first date to the second date.
• By mapping these change variables from certain bounds that made sense (-9 to 9 change for crimes, -900 to 900 for cases) to the angle restrictions of the servo (0° to 180°), we can display our data.

The final product with the scale appears like this:

When I wake up and plug in the Argon, a single click on the button (after it processes the data) will display the visualization of the data change for COVID-19 cases, a second click will display the visualization of the data change for crimes, and then it will alternate accordingly. In action, it only takes a couple of seconds of waiting to be able to start seeing the "recommendations" of the scale:

Usage & Conclusion

I found this to be very simple to use whenever I checked in the morning. The displays of data across various days I found it useful to know how the situation was outside:

While using it, I thought that possible improvements for the device could be:

• The use of an LED to not rely on knowing that the display for COVID-19 cases come with first click.
• Possible work-arounds for the limitations of the data APIs discussed above (e.g. having an algorithm to compare change across more than 2 dates, etc.).