a3-JoshuaMalcarne-MilesGregg-TimConnors-FedericoGalbiati

Error Analysis/error-graph.Rmd

@@ -0,0 +1,84 @@
+---
+title: "Error Analysis"
+output: html_notebook
+---
+
+```{r}
+library(tidyverse)
+library(ggplot2)
+library(plotly)
+library(gridExtra)
+library(boot)
+
+
+
+# Colorblind-friendly palette, found at 
+# http://www.cookbook-r.com/Graphs/Colors_(ggplot2)/#a-colorblind-friendly-palette
+
+cbbPalette <- c("#E69F00", "#56B4E9", "#009E73", "#F0E442", "#0072B2", "#D55E00", "#CC79A7")
+
+df <- read.csv('a3 csv values.csv')
+
+labelConversion <- setNames(c("Bar", "Pie", "L", "H", "S"),
+                            c("Bar Chart", "Pie Chart", "Luminance", "Hue", "Saturation"))
+
+summaryBarAndPiePlot <- ggplot(df, aes(x = reorder(testType,error), error, fill=testType)) +
+  stat_summary(fun = mean, geom = "bar") +
+  stat_summary(fun.data = mean_cl_normal, geom = "errorbar", fun.args = list(mult = 1)) +
+  ggtitle("Mean Error for all Visualization Types") +
+  xlab("Visualization Type") +
+  theme(plot.title = element_text(hjust = 0.5),
+        legend.position = "none") +
+  scale_fill_manual(values=cbbPalette) +
+  scale_x_discrete(breaks = c("Bar", "Pie", "L", "H", "S"),
+                   labels = c("Bar Chart", "Pie Chart", "Luminance", "Hue", "Saturation"))
+
+print(htmltools::tagList(ggplotly(summaryBarAndPiePlot, tooltip = c("y"))))
+
+summaryBoxPlot <- ggplot(df, aes(x=reorder(testType,error), y=error, fill=testType)) +
+                  geom_boxplot() +
+                  ggtitle("Box Plot of Error for all Visualization Types") +
+                  xlab("Visualization Type") +
+                  theme(plot.title = element_text(hjust = 0.5),
+                        legend.position = "none") +
+                  scale_fill_manual(values=cbbPalette) +
+                  scale_x_discrete(breaks = c("Bar", "Pie", "L", "H", "S"),
+                                   labels = c("Bar Chart", "Pie Chart", "Luminance", "Hue",    "Saturation"))
+
+print(htmltools::tagList(ggplotly(summaryBoxPlot)))
+
+summaryDensityPlot <- ggplot(df, aes(x=error, color=testType)) + 
+  geom_density() +
+  ggtitle("Distribution of Error for all Visualization Types") +
+  scale_color_manual(values= c("#E69F00", "#F0E442", "#009E73", "#0072B2", "#56B4E9"),
+                     labels = c("Bar Chart", "Pie Chart", "Luminance", "Hue", "Saturation")) +
+  guides(color=guide_legend(title="Visualization Type")) +
+  theme(plot.title = element_text(hjust = 0.5))
+
+
+summaryDensityPlot
+
+for (uniqueID in unique(df$userID)) {
+
+  userDF <- subset(df, subset = userID == uniqueID)
+
+  if (nrow(userDF) >= 25) {
+
+    userSummaryPlot <- ggplot(userDF, aes(x=reorder(testType,error), y=error, fill=testType)) +
+    stat_summary(fun = mean, geom = "bar") +
+    stat_summary(fun.data = mean_cl_normal, geom = "errorbar", fun.args = list(mult = 1)) +
+    ggtitle(paste("Error Summary for User", uniqueID, sep = " ")) +
+    xlab("Chart Type") +
+    theme(plot.title = element_text(hjust = 0.5),
+          legend.position = "none") +
+    scale_color_manual(values=cbbPalette) +
+    scale_x_discrete(breaks = c("Bar", "Pie", "L", "H", "S"),
+                                   labels = c("Bar Chart", "Pie Chart", "Luminance", "Hue",    "Saturation"))
+
+    print(htmltools::tagList(ggplotly(userSummaryPlot, tooltip=c("y"))))
+
+  }
+}
+
+```
+

README.md

@@ -1,106 +1,81 @@
-Assignment 3 - Replicating a Classic Experiment  
-===
+[Link to experiment](https://joshuamalcarne.github.io/a3-Experiment/)
 
-For the scope of this project, assume the role of a scientist who runs experiments for a living.
-
-Q: How do we know that bar charts are "better" than pie charts?  
-A: Controlled experiments!
-
-In this assignment you'll implement a simple controlled experiment using some of the visualizations you’ve been building in this class. 
-You'll need to develop support code for the experiment sequence, results file output, and other experiment components. 
-(These are all simple with Javascript buttons and forms.)
-The main goals for you are to a) test three competing visualizations, b) implement data generation and error calculation functions inspired by Cleveland and McGill's 1984 paper, c) run the experiment with 10 participants (or a trial equivalent), and d) do some basic analysis and reporting of the results.
-
-For this assignment you should aim to write everything from scratch. For experimentation it is often necessary to control all elements of the chart.
-You should definitely *reference* demo programs from books or the web, and if you do please provide a References section with links at the end of your Readme.
-
-Going Beyond Cleveland-McGill
----
-
-Several have expressed interest in conducting surveys of various sorts. I encourage you go move beyond Cleveland and McGill if you can think of other interesting visualization experiment designs and corresponding analyses. 
-
-You might study how people interpret COVID visualizations, for example.
-If you decide to go in a custom route, simply contact staff so we can help you set acceptable parameters.
-Basically, we still want you to do a multi-trial study with each participant, to raise the chance that you get solid results.
-How you measure "error" and similar facets also matter. But you can't go wrong with finding a visualization study online to start from :)
-
-Requirements
+[Link to csv of collected data](https://github.com/fedeit/a3-Experiment/blob/a88e0a1a3fd7594926ae5117a175609611ef2848/Error%20Analysis/a3%20csv%20values.csv)
+
+Experiment Description
 ---
+We compared the effectiveness of various visualization types in allowing people to discern the size of different data points. These visualization methods include bar charts, pie charts, hue, saturation, and luminance. The survey consisted of 25 trials, with 5 trials for each visualization type.
+
+When working with bar and pie charts, participants were given five random data points, two of which were highlighted. They were tasked with finding what percentage of the larger data point the smaller data point was and to input their answer into a text box.
 
-- Look it over Cleveland and McGill's original experiment (see the section below) and [watch this video](experiment-example.mp4) to get a sense of the experiment structure and where your visualizations will go.
-- When viewing the example experiment video, note the following:
-    - Trials are in random order.  
-    - Each trial has a randomly generated set of 5-10 data points.  
-    - Two of these data points are marked.  
-    - (Note: the experiment UI and User Experience could be better. Plenty of design achievements here).
-- Implement the data generation code **as described in the Cleveland & McGill paper**. 
-    - The goal is to generate a set of random datapoints (usually 5 or 10, with values be between 0 and 100) and to mark two of them for comparison in the trial. 
-- Add 3 visualizations (i.e. conditions) to your experiment. When you are adding these visualizations, think about *why* these visualizations are interesting to test. In other words, keep in mind a *testable hypothesis* for each of the added visualization. Some good options include bar charts, pie charts, stacked-bar charts, and treemaps. You can also rotate your bar chart to be horizontal or upside-down as one of your conditions. You are encouraged to test unorthodox charts -- radar charts come to mind, but there are MANY possibilities here-- feel free to be creative!
-    - Follow the style from Cleveland and McGill closely (e.g. no color, simple lines) unless you are specifically testing a hypothesis (e.g. color versus no color). Pay attention to spacing between elements like bars. Do not mark bars for comparison using color-- this makes the perceptual task too easy.
-- After each trial, implement code that grades and stores participant’s responses.
-- At the end of the experiment, to get the data, one easy option use Javascript to show the data from the current experiment\* (i.e. a comma separated list in a text box) and copy it into your master datafile. See the Background section below for an example of what this file should look like. (\*Alternately implement a server, if you're experienced with that sort of thing.)
+![Bar Chart](img/a3-bar.PNG)
 
-- Figure out how to calculate "Error", the difference between the true percentage and the reported percentage.
-- Scale this error using Cleveland and McGill’s log-base-2 error equation. For details, see the background section (there’s a figure with the equation). This becomes your “Error” column in the output. Make sure you use whole percentages (not decimal) in the log-base-2 equation. Make sure you handle the case of when a person gets the exact percentage correct (log-base-2 of 1/8 is -3, it is better to set this to 0). 
-- Run your experiment with 10 or more participants, or-- make sure you get at least 200 trials **per visualization type** in total.  
-    - Grab friends or people in the class.   
-    - Run at least 20 trials per visualization type, per participant. This is to ensure that you cover the range of possible answers (e.g. 5%, 10%, ..., 95%)
-- Make sure to save the resulting CSV after each participant. Compile the results into a master csv file (all participants, all trials).
-- Produce a README with figures that shows the visualizations you tested and results, ordered by best performance to worst performance. Follow the modern Cleveland-McGill figure below -- though note that using names instead of icons is fine.
-- To obtain the ranking, calculate and report the average log2Error for each visualization across all trials and participants. This should be straightforward to do in a spreadsheet.
-- Use Bootstrapped 95\% confidence intervals for your error upper and lower bounds. Include these in your figures. Bootstrapped confidence intervals are easily implemented in R + ggplot2 using the `stat_summary` geom. You can also use Excel, Python, or many many other tools. Bootstrapped 95% CIs are **very** useful in modern experiment practice.
-- Include example images of each visualization as they appeared in your experiment (i.e. if you used a pie chart show the actual pie chart you used in the experiment along with the markings, not an example from Google Images).
+![Pie Chart](img/a3-pie.PNG)
 
-## General Requirements
+When working with hue, saturation, and luminance color scales, participants were presented with 3 circles in a line in increasing color value, with the middle circle in equal proportion with the left and right circle within the color scale being used. Participants were prompted to figure out what percentage of the right circle's color value the left circle was. The decision to use 3 circles came about when Professor Harrison suggested that we trim our previously 5 circles down to 3. After researching more about color triads, we decided this made the prompts much easier to interpret.
 
-0. Your code should be forked from the GitHub repo and linked using GitHub pages.
-2. Your project should use d3 to build visualizations. 
-3. Your writeup (readme.md in the repo) should contain the following:
+![Hue](img/a3-hue.PNG)
 
-- Working link to the experiment hosted on gh-pages or some other site.
-- Concise description and screenshot of your experiment.
-- Description of the technical achievements you attempted with this project.
-- Description of the design achievements you attempted with this project.
+![Saturation](img/a3-saturation.PNG)
 
-Background
+![Luminance](img/a3-luminance.PNG)
+
+Data Analysis and Ranking Visualizations
 ---
+Overall, we collected data from over 1,000 trials, with at least 200 trials performed on each visualization type. Error was calculated using McGill and Cleveland's logarithmic scale and plotted into a ranked bar chart. The ordering of the types, from lowest error to highest error, is as followed:
+
+1. Bar Chart: **mean error** of **1.58** +/- 0.068
+2. Pie Chart: **mean error** of **2.50** +/- 0.087
+3. Luminance: **mean error** of **3.58** +/- 0.110
+4. Hue: **mean error** of **3.71** +/- 0.126
+5. Saturation: **mean error** of **3.76** +/- 0.103
+
+Our findings for bar charts and pie charts are consistent with Cleveland and McGill's results as well as the additional crowdsourced results included in the assignment page. Our findings for color scales generally match the convention that luminance scales are more effective than hue scales as well.
 
-In 1984, William Cleveland and Robert McGill published the results of several controlled experiments that pitted bar charts against pies and stacked-bar variants. 
-Their paper (http://www.cs.ubc.ca/~tmm/courses/cpsc533c-04-spr/readings/cleveland.pdf) (http://info.slis.indiana.edu/~katy/S637-S11/cleveland84.pdf) is considered a seminal paper in data visualization.
-In particular, they ran a psychology-style experiment where users were shown a series of randomly-generated charts with two graphical elements marked like this:
+![Summary Bar Plot](img/a3-summary-bar.PNG)
 
-![cleveland bar chart](img/cleveland-bar.png)
+![Summary Box Plot](img/a3-summary-box.PNG)
 
-Participants were then asked, "What percentage is the smaller of the larger?". 
-This was repeated hundreds of time with varying data and charts. 
-By the end of the study, Cleveland and McGill had amassed a large dataset that looked like this:
+![Summary Density Plot](img/a3-summary-density.PNG)
 
-![cleveland table](img/cleveland-table.png)
-
-__Log-base-2 or "cm-error"__: The true percent is the actual percentage of the smaller to the larger, while the reported percent is what participants reported. 
-Cleveland and McGill recognized that their analyses would be biased if they took `abs(ReportedPercent – TruePercent)` as their score for error. 
-To compensate, they came up with a logarithmic scale for error with this equation:
-
-![cleveland equation](img/cleveland-equation.png)
-
-You’ll be implementing this error score as part of the lab. 
-(Hint: it’s not a trick question, this is just to familiarize you with the experiment protocol). 
-With this Cleveland-McGill error score you can better compare the performance of the charts you test to figure out which one performs the best.
-
-As a baseline, compare your average Error scores to the following chart, which include both Cleveland and McGill’s results as well as more recent extensions of this experiment (lower error indicates better performance, and error bars are bootstrapped 95% confidence intervals (`http://en.wikipedia.org/wiki/Confidence_interval#Meaning_and_interpretation`)):
-
-![cleveland results](img/cleveland-results.png)
-
-GitHub Details
+![Summary User Plot](img/a3-user-bar.PNG)
+
+Design Achievements
 ---
-
-- Fork the GitHub Repository. You now have a copy associated with your username.
-- Make changes to index.html to fulfill the project requirements. 
-- Make sure your "master" branch matches your "gh-pages" branch. See the GitHub Guides referenced above if you need help.
-- Edit this README.md with a link to your gh-pages site: e.g. http://YourUsernameGoesHere.github.io/Experiment/index.html
-- Replace this file (README.md) with your writeup and Design/Technical achievements.
-- To submit, make a [Pull Request](https://help.github.com/articles/using-pull-requests/) on the original repository.
-- Name your submission using the following scheme: 
-```
-a3-FirstLastnameMember1-FirstLastnameMember2-FirstLastnameMember3-...
-```
+* Instructions given to match each trial
+    * Helps participants understand the survey better, leading to more accurate results
+* Error message for improper input
+    * Notifies participants when they input a numeric value outside the accepted range of [0.01, 100]
+* Progress tracker at the top of the page
+    * Displays current progress out of 25 total trials
+* Survey graphs follow pleasant color schemes
+    * Used grey-brown-orange for saturation, white-aqua-green for luminance, all colors for hue, and ashen gray-white for pie and bar charts
+* Data scales with content plane size
+    * Allows for consistent proportions of data values across different hardware devices and enables loading on mobile devices
+* Color-blind-friendly scale used for error analysis charts
+    * Helps improve the accessibility of analysis and makes differentiating between data points easier
+* Consistent style maintained in error analysis charts
+    * Colors consistently map to the same visualization type, general color scheme and layout follows the same design
+* Multiple chart types used for error analysis
+    * Bar plot, box plots, and density plots were all used to help communicate and compare the data
+
+Technical Achievements
+---
+* Linked backend to Google Sheets for data collection
+    * Used Google Forms to add trial results to Google Sheets in real-time
+* Perfect score on all Lighthouse Performance Test Scores (see img folder)
+    * Survey website scored 100 in all categories tested
+* Survey only accepts proper input from the user and an error message is given otherwise
+    * Letters and numeric values outside the specified range cannot be inputted and sent as a response for a trial
+* Survey loads and works properly on mobile devices
+    * Increases survey accessibility
+* Trials appear in pseudo-random order
+    * Participants will receive the trials in a random order, including across visualization types. This helps correct for any learning curve participants may have when taking the survey.
+* Graphed data for each trial randomly generated
+    * The two data points being compared are in random proportion to each other, allowing data to be distributed evenly
+* HSL Values Used
+    * Made isolating and testing the color scales of hue, saturation, and luminance possible
+* Error analysis graph generated for each participant who fully completed the survey
+    * Parsed through csv file to display data for unique users, allowing analysis of error data for individual participants
+* Interactive tooltip integrated into error analysis plots
+    * Displays exact values of error for bar plot and specific data points for box plot