Bad plot

Stuff

• any questions?
• assignment #1

About the project

• team project
  
• content
  • should include multiple data sources
    
  • you must check data tos
    
  • at least one data source should be messy
    
  • demonstrate all areas of data science: coding, stats, domain knowledge & communication
• proposal presentation
  • expecting rmd & html slides
    
  • a few slides, 10 min presenting, 5 min questions
    
  • what, why, how
    
  • timeline:
    • need topic approved by me (in writing, via email)
      
    • this may take several iterations
      
    • don’t attempt to get my approval Feb 13
      
    • present on Feb 14

How to do slides in knitr

Assignment 2 - Comments about the ones I have seen so far

One advantage to submitting early is that if I have time, I can have a look and provide feedback before it is due.

• You can interpret the question (Did the winner beat whoever finished second in every polling station?) in two ways: overall winner vs overall 2nd place, or overall winner vs 2nd place in each precinct. Either is fine, just make sure you state which one you are doing, and do it correctly! .
• Again - imagine you are a consultant to the candidate. If you mis-gender them they probably will not hire you again. Google is your friend.
• If you are using a function we have not talked about in class, please provide a reference to tell me where you found out about it.
• If you use an abbreviation for something, or a letter for a parameter of a distribution, make sure the first time you use it you define it. Don’t expect your client to try to figure out what what you mean by mu or MOV.
• For the probability distribution question: think about what the possible values are to help you make your choice. I expect a data science answer, not a dramatic novel.
  • Example: if we were describing the number of people arriving at a bus stop between 8:00 and 8:05 on a Monday morning, I’d probably say something like : “The possible number of people arriving at a bus stop in an interval of time is a non-negative integer, and (generally) people arrive independently at a bus stop. A Poisson distribution is a discrete distribution that counts the number of events (people arriving) in a fixed interval of time (8:00-8:05), where the events occur independently (as the riders likely do), it is a suitable distribution for this problem”. And then I’d put in a reference…
• For the distribution question you dont have to calculate the parameters of the distribution, just tell me which kind of distribution may be appropriate.
• Not just 10 or 20 precincts - all of them. I only did 20 to demo last week because there were (iirc) like 100 in my plot.
• How are you defining ‘margin of victory’? Is it votes? %? Something else? Make sure you say what you mean by it, and that your plots reflect that (i.e. correct units).
  • In many companies there is an implicit acknowledgement that any plot may be copied & sent around in an email - so it has to be stand-alone.

Assignments in general

• Again, write it like a report to your boss
  • no results hanging around without explanation
  • if you show something there should be a reason
  • alternatively, simply saying ‘yes’ is not enough.
    • Plots help, so do sentences.
  • be careful with labeling of axes.
  • be careful of losing data
    • if you start out with 46 rows and end up with 23, you better explain why
  • figure out what you are going to do about ties and make your sentences and code match
  • if you have a reference for something, please indicate where it was used, eg # ref 1
  • if somebody helped you, please say exactly what they helped you with
    
  • hiding code with echo=false is a bad idea because when you are explaining what went wrong to your boss you won’t recall what the hidden code did

How to learn this stuff

• read the code
  
• look up any functions you don’t know
  
• build small examples
  
• execute line by line

How to ask a question

• create a minimal reproducible example
  • minimal: eliminate anything not related to your problem
  • reproducible: includes all needed parts
  • tribble is your friend
• show:
  • your input
  • your output
  • what you expected
  • what you tried

Example: can’t figure out how to exclude rows with filter

library(tidyverse)

## ── Attaching packages ─────────────────────────────────────── tidyverse 1.3.2 ──
## ✔ purrr   1.0.0     ✔ forcats 0.5.2
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag()    masks stats::lag()

library(readxl)
df <- read_excel("statementofvotescastoctober242018.xls",skip=11)

## New names:
## • `` -> `...3`

df %>% tail()

## # A tibble: 6 × 4
##   Precinct                                               Registe…¹ ...3  Cards…²
##   <chr>                                                      <dbl> <lgl>   <dbl>
## 1 Spc Adv 4 99-002 - City Hall                                   0 NA        487
## 2 Spc Adv 4 99-003 - Greenboro Community Centre                  0 NA        515
## 3 Spc Adv 4 99-004 - Ben Franklin Place                          0 NA        600
## 4 Spc Adv 4 99-005 - Minto Recreation Complex-Barrhaven          0 NA        620
## 5 Spc Adv 4 99-006 - Richcraft Recreation Complex-Kanata         0 NA        501
## 6 City / Ville - Total                                      633946 NA     269772
## # … with abbreviated variable names ¹​`Registered\nVoters`, ²​`Cards Cast`

# want to get rid of "City" line

df <- tribble(~precinct, ~votes, #input
  "99-002 - City Hall",0,
  "99-003 - Greenboro Community Centre", 515,
  "99-006 - Richcraft Recreation Complex-Kanata",501,
  "City / Ville - Total", 633946
  )

# want
#   precinct                                     votes
# 1 99-002 - City Hall                               0
# 2 99-003 - Greenboro Community Centre            515
# 3 99-006 - Richcraft Recreation Complex-Kanata   501

df %>% filter(str_detect(precinct, 'city'))

## # A tibble: 0 × 2
## # … with 2 variables: precinct <chr>, votes <dbl>

df %>% filter(str_detect(precinct, 'City'))

## # A tibble: 2 × 2
##   precinct              votes
##   <chr>                 <dbl>
## 1 99-002 - City Hall        0
## 2 City / Ville - Total 633946

# ok, realized I need !
df %>% filter(!str_detect(precinct, 'City'))

## # A tibble: 2 × 2
##   precinct                                     votes
##   <chr>                                        <dbl>
## 1 99-003 - Greenboro Community Centre            515
## 2 99-006 - Richcraft Recreation Complex-Kanata   501

# but how to get back the 'City Hall' row?

Note: frequently the act of reducing your problem to minimal reproducible example will help you figure out what the problem is

Also: if you have to compress you example use a format that can be decompressed free and commonly (i.e. zip). Do not require your helper to install or buy software.

RStudio Projects

• why
  
• how
  
• .R vs .Rmd
• new session, library, etc

How to hide stuff

{r, warning=FALSE, message=FALSE, error=TRUE,eval=TRUE, fig.height=300px}
See https://kbroman.org/knitr_knutshell/pages/Rmarkdown.html

tibble(x=rnorm(100))%>%
  ggplot(aes( x= x))+
  geom_histogram()

## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.

tibble(x=rnorm(100))%>%
  ggplot(aes( x= x))+
  geom_histogram()

## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.

Homework (Individual)

• Out of 8.
  • I want a plot that shows me, for each ward, the fraction of the precincts where the winner of the ward beat the runner up in the ward. BUT, pls exclude all the special and advance precincts.
  • The plot should show all wards, be properly labeled, easy to read, etc.
  • The doc should use the template, and write it up like it is going to your (potential) boss
  • you will need to:
    • write a function to get all the info you need for one ward
      
    • write a function that looks at that one ward and figures out who won and who finished second
      
    • put these two together in a loop that goes through all the wards and with the above info figures out that fraction
      
    • plot the results
      
• ALSO (out of 2):
  • what distribution do you think would be appropriate to describe the fraction of the precincts where the winner beat the runner-up? What parameters do you think would be reasonable for this distribution? Make sure to explain why.
    
• Grading on this homework will be according to the rubric in the course outline.
• due Feb 1, 23:59

Tonight

• We will discuss writing your own functions
• We will discuss writing loops
• We will discuss pulling things out of data frames using positions
• We will discuss a bunch of useful functions

open code_walkthrough.R

Stats

open Stats_New_1.Rmd

open Stats_Coin_Tossing.R

Watch

https://www.youtube.com/watch?v=5Dnw46eC-0o

Less than 15 minutes