ST3009: Statistical Methods for Computer Science
Topics Covered
Part One
- Sum rule of counting
- Product rule of counting
- Number of permutations, with/without repeated objects
- Number of combinations (n choose k)
- Effect of simple constraints on counting e.g. two people must sit together or must not
- Use the above to carry out simple counting calculations
- Definition of sample space and random event
- Set operations, union, intersection, complement and combinations of these
- Axioms of probability
- Plus immediate consequences: P(Ec)=1-P(E), E ⊂ F then P(E)< P(F), equally likely outcomes incl proof.
- Sampling with and without replacement e.g. bag of balls
- Use these to carry out simple calculations of probability of events
- Definition of conditional probability, chain rule
- Use this to carry out simple calculations of conditional prob
- Conditional prob is a probability (satisfies axioms, incl proof.)
- Conditional prob can be used to make predictions
- Marginalisation, incl proof.
- Bayes rule
- Use of Bayes rule in simple examples e.g. HIV testing
- Definition of independence
- Use of definition to test for independence
- Caution re fragility of independence assumptions
- Definition of conditional independence
Part Two
- Definition of a random variable
- Definition of indicator random variable
- Events are linked to values of random variables, so can apply ideas for events directly to RVs (chain rule, Bayes, marginalisation)
- Probability mass function, plotting as histogram
- Cumulative distribution function
- Deriving PMF from CDF and vice versa
- Bernoulli RV
- Binomial RV
- Simple stochastic simulation: generating bernoulli and binomial samples in matlab
- Definition of expected value
- Use this to carry out simple calculations of expected value
- Interpretation of expected value in games of chance/reward
- Expected value of an indicator RV
- Expected value of number of iterations of repeated game (coin tossing etc)
- Limitations of expected value in games of chance/reward e.g. gamblers ruin
- Linearity of expected value incl proof.
- Use of linearity in expected value of sums of random variables
- Expected value of product of independent RVs, incl proof.
- Definition of variance
- var(aX+b)=a2var(X), incl proof.
- Variance of sum of independent RVs, incl proof.
- Definition of joint PMF
- Use this to carry out simple calculations of joint PMF
- Definition of covariance
- Definition of correlation
- Correlation vs causality
- Conditional expectation
- Conditional expectation can be used to make predictions
- Linearity of conditional expectation, incl proof.
- Marginalisation and conditional expectation, incl proof.
- Use of conditional expectation in random sums
Part Three
- Markov inequality
- Chebyshev inequality
- Definition of sample mean, awareness that its an RV
- Expected value and variance of sample mean when sum of iid RVs
- Weak law of large numbers, incl proof via Chebyshev
- Deriving frequency interpretation of probability using the weak law
- Definition of confidence interval
- Use of Chebyshev for confidence interval, and its limitations
- Bootstrapping for confidence intervals, incl matlab implementation
- Definition of continuous RV
- CDF and probability density function for continuous RV
- Probability over interval equal to area under PDF (integral)
- Probability of continuous RV taking a single value is zero
- Calculation of area of rectangles and triangles, plus combinations of these
- Expectation and variance for continuous RVs
- Definition of normal distribution
- Central limit theorem
- Use of CLT to estimate confidence intervals, and its limitations
- Joint CDF for two continuous RVs
- Conditional PDF and chain rule
- Bayes rule for PDFs
- Independence for PDFs
Part Four
- What is a statistical model
- Statistical model used in Logistic Regression
- Linear separability
- Model parameter estimation, Maximum Likelihood and MAP estimates
- Likelihood for Logistic Regression
- Statistical model used in Linear Regression
- Likelihood for Linear Regression
- Fitting polynomials and other nonlinear functions using linear regression
- Closed form Maximum Likelihood solution to scalar Linear Regression
- MAP estimate and Ridge Regression
- Relationship between MAP and Maximum Likelihood estimates as volume of data observed increases
Plus higher order skills including:
- Interpreting simple examples/scenarios and expressing in terms of probability
- Selecting appropriate analysis tools to solve problems
- Self-validation of analysis, e.g. using stochastic simulation to check math analysis and vice-versa
- Awareness of statistical assumptions (including implicit assumptions) and of the potential consequences of their violation e.g. potential fragility of independence assumptions