ISA 419: Data-Driven Security

class: center, middle, inverse, title-slide

.title[
# ISA 419: Data-Driven Security
]
.subtitle[
## 14: Regression Analysis
]
.author[
### <br>Fadel M. Megahed, PhD <br><br> Professor <br> Farmer School of Business<br> Miami University<br><br> <a href="https://twitter.com/FadelMegahed"><svg viewBox="0 0 512 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z" /></svg> <span class="citation">@FadelMegahed</span></a> <br> <a href="https://github.com/fmegahed/"><svg viewBox="0 0 496 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3 0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9 0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4 0 33.7-.3 75.4-.3 83.6 0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6 0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9 0-6.2-1.4-2.3-4-3.3-5.6-2z" /></svg> fmegahed</a> <br> <a href="mailto:fmegahed@miamioh.edu"><svg viewBox="0 0 512 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M476 3.2L12.5 270.6c-18.1 10.4-15.8 35.6 2.2 43.2L121 358.4l287.3-253.2c5.5-4.9 13.3 2.6 8.6 8.3L176 407v80.5c0 23.6 28.5 32.9 42.5 15.8L282 426l124.6 52.2c14.2 6 30.4-2.9 33-18.2l72-432C515 7.8 493.3-6.8 476 3.2z" /></svg> fmegahed@miamioh.edu</a><br> <a href="https://calendly.com/fmegahed"><svg viewBox="0 0 384 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M202.021 0C122.202 0 70.503 32.703 29.914 91.026c-7.363 10.58-5.093 25.086 5.178 32.874l43.138 32.709c10.373 7.865 25.132 6.026 33.253-4.148 25.049-31.381 43.63-49.449 82.757-49.449 30.764 0 68.816 19.799 68.816 49.631 0 22.552-18.617 34.134-48.993 51.164-35.423 19.86-82.299 44.576-82.299 106.405V320c0 13.255 10.745 24 24 24h72.471c13.255 0 24-10.745 24-24v-5.773c0-42.86 125.268-44.645 125.268-160.627C377.504 66.256 286.902 0 202.021 0zM192 373.459c-38.196 0-69.271 31.075-69.271 69.271 0 38.195 31.075 69.27 69.271 69.27s69.271-31.075 69.271-69.271-31.075-69.27-69.271-69.27z" /></svg> Automated Scheduler for Office Hours</a><br><br>
]
.date[
### Spring 2025
]

---

## Quick Refresher of Last Class

✅ Introduce how LLMs (with prompt engineering) can be used to assist in cyber security analytics.

---

## Learning Objectives for Today's Class

- Describe the basic concepts of regression analysis, including the roles of independent and dependent variables.

- Assess regression models using metrics like R-squared and Mean Squared Error and interpret the results.

- Describe the two modeling mindsets: explanatory and predictive.

---

class: inverse, center, middle

# What is regression analysis?

---

## An Overview of Regression Models

- **Given:** `$(x_1, y_1), \ (x_2, y_2), \ \dots , (x_n, y_n)$`.

- Learn a function `$f(x)$` to predict `$y$` given `$x$`.

+ If `$y$` is a continuous variable, we do have a regression type problem.  
  
  + In this class, we will **not** limit ourselves to simple and multiple linear regression models. Hence, we will also explore: (a) more advanced linear regression models (e.g., LASSO), (b) non-linear regression models (e.g., polynomial regression ), and (c) machine learning models like decision trees, random forests, etc.   
      * Simple linear regression: single predictor `$x \rightarrow y$`.  
      * Multiple linear regression: multiple predictors `$x_1, x_2, \dots, x_p \rightarrow y$`.

---

## Simple Linear Regression

.font90[
- Simple linear regression is a statistical method that allows us to summarize and study the relationship between a **response variable** and **one predictor variable**.

`$$Y_i = \beta_0 + \beta_1X_i + \varepsilon_i$$`

where:  
  - `$Y_i$` is the dependent variable for the `$i^{th}$` observation.  
  - `$X_i$` is the independent variable for the `$i^{th}$` observation.  
  - `$\beta_0$` is the intercept, which is the value of `$Y$` when `$X=0$`.  
  - `$\beta_1$` is the slope, which is the change in `$Y$` for a one-unit change in `$X$`.  
  - `$\varepsilon_i$` is the error term, i.e., the difference between observed value of `$Y$` and its predicted value from the model.  
      + It captures the effect of all other factors that influence the dependent variable.  
      + We assume that the error term is ~ `$N(0, \sigma^2)$`.
]

---

## Recall ISA 225: A Simple Linear Regression Example

- Using any software of your choice, analyze the [zeroAccessUFO.csv](https://github.com/fmegahed/isa419/raw/main/data/zeroAccessUFO.csv)  dataset by answering the following questions:

1. Plot the relationship between the number of UFO sightings `$x=$`ufo2010 and the number of zeroAccess infections `$y=$`infections.  
  2. Use the plot to determine if a linear relationship exists between the two variables.  
  3. If a linear relationship exists, fit a simple linear regression model to the data.  
  4. Describe the goodness of fit of the model.  
  5. What is your conclusion?

---

## "Miami Univeristy Links UFO Sightings to ZeroAccess Infections"

- **Question 1:** What happens if you: (a) add `pop` to your model to fit a multiple linear regression model? and (b) fit a single linear regression model using `pop` alone as the predictor?

- **Question 2:** Compute the correlation between `pop` and `ufo2010`.

- **Question 3:** Describe what you have learned from these two exercises.

.can-edit.key-activity12[
- **Q1:** Edit me in your web browser.

- **Q2:** Edit me.

- **Q3:** Edit me.

]

---
class: inverse, center, middle

# On the Difference between R and Python

---

## R and Python Implementations: A Brief Comparison

.font80[
| **Aspect**               | **R (using the 'stats' package)**                                                                          | **Python (using 'PyCaret')**                                                                                             |
|--------------------------|------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------|
| **Primary Library**      | `stats`                                                                                                    | `PyCaret`                                                                                                                 |
| **Modeling Mindset**     | Explanatory (focus on understanding the relationships between variables)                                   | Predictive (focus on accurately predicting outcomes)                                                                     |
| **Data Handling**        | Models often fit on entire dataset without explicit splitting (though splitting can be done manually)      | Automatic splitting of data into training and testing sets, with cross-validation implemented on the training set        |
| **Implications**         | Emphasis on the quality of the model fit and understanding variable relationships                         | Emphasis on the model's predictive performance on unseen data                                                            |
| **Model Extraction**     | Easier to extract and interpret models, facilitating detailed analysis of coefficients and statistical tests | More complex to extract specific model details due to a focus on prediction; models are often part of a larger pipeline |
| **Model Evaluation**     | Emphasis on statistical tests and model fit statistics (e.g., R-squared, p-values)                         | Emphasis on predictive performance metrics (e.g., RMSE, R-squared, MAE)                                                    |
]

---

## The Two Modeling Mindsets: Explanatory Vs. Predictive

- **Explanatory Modeling**:  
  + Focuses on understanding the relationships between variables.  
  + Emphasizes the quality of the model fit and the understanding of variable relationships.  
  + Easier to extract and interpret models, facilitating detailed analysis of coefficients and statistical tests.  
  + Emphasis on statistical tests and model fit statistics (e.g., R-squared, p-values).

- **Predictive Modeling**:  
  + Focuses on accurately predicting outcomes.  
  + Emphasizes the model's predictive performance on unseen data.  
  + More complex to extract specific model details due to a focus on prediction; models are often part of a larger pipeline.  
  + Emphasis on predictive performance metrics (e.g., RMSE, R-squared, MAE).

---

## The Predictive Modeling Mindset

.footnote[
<html>
<hr>
</html>

**Note:** While this video provides a classification example (i.e., a dichotomous dependent variable), the same steps apply to regression ML models. Hence, I am sharing it with you to illustrate the predictive modeling mindset.

]

---

## Comments: Preparing the Data for Modeling

.footnote[
<html>
<hr>
</html>

**Source:** Adi Bronshtein. (2017). [Train/Test Split and Cross Validation in Python](https://towardsdatascience.com/train-test-split-and-cross-validation-in-python-80b61beca4b6).
]

---

## Comments: Preparing the Data for Modeling

.footnote[
<html>
<hr>
</html>

**Source:** Adi Bronshtein. (2017). [Train/Test Split and Cross Validation in Python](https://towardsdatascience.com/train-test-split-and-cross-validation-in-python-80b61beca4b6). Figure is by [Joseph Nelson](https://medium.com/@josephofiowa).
]

---

## Comments: Preparing the Data for Modeling

.footnote[
<html>
<hr>
</html>

---
class: inverse, center, middle

# Recap

---

## Summary of Main Points

By now, you should be able to do the following:

- Describe the basic concepts of regression analysis, including the roles of independent and dependent variables.

- Assess regression models using metrics like R-squared and Mean Squared Error and interpret the results.

- Describe the two modeling mindsets: explanatory and predictive.

---

## 📝 Review and Clarification 📝

1. **Class Notes**: Take some time to revisit your class notes for key insights and concepts.
2. **Zoom Recording**: The recording of today's class will be made available on Canvas approximately 3-4 hours after the end of class.
3. **Questions**: Please don't hesitate to ask for clarification on any topics discussed in class. It's crucial not to let questions accumulate.