ISA 419: Data-Driven Security

class: center, middle, inverse, title-slide

.title[
# ISA 419: Data-Driven Security
]
.subtitle[
## 08: Visualizing Data with Pandas
]
.author[
### <br>Fadel M. Megahed, PhD <br><br> Professor <br> Farmer School of Business<br> Miami University<br><br> <a href="https://twitter.com/FadelMegahed"><svg viewBox="0 0 512 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z" /></svg> <span class="citation">@FadelMegahed</span></a> <br> <a href="https://github.com/fmegahed/"><svg viewBox="0 0 496 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3 0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9 0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4 0 33.7-.3 75.4-.3 83.6 0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6 0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9 0-6.2-1.4-2.3-4-3.3-5.6-2z" /></svg> fmegahed</a> <br> <a href="mailto:fmegahed@miamioh.edu"><svg viewBox="0 0 512 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M476 3.2L12.5 270.6c-18.1 10.4-15.8 35.6 2.2 43.2L121 358.4l287.3-253.2c5.5-4.9 13.3 2.6 8.6 8.3L176 407v80.5c0 23.6 28.5 32.9 42.5 15.8L282 426l124.6 52.2c14.2 6 30.4-2.9 33-18.2l72-432C515 7.8 493.3-6.8 476 3.2z" /></svg> fmegahed@miamioh.edu</a><br> <a href="https://calendly.com/fmegahed"><svg viewBox="0 0 384 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M202.021 0C122.202 0 70.503 32.703 29.914 91.026c-7.363 10.58-5.093 25.086 5.178 32.874l43.138 32.709c10.373 7.865 25.132 6.026 33.253-4.148 25.049-31.381 43.63-49.449 82.757-49.449 30.764 0 68.816 19.799 68.816 49.631 0 22.552-18.617 34.134-48.993 51.164-35.423 19.86-82.299 44.576-82.299 106.405V320c0 13.255 10.745 24 24 24h72.471c13.255 0 24-10.745 24-24v-5.773c0-42.86 125.268-44.645 125.268-160.627C377.504 66.256 286.902 0 202.021 0zM192 373.459c-38.196 0-69.271 31.075-69.271 69.271 0 38.195 31.075 69.27 69.271 69.27s69.271-31.075 69.271-69.271-31.075-69.27-69.271-69.27z" /></svg> Automated Scheduler for Office Hours</a><br><br>
]
.date[
### Spring 2025
]

---

## Quick Refresher of Last Week's Class

✅ Ensure that your imported data is **technically correct** (rename columns and fix `dtypes`)

✅ Understand how to change the unit of analysis by grouping and aggregating data.

✅ Use the `agg()` function to do aggregations on grouped data.

---

## Learning Objectives for Today's Class

- Create quick visualizations using the `plot` method from [pandas](https://pandas.pydata.org/docs/user_guide/visualization.html) (with an understanding of the effect of different backends).

- Utilize `auto-viz` type plots to create a quick EDA of your data.

---
class: inverse, center, middle

# Plotting with Pandas

---

## Our Data

- We will use the `merged_ips` data set from a previous class to demonstrate how to plot data in pandas.

.font80[

``` python
import pandas as pd

toxic_ips = pd.read_csv(
  "https://raw.githubusercontent.com/fmegahed/isa419/main/data/listed_ip_90_all.csv", 
  header = None, names = ['ip', 'frequency', 'lastseen']
)

geolocation = pd.read_csv(
  'https://raw.githubusercontent.com/fmegahed/isa419/main/data/ip_geolocation.csv', 
  names = ['ip', 'country', 'city', 'latitude', 'longitude']
)

merged_ips = (
  toxic_ips
  .merge(right = geolocation, how = 'left', on ='ip')
  .dropna()
  .assign( lastseen = lambda df: df['lastseen'].astype('datetime64[ns]') )
)
merged_ips.dtypes[0:3]
```

```
## ip                   object
## frequency             int64
## lastseen     datetime64[ns]
## dtype: object
```
]

---

## Plotting with Pandas

- The `plot` method in pandas is a wrapper around `matplotlib` (by default) and is a quick way to visualize data.

- The `plot` method is available on both `Series` and `DataFrame` objects.

![pandas_plot](data:image/png;base64,#https://pandas.pydata.org/docs/_images/04_plot_overview.svg)

.footnote[
<html>
<hr>
</html>

**Source:** The figure is from the [Pandas Documentation](https://pandas.pydata.org/docs/getting_started/intro_tutorials/04_plotting.html)
]

---

## Class Activity to Assess your Understanding so Far

.panelset[

.panel[.panel-name[Task]
- Write Python code to produce a data frame containing the total number of toxic IP frequencies by country.

- Then, identify the top 10 countries with the highest toxic IP frequencies.
]

.panel[.panel-name[Hints]
<html>
<details>
  <summary>Please let me know in class if you need any hints.</summary>
  <p>Use the `groupby` method to group the data by country and then use the `agg` method to aggregate the data. Then, use the <i>sort_values</i> method to sort the data in descending order by setting the <i>ascending</i> parameter to `False`.</p>
</html>
]

.panel[.panel-name[Solution]
.font80[

```
##                  frequency
## country                   
## Georgia            1712171
## Ukraine             928770
## Russia              661849
## Germany             444641
## Canada              443046
## United States       230434
## Finland             188501
## Poland              152569
## India               120881
## The Netherlands      84973
```
]
]

]

---

## Plotting with Pandas (Plot `kind`)

---

## Plotting with Pandas (`line` Plot)

.pull-left[
.font80[

.center[**Data Prep:**]

``` python
# Aggregating the frequencies by day
daily_freq = (
  merged_ips
  .groupby(merged_ips['lastseen'].dt.date)
  .agg(sum_freq = ('frequency', 'sum'))
  .reset_index() # to have last seen as col
  .rename(columns = {'lastseen': 'date'})
)

daily_freq.head(n=2)
```

```
##          date  sum_freq
## 0  2023-10-26      8952
## 1  2023-10-27      4642
```
]
]

.pull-right[
.font80[

.center[**Plotting:**]

``` python
daily_freq.plot(
  x = 'date', y = 'sum_freq', kind = 'line',
  title = 'Toxic IP Frequencies by Day',
  xlabel = 'Date', ylabel = 'Frequency',
  figsize = (10, 4)
  )
```

<img src="data:image/png;base64,#08_data_viz_intro_files/figure-html/line_plot-1.png" width="960" style="display: block; margin: auto;" />
]
]

---

## Plotting with Pandas (`bar` Plot)

.pull-left[

.font80[

.center[**Data:**]

``` python
# Aggregating the frequencies by country
country_freq = (
  merged_ips
  .groupby('country')
  .agg(sum_freq = ('frequency', 'sum'))
  .sort_values('sum_freq', ascending = False)
  .head(10)
  .reset_index()
)
country_freq.head(n=2)
```

```
##    country  sum_freq
## 0  Georgia   1712171
## 1  Ukraine    928770
```
]
]

.pull-right[

.font80[
.center[**Plotting:**]

``` python
country_freq.plot(
  x = 'country', y = 'sum_freq', kind = 'barh',
  title = 'Top 10 Countries with Toxic IP Freq',
  xlabel = 'Country', ylabel = 'Frequency',
  figsize = (10, 6),
  color = 'red'
  )
```

<img src="data:image/png;base64,#08_data_viz_intro_files/figure-html/bar_plot-3.png" width="960" style="display: block; margin: auto;" />
]
]

---

## Plotting with Pandas (`scatter` Plot)

.pull-left[
.font80[
.center[**Data:**]

``` python
country_freq.head(n=5)
```

```
##    country  sum_freq
## 0  Georgia   1712171
## 1  Ukraine    928770
## 2   Russia    661849
## 3  Germany    444641
## 4   Canada    443046
```
]
]

.pull-right[
.font80[
.center[**Plotting:**]

``` python
country_freq.plot(
  # scatter plots are better with two numeric vars 
  # (this example is for illustration only)
  x = 'country', y = 'sum_freq', kind = 'scatter',
  title = 'Top 10 Countries with Toxic IP Freq',
  xlabel = 'Country', ylabel = 'Frequency',
  figsize = (12, 6)
  )
```

<img src="data:image/png;base64,#08_data_viz_intro_files/figure-html/scatter_plot-5.png" width="1152" style="display: block; margin: auto;" />
]
]

---

## Class Activity to Assess your Understanding so Far

.panelset[

.panel[.panel-name[Task]

- Read the [`simulated_attack_data.csv`](https://raw.githubusercontent.com/fmegahed/isa419/main/data/simulated_attack_data.csv) file into a pandas data frame.

- Then, answer the questions in the next three tabs.  
]

.panel[.panel-name[Task 1]

- Create a histogram of the `Attempt Count` variable.

- What does the histogram tell you about the `Attempt Count` variable?

.can-edit.key-activity12a[
- Edit me to answer the question above.
]
]

.panel[.panel-name[Task 2]
 - Create a scatter plot of the `Source Latitude` and `Source Longitude` variables.  
 
]

.panel[.panel-name[Task 3]
 - Utilize [this Stack Overflow thread](https://stackoverflow.com/a/67428742/10156153) to convert the scatter plot of the `Source Latitude` and `Source Longitude` variables into an interactive symbols map.

]
]

---
class: inverse, center, middle

# Automated Viasualizations in Python

---

## The `ydata-profiling` Package

Data quality profiling and exploratory data analysis (EDA) are crucial steps in any business analytics application.

[ydata-profiling](https://docs.profiling.ydata.ai/latest/) automates and standardizes the generation of detailed reports, complete with statistics and visualizations.

The significance of the package lies in how it streamlines the process of understanding and preparing data for analysis in a single line of code!

---

## Usage of `ydata-profiling`

``` python
import pandas as pd
*from ydata_profiling import ProfileReport

*profile = ProfileReport(sim_attack_df, title="Pandas Profiling Report", explorative=True)

# the next line is needed since I am not using Colab for making the slides
profile.to_file("../../figures/sim_attack_data_report.html") 
```

---

## Output of `ydata-profiling`

---

## Class Activity

.panelset[

.panel[.panel-name[Task]

- Identify a similar package to `ydata-profiling` in Python.

- Then, use the package to generate a report for the `merged_ips` data set.

- Share the report with your neighboring classmate.

- Discuss the insights and visualizations in your approach(es).

]
]

---

class: inverse, center, middle

# Recap

---

## Summary of Main Points

By now, you should be able to do the following:

- Create quick visualizations using the `plot` method from [pandas](https://pandas.pydata.org/docs/user_guide/visualization.html) (with an understanding of the effect of different backends).

- Utilize `auto-viz` type plots to create a quick EDA of your data.

---

## 📝 Review and Clarification 📝

1. **Class Notes**: Take some time to revisit your class notes for key insights and concepts.
2. **Zoom Recording**: The recording of today's class will be made available on Canvas approximately 3-4 hours after the end of class.
3. **Questions**: Please don't hesitate to ask for clarification on any topics discussed in class. It's crucial not to let questions accumulate.