Power-Hungry Correlations at their (HPC) Cores

class: left, middle, hide-count
background-image: url(img/tron_legacy_wallpaper_dim.png) 
background-size: 100%

.talk-meta[
.talk-title[
# Power-Hungry Correlations at their (HPC) Cores
]
]

.talk-author[
Eric Nantz
]

.talk-event[
Data Mishaps Night
]

.talk-date[
February 24th, 2022
]

.talk-link[
Slides available at [bit.ly/hpcmishap](https://bit.ly/hpcmishap)
]

.talk-author {
  font-weight: bold;
  font-size: 2.0em;
  line-height: 1em;
}

.talk-event {
  font-weight: bold;
  font-size: 1.5em;
  line-height: 1em;
}

.talk-date {
  font-size: 1.25em;
  line-height: 0;
}

.talk-link {
  font-size: 1.25em;
  position: absolute;
  bottom: 5%;
  left: 4%;
}

.talk-logo {
  width: 320px;
  height: 320px;
  position: absolute;
  top: 50%;
  left: 70%;
}

.event-logo {
  background-image: url('img/datamishaps_hexsticker.png');
  background-size: cover;
  background-repeat: no-repeat;
}

.koopa-img {
  background-image: url('img/koopa.png');
  background-size: cover;
  background-repeat: no-repeat;
}

.koopa-logo {
  width: 150px;
  height: 200px;
  position: absolute;
  top: 38%;
  left: 65%;
}
</style>

---
background-image: url(img/network_background_mod.png) 
background-size: cover

## The Journey Begins

.pull-left[
.big[
Early days of my career in life sciences

Ready to contribute!
]

]

.pull-right[
<img src="img/kevin-flynn-tron.jpg" alt="Kevin Flynn ready for the grid" width = "100%">
]

---
background-image: url(img/network_background_mod.png) 
background-size: cover

# A New Challenge Awaits

.big[
* Thousands of genetic biomarkers
* Explore possible associations with multiple clinical outcomes
* Analyze BIG DATA with <svg aria-hidden="true" role="img" viewBox="0 0 581 512" style="height:1em;width:1.13em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:currentColor;overflow:visible;position:relative;"><path d="M581 226.6C581 119.1 450.9 32 290.5 32S0 119.1 0 226.6C0 322.4 103.3 402 239.4 418.1V480h99.1v-61.5c24.3-2.7 47.6-7.4 69.4-13.9L448 480h112l-67.4-113.7c54.5-35.4 88.4-84.9 88.4-139.7zm-466.8 14.5c0-73.5 98.9-133 220.8-133s211.9 40.7 211.9 133c0 50.1-26.5 85-70.3 106.4-2.4-1.6-4.7-2.9-6.4-3.7-10.2-5.2-27.8-10.5-27.8-10.5s86.6-6.4 86.6-92.7-90.6-87.9-90.6-87.9h-199V361c-74.1-21.5-125.2-67.1-125.2-119.9zm225.1 38.3v-55.6c57.8 0 87.8-6.8 87.8 27.3 0 36.5-38.2 28.3-87.8 28.3zm-.9 72.5H365c10.8 0 18.9 11.7 24 19.2-16.1 1.9-33 2.8-50.6 2.9v-22.1z"/></svg>!
]

---
class: inverse, middle
background-image: url(img/network_background_mod.png) 
background-size: cover
exclude: true

.f6.code[
```r
for (x in biomarker_var) {
  for (y in response_var) {
    cor(x, y)
  }
}
```
]

---
class: inverse, middle
background-image: url(img/network_background_mod.png) 
background-size: cover
exclude: true

.f6.code[
```r
for (x in biomarker_var) {
  for (y in response_var) {
    # permuted data sets
    lapply(
      1:1000, 
      custom_cor, 
      x = x, 
      y = y)
  }
}
```
]

---
class: inverse, middle
background-image: url(img/network_background_mod.png) 
background-size: cover

.f6.code[
```r
for (x in biomarker_var) {
  for (y in response_var) {
    # permuted data sets
    mclapply(
      1:1000, 
      custom_cor, 
      x = x, 
      y = y,
      mc.cores = detectCores())
  }
}
```
]

---
class: middle
background-image: url(img/hpc-cluster.jpg) 
background-size: cover

### Batches to the Grid

```bash
#!/bin/bash
Rscript script.R --batch1
Rscript script.R --batch2
Rscript script.R --batch3
Rscript script.R --batch4
Rscript script.R --batch5
Rscript script.R --batch6
...
Rscript script.R --batch1000
```

---
class: inverse hide-count animated fadeIn

<ul class="nobull">
<li>
<div class="container">
  <img src="img/ed_dillinger.png" alt="Ed, the lead from research IT">
  <p>Hello Eric, this is Ed from the research IT team. Did you submit a job array with ID 8675309?</p>
</div>
</li>

<li>
<div class="container darker">
  <img src="img/eric_happy.png" alt="Eric, the happy statistician" class="right">
  <p>Hello Ed, nice to meet you! Yes, I submitted that job earlier today. I guess it is taking longer than I anticipated to complete.</p>
</div>
</li>

<li>
<div class="container">
  <img src="img/ed_dillinger.png" alt="Ed, the lead from research IT">
  <p>Your HPC jobs have brought the entire HPC cluster down.</p>
</div>
</li>

---
background-image: url(img/network_background_mod.png)
background-size: cover

## Small Glitch ...

.pull-left[
.large[
* Compute servers running my jobs had memory and CPU usage **maxed out**
* All other jobs were stopped, even the really important ones
* Emergency server reboot was required
]
]

.pull-right[
<img src="img/star-trek-explode.gif" alt="Star Trek gif of destroying the ship computer" width = "80%">
]

---
background-image: url(img/network_background_mod.png) 
background-size: cover

## New Chapter Begins

.pull-left[
.large2[
* After correcting the mistake, eager to learn better approaches to HPC computing
* Launched my most valuable relationships in the company
* Kickstarted amazing innovations in our statistical computing infrastructure
]
]

.pull-right[
<img src="img/tron_group.png" alt="The Tron programs unite" width = "100%">
]

---
background-image: url(img/network_background_mod.png) 
background-size: cover

# Thank you!

.twitter-logo[
## <svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:2em;width:2em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:steelblue;overflow:visible;position:relative;"><path d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z"/></svg>
]
.twitter-text[
.large2[
[@theRcast](https://twitter.com/thercast)
]
]

.github-logo[
## <svg aria-hidden="true" role="img" viewBox="0 0 496 512" style="height:2em;width:1.94em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:steelblue;overflow:visible;position:relative;"><path d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3 0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9 0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4 0 33.7-.3 75.4-.3 83.6 0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6 0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9 0-6.2-1.4-2.3-4-3.3-5.6-2z"/></svg>
]

.github-text[
.large2[
[@rpodcast](https://github.com/rpodcast)
]
]

.podcast-logo[
## <svg aria-hidden="true" role="img" viewBox="0 0 352 512" style="height:2em;width:1.38em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:steelblue;overflow:visible;position:relative;"><path d="M176 352c53.02 0 96-42.98 96-96V96c0-53.02-42.98-96-96-96S80 42.98 80 96v160c0 53.02 42.98 96 96 96zm160-160h-16c-8.84 0-16 7.16-16 16v48c0 74.8-64.49 134.82-140.79 127.38C96.71 376.89 48 317.11 48 250.3V208c0-8.84-7.16-16-16-16H16c-8.84 0-16 7.16-16 16v40.16c0 89.64 63.97 169.55 152 181.69V464H96c-8.84 0-16 7.16-16 16v16c0 8.84 7.16 16 16 16h160c8.84 0 16-7.16 16-16v-16c0-8.84-7.16-16-16-16h-56v-33.77C285.71 418.47 352 344.9 352 256v-48c0-8.84-7.16-16-16-16z"/></svg>
]

.podcast-text[
.large2[
[r-podcast.org](https://r-podcast.org)
]
]

.sds-logo[
## <svg aria-hidden="true" role="img" viewBox="0 0 576 512" style="height:2em;width:2.25em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:steelblue;overflow:visible;position:relative;"><path d="M549.655 124.083c-6.281-23.65-24.787-42.276-48.284-48.597C458.781 64 288 64 288 64S117.22 64 74.629 75.486c-23.497 6.322-42.003 24.947-48.284 48.597-11.412 42.867-11.412 132.305-11.412 132.305s0 89.438 11.412 132.305c6.281 23.65 24.787 41.5 48.284 47.821C117.22 448 288 448 288 448s170.78 0 213.371-11.486c23.497-6.321 42.003-24.171 48.284-47.821 11.412-42.867 11.412-132.305 11.412-132.305s0-89.438-11.412-132.305zm-317.51 213.508V175.185l142.739 81.205-142.739 81.201z"/></svg>
]

.sds-text[
.large2[
[shinydevseries.com](https://shinydevseries.com)
]
]

.twitch-logo[
## <svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:2em;width:2em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:steelblue;overflow:visible;position:relative;"><path d="M391.17,103.47H352.54v109.7h38.63ZM285,103H246.37V212.75H285ZM120.83,0,24.31,91.42V420.58H140.14V512l96.53-91.42h77.25L487.69,256V0ZM449.07,237.75l-77.22,73.12H294.61l-67.6,64v-64H140.14V36.58H449.07Z"/></svg>
]

.twitch-text[
.large2[
[twitch.tv/rpodcast](https://twitch.tv/rpodcast)
]
]

.rweekly-logo[
## <svg aria-hidden="true" role="img" viewBox="0 0 576 512" style="height:2em;width:2.25em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:steelblue;overflow:visible;position:relative;"><path d="M552 64H88c-13.255 0-24 10.745-24 24v8H24c-13.255 0-24 10.745-24 24v272c0 30.928 25.072 56 56 56h472c26.51 0 48-21.49 48-48V88c0-13.255-10.745-24-24-24zM56 400a8 8 0 0 1-8-8V144h16v248a8 8 0 0 1-8 8zm236-16H140c-6.627 0-12-5.373-12-12v-8c0-6.627 5.373-12 12-12h152c6.627 0 12 5.373 12 12v8c0 6.627-5.373 12-12 12zm208 0H348c-6.627 0-12-5.373-12-12v-8c0-6.627 5.373-12 12-12h152c6.627 0 12 5.373 12 12v8c0 6.627-5.373 12-12 12zm-208-96H140c-6.627 0-12-5.373-12-12v-8c0-6.627 5.373-12 12-12h152c6.627 0 12 5.373 12 12v8c0 6.627-5.373 12-12 12zm208 0H348c-6.627 0-12-5.373-12-12v-8c0-6.627 5.373-12 12-12h152c6.627 0 12 5.373 12 12v8c0 6.627-5.373 12-12 12zm0-96H140c-6.627 0-12-5.373-12-12v-40c0-6.627 5.373-12 12-12h360c6.627 0 12 5.373 12 12v40c0 6.627-5.373 12-12 12z"/></svg>
]

.rweekly-text[
.large2[
[rweekly.org](https://rweekly.org)
]
]