text:
## Using service workers with Cloudinary!

### What's inside?
The main directory consists of an `index.js` script and the `templates` directory containing service worker file and its configuration.

This is published to NPM as `@cloudinary/create-cloudinary-service-worker-lab` allowing it to be run with the `npx` or `yarn create` commands.

Example:
```bash
yarn create @cloudinary/cloudinary-service-worker-lab
npx @cloudinary/create-cloudinary-service-worker-lab
```

Those commands will copy the service worker files to the desired location (working directory by default, can be overriden with `-p` | `--path` parameter).

Example:
```bash
yarn create @cloudinary/cloudinary-service-worker-lab --path my-app/src
npx @cloudinary/create-cloudinary-service-worker-lab --path my-app/src
```

The script accepts also a `--verbose` flag to provide additional runtime information. See `--help` for more options.

### Demo project
Besides the main package this repository contains also the demo application showing the example use case. It includes a small website with images that does not use Cloudinary. You can run it locally, use the `@cloudinary/create-cloudinary-service-worker-lab` script and enable it to see the difference.

In order to check it yourself please clone this repository and run the following.

```bash
cd demo
yarn
yarn start

# if you prefer to use npm please run the following
cd demo
npm install
npm start
```

You should have a working page without service workers running locally. In order to enhance it with the power of the Cloudinary service worker run the following:

```bash
yarn create @cloudinary/cloudinary-service-worker-lab

# if you prefer to use npm please run the following
npx @cloudinary/create-cloudinary-service-worker-lab
```

The service worker is now copied to your (demo) project directory. Please edit `index.html` and add the service worker setup script `cloudinaryServiceWorkerSetup.js`.

```html
<script src="./cloudinaryServiceWorkerSetup.js"></script>
```

You're done! Once you refresh the page the service worker should be registered. Once you refresh the page for the second time all assets will be served through cloudinary. You can edit the config JSON inside the `cloudinaryServiceWorkerSetup.js` file to customize the behaviour.

# Disclaimer
Cloudinary makes no warranties regarding the currency, accuracy or completeness of the information provided herein (“Information”). Such Information could include technical or other mistakes, inaccuracies or typographical errors and may be out of date, and Cloudinary makes no commitment to update it.
Cloudinary expressly reserves the right without prior notice to make changes to the Information made available or to cease publication temporarily or permanently.
Cloudinary assumes no responsibility for errors or omissions in the Information.
In no event will Cloudinary be liable to you or any third parties for any special, punitive, incidental, indirect or consequential damages of any kind, or any damages whatsoever, including, without limitation, those resulting from loss of use, data or profits, whether or not Cloudinary has been advised of the possibility of such damages, and on any theory of liability, arising out of or in connection with the use of the Information.
The use of the Information is done at your own discretion and risk and with agreement that you will be solely responsible for any damage to your computer system or loss of data that results from such activities. The Information will not create any warranty for the software provided or made available by Cloudinary in connection with the Information.


text:
---
title: "Perscotty Pt II"
date: 2015-05-04
layout: post
categories: programming
---

## Some updates

(this is part two of three: [one](programming/2015/05/02/scotty_and_persistent.html) and [three](/programming/2015/05/10/perscotty_pt_iii.html) are linked)

The [reddit thread](https://www.reddit.com/r/haskell/comments/34nxtu/scotty_and_persistent_a_beginners_voyage/) about my [previous post](http://www.parsonsmatt.org/programming/2015/05/02/scotty_and_persistent.html) generated good discussion and advice. I'm going to attempt to work through them now.

[This guide](http://taylor.fausak.me/2014/10/21/building-a-json-rest-api-in-haskell/) contains a lot of information on what this all should look like in the end. I'll be taking a decent bit of information from that.

## Pooling

The most urgent issue is that my database pool is getting recreated every time I make a query, and then closed. Noo! Instead, I need to create a pool, and pass that to the database functions so they can efficiently reuse the resources. Also, it's not really necessary to run the migrations from within the server, so we'll extract that out. I'm also going to unqualify the scotty import to make the code a bit more readable.

```haskell
runDb pool query = liftIO (runSqlPool query pool)

main :: IO ()
main = do
    pool <- runStdErrLoggingT $ createPostgresqlPool connStr 10
    runDb pool doMigrations
    rubDb pool doDbStuff
    scotty 3000 $ do -- ...
```
Pool acquired! Now we can delete the `inAppDb` function. Let's get the inHandlerDb stuff working too.  I was kind of expecting a type mismatch that would require another function to be made, but this didn't. Let's inspect the inferred types?

```haskell
inAppDb :: SqlPersistM a -> ScottyT T.Text IO a
inHandlerDb :: SqlPersistM a -> ActionT T.Text IO a
runDb :: MonadIO m => SqlPersistT IO a -> Pool SqlBackend -> m a
```

It looks like the reason that `runDb` is more general is because the inferred type doesn't restrict it to a given monad, and it is expecting a transformer `SqlPersistT` instead of the `SqlPersistM`.

## Fat Stacks of Monads

Passing around the `pool` is kind of annoying, especially when the application gets more complex. Let's try to make a helper function that will encapsulate that process.

```haskell
main = do
    pool <- etc...
    let runDb' = runDb pool
    runDb' doMigrations
    ...
```
This works, at first! Unfortunately, it only works when we keep it in the top level. It doesn't let us use this function in the application. The types don't line up. Experiment with it a bit: the first place that you *use* the `runDb'` function is what coerces the type, and the type of function to run the database inside the application is incompatible with the type of function to run the database outside of the database. Let's use `ghc-mod` to inspect the inferred type of `runDb'` in the above context:

```haskell
SQLPersistT IO -> IO ()
```
And the type of `runDb pool` in the above context:

```haskell
SqlPersistT [Entity Blogpost] -> Web...ActionT T.Text IO [Entity Blogpost]
```
In SQLPersistT and ActionT, the T indicates that these are monad transformers. A monad transformer gets stacked on top of another monad, allowing you to access two monads. And monad transformers are themselves monads, so you can stack as many as you want! So we want to somehow generalize `SqlPersistT [Entity Blogpost] -> ActionT Text IO [Entity Blogpost]`. I tried a number of possible avenues for that, but wasn't able to derive a function that would work generically.

## Reader

The Haskell idiom for implicitly threading some read-only information throughout a program is the Reader monad. The JSON API linked above uses this, along with a Config data type, to build the application up. Let's start small and build something similar. Let's keep the `Config` data type small, and just store the connection pool for now:

```haskell
data Config = Config { getPool :: ConnectionPool }
```
Next up is defining the `Reader` monad for this. The bottom of the stack is `IO`, so our `Reader` will read from Config and sit on top of `IO`. Here's the code (pulled from Taylor Fausak's post):

```haskell
newtype ConfigM a = ConfigM
    { runConfigM :: ReaderT Config IO a
    } deriving (Applicative, Functor, Monad,
                MonadIO, MonadReader Config)
```

We had to add a few imports up top to get this to work, and add the `mtl` library to the cabal file. Alright! How do we actually use this thing? It turns out, we need to stop calling `scotty` and call something else entirely. Taylor's guide calls `scottyOptsT` which has a pretty full configuration set. For right now, I'd like to keep it a bit simpler. Let's explore the [Hackage documentation](https://hackage.haskell.org/package/scotty-0.9.1/docs/Web-Scotty-Trans.html) for scotty's types and see what we can do. `scottyT` looks like the simplest of the bunch, so let's run with that.

Before we get too crazy, let's make sure we can get `scottyT` working just by passing `id` in. Our main function now reads:

```haskell
main = do
    pool <- runStdoutLoggingT $ createPostgresqlPool connStr 10
    runDb pool migrations
    runDb pool doDbStuff
    scottyT 3000 id id $ do
      -- ...
```
This works! Awesome! So, what's going on with those `id`s there? Inspecting the type with ghc-mod gives us `IO a -> IO a` for the first one, and `IO Response -> IO Response`. Cool. Let's add a line `let c = Config pool` under the pool declaration to make our Config data. Taylor's guide has the following function, which I'm going to copy in: `let r m = runReaderT (runConfigM m) c`. The text about that function reads:

> This takes Scotty’s monad m and adds the ability to read our custom config c from it. This is called a monad transformer stack. It allows us to use any monad in the stack. So after adding our reader monad, we can both deal with requests (using Scotty’s monad) and read our config (using our monad).

Cool! Let's change `id` to `r` in both of those and see what happens...

## BOOM

Type errors! Type errors everywhere! While looking at the hackage documentation above, I noticed that all of the normal methods were duplicated in the Web.Scotty.Trans package. Let's swap out the Scotty version of those functions with the ScottyT versions. And now we're getting entirely different type errors! We're missing an instance for `ScottyError`. So let's break the application code into it's own function, give that a type signature, and see what happens.

Now we're doing `scottyT 3000 r r application` and defining `application` below. The `pool` went out of scope. Let's just be a tiny bit lazy, and comment out the whole body of that function, and just do a basic `"hello world"` for now. We want to get the monad stack working, and database access will be easy as pie after that. Here's what our main function and app functions look like now:

```haskell
main = do
    pool <- runStdOutLogggingT $ createPostgresqlPool connStr 10
    let cfg = Config pool
        r m = runReaderT (runConfigM m) cfg
    scottyT 3000 r r app

app :: ScottyT T.Text ConfigM ()
app = S.get "/" $ S.html "Hello world"
```
Now we're getting a new and entirely vexing type error:

> Couldn't match type ‘a’ with Response’
>
> ‘a’ is a rigid type variable bound by a type expected by the context:
>
> ConfigM a -> IO a at Main.hs:60:5
>
> Expected type: ConfigM a -> IO a
> Actual type: ConfigM Response -> IO Response
>
> In the second argument of ‘scottyT’, namely ‘r’
>
> In a stmt of a 'do' block: scottyT 3000 r r app

(the actual error said "wai-<IP_ADDRESS>:Network.Wai.Internal.Response", and I trimmed it for readability)

## A detour

I dug around the internet for *hours* trying to find the solution to this, and I never really got there. I tried so many things, and nothing fixed it. Finally, I decided I'd rip out as much code as possible for a minimal reproduction to be able to ask the Internet, and something magical happened...

First, I deleted all the code except for stuff directly required for the `main` and `app` functions above. `ghc-mod` let me know about a bunch of unused imports, so I trimmed the import list down until it was *the bare necessities*. ghc-mod was kind enough to let me know that I had a bunch of unused language pragmas, so I removed them. At this point, the type error goes away, and everything works. *What*. What. I don't even know. I re-add them, problem recurs. I remove them one-by-one, and the problem was evidently with the `GADTs` language pragma, which was used by Persistent's Template Haskell implementation. Weird. Let this be a warning -- break your code into modules, and localize things as much as possible!

### Break out the data model

`touch Model.hs`, throw all the Persistent stuff in there (minus the stuff required for `ConnectionPool`, etc) and start taking language pragmas out of Main. This time, GADTs didn't fix it, but TypeFamilies did. Well, whatever. Our minimal HelloWorld with the right monad is finally working.

The repository in the current state is available [here](https://github.com/parsonsmatt/scotty-persistent-example/tree/monad-stacks). You can look through the commit history and see the incremental changes that I made. This blog post is already way longer than is necessary or normal, so I'll actually get the little demo working with the database next time.

branch_name: refs/heads/master

extension: markdown

license_type: permissive


text:
---
templateKey: blog-post
title: Angular Animations
date: 2014-10-22T20:13:35.688Z
description: I've typed ng-something enough times to make me want to blow my brains out.
tags:
  - learning
---

Couldn't get Angular animations working. Why? Because I hadn't defined 'ngAnimate' in my app module.

I wanted the receipt to animate in, a simple slide-down-and-in type thing. Most of the angular tutorials look god awful, so this is how you want to do it ifyou're not writing plain CSS:

```
.receipt {
// Animations
&.ng-hide-remove {
  @include animation(fade-in-and-down .35s ease-in);
  // @include animation-fill-mode(forwards);
    }
  }
}
```

Angular works by applying .ng-hide-remove, or .ng-hide-add for when you're toggling a box. (Seems like a roundabout naming schema, but whatever).

Unfortunately applying Bourbon's @include animation-fill-mode(forwards) didn't work in the way I was expecting it to. However, applying a transform: translateY(10px); in the base class did. Meh.

license: MIT

repository: ThomasAndrewMacLean/portfolio

extension: md

gha_license_id: MIT

repo_name: ThomasAndrewMacLean/portfolio


text:
# CheckSumStats

CheckSumStats is an R package for checking the accuracy of meta- and summary-data from genetic association studies prior to their use in post-GWAS applications, such as two-sample Mendelian randomisation. For example, the package provides tools for checking that the reported effect allele and effect allele frequency columns are correct. It also checks for possible errors in the reported effect sizes that might introduce bias into downstream analyses.

## Installation

To install the latest version of CheckSumStats, perform as normal:

``` r
install.packages("devtools")
devtools::install_github("MRCIEU/CheckSumStats")
```

## General overview

This package exploits three groups of single nucleotide polymorphisms (SNPs) in order to identify potential errors or issues: 1) **A MAF 1KG reference set**. This is a set of 2297 SNPs that have the same minor allele across the 1000 genomes super populations and that have a minor allele frequency between 0.1 and 0.3; 2) **GWAS catalog top hits**. These are SNPs that are strongly associated with the trait of interest in the GWAS catalog; and 3) the **GWAS top hits**. These are SNPs that are strongly associated with the trait of interest in the GWAS or genetic association study of interest.

The package performs 4 types of checks:

1.  confirm the identity of the effect allele frequency column. This step relies on comparison of the MAF reference set between the GWAS study and the 1000 genomes reference dataset.
2.  confirm the identity of the effect allele column. This step relies on comparison of GWAS catalog top hits between the GWAS study and the GWAS catalog.
3.  assess the ancestral origins of the study. This step relies on comparison of the MAF reference set between the GWAS study and 1000 genomes reference dataset.
4.  identify possible errors in the reported effect sizes. This step exploits the GWAS top hits and the GWAS catalog top hits.

# Example 1. Check the summary and meta data from a disease GWAS

In this example we show how the package can be used to check the quality of reported summary and meta data from case-control genome-wide association studies. Example 2 focuses on genome-wide association studies of non-disease continuous traits.

## Step 1. Extract and format the outcome summary data

The first step is to make a list of SNP rsids. Let's assume our trait of interest is glioma. Let's identify the top GWAS hits for glioma in the GWAS catalog. We will search using both the reported trait name as well as the trait experimental factor ontology (EFO). It is advisable to search for both because not all studies in the GWAS catalog have up-to date EFO annotations. Searching for both therefore maximises the number of retrieved hits.

``` r
library(mrQC)
snplist<-make_snplist(efo="glioma",trait="glioma",ref1000G_superpops=FALSE)
head(snplist)
#> [1] "rs2736100" "rs2853676" "rs891835"  "rs4295627" "rs4977756" "rs498872"
length(snplist)
#> [1] 54
```

We see that searching on the glioma EFO and glioma trait retrieves 54 unique SNP rsids. We could also have searched on the glioma efo\_id (EFO\_0005543), which would have retrieved the same SNPs as searching on the "glioma" efo.

``` r
snplist<-make_snplist(efo_id="EFO_0005543",ref1000G_superpops=FALSE)
```

In the above examples we set ref1000G\_superpops to FALSE. We now set this to TRUE, which will result in a SNP list that includes the "MAF reference set". This is a set of 2297 SNPs that have the same minor allele across all 1000 genomes super populations and a minor allele frequency of 0.1-0.3.

``` r
snplist<-make_snplist(efo="glioma",trait="glioma",ref1000G_superpops=TRUE)
head(snplist)
#> [1] "rs2736100" "rs2853676" "rs891835"  "rs4295627" "rs4977756" "rs498872"
length(snplist)
#> [1] 2351
```

We can also define a set of "exposure SNPs" and include this in the list of rsids. For example, let's assume we are conducting a Mendelian randomisation study to assess the effect of polyunsaturated fatty acid exposure on risk of glioma. Let's define the "exposure SNPs", or genetic instrument, as SNPs associated with polyunsaturated fatty acids with a P value \<5e-8. Let's use the ieugwasr package to extract "exposure SNPs" for polyunsaturated fatty acids from the Open GWAS project and also add these to the SNP list.

``` r
instruments<-ieugwasr::tophits(id="met-d-PUFA",pval = 5e-08)
#> API: public: http://gwas-api.mrcieu.ac.uk/
#> Using access token. For info on how this is used see logging_info()
#> ℹ 2021-04-08 22:34:02 > Setting client.id from options(googleAuthR.client_id)
#> Using an auto-discovered, cached token.
#> To suppress this message, modify your code or options to clearly consent to the use of a cached token.
#> See gargle's "Non-interactive auth" vignette for more details:
#> https://gargle.r-lib.org/articles/non-interactive-auth.html
#> The googleAuthR package is using a cached token for<EMAIL_ADDRESS>snplist<-make_snplist(efo = "glioma",trait="glioma",ref1000G_superpops=TRUE,snplist_user=instruments$rsid)
head(snplist)
#> [1] "rs2736100" "rs2853676" "rs891835"  "rs4295627" "rs4977756" "rs498872"
length(snplist)
#> [1] 2417
```

Our SNP list now contains the rsids for: 1) the GWAS catalog top hits, 2) the MAF reference set and 3) the "exposure SNPs". Next, we extract the summary associations statistics for these SNPs from the glioma outcome dataset. Note: the extract\_snps function only works on MAC or linux operating systems.

``` r
File<-system.file("extdata", "glioma_test_dat.txt", package = "mrQC")
gli<-extract_snps(snplist=snplist,path_to_target_file=File,path_to_target_file_sep="\t")
dim(gli)
#> [1] 115  16
head(gli)
#>       Locus Allele1 Allele2         MAF                Geno_Counts  Subjects
#> 1 rs6010620       G       A 0.229|0.175  2969|1698|287/1254|554|48 4954|1856
#> 2 rs2736100       G       T 0.506|0.437 1211|2465|1273/594|900|360 4949|1854
#> 3 rs2157719       A       G 0.450|0.511 1508|2428|1013/454|905|495 4949|1854
#> 4 rs4977756       A       G 0.423|0.481  1669|2374|908/509|905|438 4951|1852
#> 5 rs2853676       G       A 0.250|0.311  2795|1835|321/874|808|172 4951|1854
#> 6 rs2297440       C       T 0.223|0.175  2668|1458|244/1039|455|42 4370|1536
#>          p     OR OR_95._CI_l OR_95._CI_u CHROMOSOME LOCATION controls cases
#> 1 1.13e-10 0.7009      0.6289      0.7812         20 61780283     4954  1856
#> 2 4.28e-09 0.7722      0.7082      0.8420          5  1339516     4949  1854
#> 3 4.57e-09 1.2936      1.1866      1.4102          9 22023366     4949  1854
#> 4 1.21e-08 1.2831      1.1774      1.3982          9 22058652     4951  1852
#> 5 2.16e-08 1.3112      1.1922      1.4420          5  1341547     4951  1854
#> 6 1.49e-07 0.7278      0.6462      0.8198         20 61782743     4370  1536
#>   eaf.controls path_to_target_file
#> 1        0.229 glioma_test_dat.txt
#> 2        0.506 glioma_test_dat.txt
#> 3        0.450 glioma_test_dat.txt
#> 4        0.423 glioma_test_dat.txt
#> 5        0.250 glioma_test_dat.txt
#> 6        0.223 glioma_test_dat.txt
```

In the above example, we extracted the summary data for the SNPs of interest from a tab separated text file that was stored on our local machine. In practice, this might correspond to a file of GWAS results obtained via correspondence with a GWAS consortium. Alternatively, we could have sourced the outcome summary data from the Open GWAS project (<https://gwas.mrcieu.ac.uk/>). For example, to extract summary data for thyroid cancer from Open GWAS we could run:

``` r
snplist<-make_snplist(efo = "thyroid carcinoma",trait="thyroid carcinoma",ref1000G_superpops=TRUE,snplist_user=instruments$rsid)
#> Warning in gwas_catalog_hits(trait = trait): search for trait - thyroid
#> carcinoma - returned 0 studies from the GWAS catalog
head(snplist)
#> [1] "rs965513"    "rs944289"    "rs966423"    "rs2439302"   "rs116909374"
#> [6] "rs6759952"
length(snplist)
#> [1] 2397
thy <- ieugwasr::associations(id="ieu-a-1082", variants=snplist,proxies=0)
#> Using access token. For info on how this is used see logging_info()
#> Using an auto-discovered, cached token.
#> To suppress this message, modify your code or options to clearly consent to the use of a cached token.
#> See gargle's "Non-interactive auth" vignette for more details:
#> https://gargle.r-lib.org/articles/non-interactive-auth.html
#> The googleAuthR package is using a cached token for<EMAIL_ADDRESS>dim(thy)
#> [1] 146  12
thy
#> # A tibble: 146 x 12
#>        se position     n chr       p    beta id    rsid  ea    nea     eaf trait
#>     <dbl>    <int> <int> <chr> <dbl>   <dbl> <chr> <chr> <chr> <chr> <dbl> <chr>
#>  1 0.104    2.75e7  1187 12    0.826 -0.0229 ieu-… rs11… A     G     0.200 Thyr…
#>  2 0.0934   6.34e7  1187 3     0.526 -0.0592 ieu-… rs68… C     T     0.271 Thyr…
#>  3 0.0891   8.64e7  1187 4     0.754  0.0279 ieu-… rs46… G     A     0.676 Thyr…
#>  4 0.0953   9.20e6  1187 5     0.764 -0.0286 ieu-… rs14… G     A     0.256 Thyr…
#>  5 0.106    1.25e8  1187 10    0.354 -0.0977 ieu-… rs10… T     C     0.190 Thyr…
#>  6 0.104    1.26e7  1187 11    0.461  0.0768 ieu-… rs10… G     A     0.803 Thyr…
#>  7 0.119    8.15e6  1187 4     0.811  0.0286 ieu-… rs12… C     T     0.144 Thyr…
#>  8 0.0886   1.65e8  1187 4     0.717 -0.0321 ieu-… rs10… C     T     0.327 Thyr…
#>  9 0.104    1.24e8  1187 5     0.187  0.137  ieu-… rs48… T     C     0.802 Thyr…
#> 10 0.133    4.07e7  1187 8     0.400  0.112  ieu-… rs78… G     A     0.112 Thyr…
#> # … with 136 more rows
```

The make\_snplist() function returns a warning that no GWAS hits were found when searching for thyroid carcinoma in the GWAS catalog's "reported trait" field. The search was, however, able to identify hits when searching for "thyroid carcinoma" in the efo field.

Returning to the glioma example, having extracted the summary data for the three sets of SNPs, we now need to format the summary data. This is to get the data into the expected format for the QC functions.

``` r
Dat<-format_data(dat=gli,outcome="Glioma",population="European",pmid=22886559,study="GliomaScan",ncase="cases",ncontrol="controls",rsid="Locus",effect_allele="Allele1",other_allele="Allele2",or="OR",or_lci="OR_95._CI_l",or_uci="OR_95._CI_u",eaf="eaf.controls",p="p",efo="glioma")
head(Dat)
#>        rsid effect_allele other_allele         MAF                Geno_Counts
#> 1 rs6010620             G            A 0.229|0.175  2969|1698|287/1254|554|48
#> 2 rs2736100             G            T 0.506|0.437 1211|2465|1273/594|900|360
#> 3 rs2157719             A            G 0.450|0.511 1508|2428|1013/454|905|495
#> 4 rs4977756             A            G 0.423|0.481  1669|2374|908/509|905|438
#> 5 rs2853676             G            A 0.250|0.311  2795|1835|321/874|808|172
#> 6 rs2297440             C            T 0.223|0.175  2668|1458|244/1039|455|42
#>    Subjects        p     OR OR_95._CI_l OR_95._CI_u CHROMOSOME LOCATION
#> 1 4954|1856 1.13e-10 0.7009      0.6289      0.7812         20 61780283
#> 2 4949|1854 4.28e-09 0.7722      0.7082      0.8420          5  1339516
#> 3 4949|1854 4.57e-09 1.2936      1.1866      1.4102          9 22023366
#> 4 4951|1852 1.21e-08 1.2831      1.1774      1.3982          9 22058652
#> 5 4951|1854 2.16e-08 1.3112      1.1922      1.4420          5  1341547
#> 6 4370|1536 1.49e-07 0.7278      0.6462      0.8198         20 61782743
#>   ncontrol ncase   eaf path_to_target_file       lnor    lnor_se     pmid
#> 1     4954  1856 0.229 glioma_test_dat.txt -0.3553901 0.05532116 22886559
#> 2     4949  1854 0.506 glioma_test_dat.txt -0.2585117 0.04414629 22886559
#> 3     4949  1854 0.450 glioma_test_dat.txt  0.2574290 0.04404068 22886559
#> 4     4951  1852 0.423 glioma_test_dat.txt  0.2492790 0.04384619 22886559
#> 5     4951  1854 0.250 glioma_test_dat.txt  0.2709427 0.04852824 22886559
#> 6     4370  1536 0.223 glioma_test_dat.txt -0.3177290 0.06070188 22886559
#>   outcome population      study UKbiobank effect_allele_confirmed open_gwas
#> 1  Glioma   European GliomaScan        NA                   FALSE     FALSE
#> 2  Glioma   European GliomaScan        NA                   FALSE     FALSE
#> 3  Glioma   European GliomaScan        NA                   FALSE     FALSE
#> 4  Glioma   European GliomaScan        NA                   FALSE     FALSE
#> 5  Glioma   European GliomaScan        NA                   FALSE     FALSE
#> 6  Glioma   European GliomaScan        NA                   FALSE     FALSE
#>      efo efo_id
#> 1 glioma     NA
#> 2 glioma     NA
#> 3 glioma     NA
#> 4 glioma     NA
#> 5 glioma     NA
#> 6 glioma     NA
```

In this example, the glioma results file contained columns for the odds ratio and 95% confidence intervals. In practice, the format of the effect size columns is highly variable across studies. For example, some studies may report the log odds ratio and its standard error or the odds ratio and P value without confidence intervals or a standard error. The format\_data() function accepts these and other effect size reporting formats. See ?format\_data() for more info.

Now we are ready to perform some quality checks on the summary data

## Step 2. Check that the effect allele frequency column is correct

Next we create some plots to visualise potential problems with the effect allele frequency column. We do this by comparing allele frequency in the outcome glioma dataset to the 1000 genomes super populations. Let's restrict the comparison to the European super population, since we know that the glioma dataset was derived from a European ancestry population.

``` r
Plot1<-make_plot_maf(ref_1000G="EUR",target_dat=Dat)
Plot1
```

SNPs with a red colour are SNPs with incompatible minor allele frequencies, i.e. the allele frequencies are above 0.5 in the outcome dataset but less than 0.5 in the 1000 genomes dataset. In this example, all SNPs are flagged as problematic and there is a strong inverse correlation in minor allele frequency between the datasets. This indicates that the reported effect allele frequency column, in the outcome dataset, does not correspond to the reported effect allele. The strong inverse correlation implies that the effect allele column actually refers to the non-effect allele.

Next we construct a similar plot but this time comparing allele frequency with all 1000 genomes super populations.

``` r
Plot2<-make_plot_maf(ref_1000G=c("AFR","AMR","EAS","EUR","SAS","ALL"),target_dat=Dat)
Plot2
```

All SNPs across all super populations are flagged as problematic. This illustrates that the function can identify problematic SNPs regardless of the ancestry of the outcome dataset. We also see a strong inverse correlation in the comparison with the European 1000 genomes super population. This is not surprising, since we know that the the glioma GWAS results were generated in a European ancestry population. This illustrates that the strength of the correlation in MAF between the datasets can also support inferences about the ancestral background of the outcome dataset, although the function was not designed with this objective in mind. A more efficient approach would be to select SNPs with a much wider range of variation in minor allele frequency that chosen here.

## Step 3. Check that the effect allele column is correct

We next compare effect alleles between the glioma outcome dataset and the GWAS catalog, in order identify potential mis-specification of the effect allele column.

``` r
Dat1<-Dat
Plot3<-make_plot_gwas_catalog(dat=Dat,efo=unique(Dat$efo),trait="glioma")
Plot3
```

We see that there are three groups of SNPs: those flagged as showing no effect size conflict; those with moderate effect size conflict; and those with high effect size conflict. The distinction between moderate and high effect size conflict is arbitrary but is specified to make allowance for chance deviations between datasets. If the effect sizes are in opposite directions, the effect size conflict flag is set to moderate or high. If the two-sided P value in both datasets is ≤0.0001 then the flag is upgraded to high. In addition, if the summary association statistics in the outcome dataset and GWAS catalog are derived from the same publication, and effect sizes are in opposite directions, the conflict flag is set to high regardless of P value.

Overall, the plot shows that the majority of SNPs show effect size conflicts. In other words, alleles associated with higher risk of glioma in the GWAS catalog tend to be associated with lower risk of glioma in the outcome dataset. This indicates that the reported effect allele column corresponds to the non-effect allele. Taken together with the previous allele frequency plots, showing an inverse correlation in MAF between the outcome dataset and 1000 genomes European superpopulation, this strongly indicates that the effect and non-effect allele columns have been incorrectly specified. The reported non-effect allele column is very likely the effect allele column.

We can also make a plot comparing effect allele frequency between the outcome dataset and the GWAS catalog, which we show in the next example.

``` r
Plot4<-make_plot_gwas_catalog(dat=Dat,plot_type="plot_eaf",efo=unique(Dat$efo),trait=unique(Dat$outcome))
Plot4
```

We see an inverse correlation in effect allele frequency (EAF) between the outcome glioma dataset and the GWAS catalog in European ancestry studies. This is the expected pattern when the effect allele column has been incorrectly specified. In general, the EAF conflict flag is set to moderate or high if EAF is not consistent between the outcome dataset and the GWAS catalog (e.g. is \<0.5 in the outcome dataset but is \>0.5 in the GWAS catalog or vice versa). For conflicting SNPs, the flag is further upgraded to high if effect allele frequency is \>0.6 or \<0.4. This makes allowance for chance deviations in allele frequency for SNPs with minor allele frequency close to 0.5.

## Step 4. Check for errors in the reported effect sizes

We next compare the predicted and reported effect sizes. Since the reported effect sizes correspond to log odds ratios, we predict the log odds ratio using the predict\_lnor\_sh() function. This function was developed by Sean Harrison. <https://seanharrisonblog.com/2020/04/11/estimating-an-odds-ratio-from-a-gwas-only-reporting-the-p-value/>

``` r
Pred<-predict_lnor_sh(dat=Dat)
Plot5<-make_plot_pred_effect(dat=Pred)
Plot5
```

 The plot shows a strong positive correlation between the predicted and reported effect sizes, an intercept close to zero and a slope that is \>0.8. This is close to expectation, since the intercept should be 0 and the slope should be 1 when the predicted and reported effect sizes are identical.

We can also plot the relative bias, i.e. the percentage deviation of the predicted effect size from the reported effect size.

``` r
Plot6<-make_plot_pred_effect(dat=Pred,bias=TRUE)
Plot6
```

 Overall the relative bias seems small and mostly varies from -10.9% to -13.5%. Since genetic effect sizes tend to be small, a relative bias of 10% is very small in absolute terms.

In the next example we show a dataset that returns a slope and intercept very different from expectation.

``` r
snplist<-make_snplist(efo = "kidney cancer",snplist_user=instruments$rsid)
ukb <- data.frame(ieugwasr::associations(id="ukb-b-1316", variants=snplist,proxies=0),stringsAsFactors=FALSE)
#> Using access token. For info on how this is used see logging_info()
#> Using an auto-discovered, cached token.
#> To suppress this message, modify your code or options to clearly consent to the use of a cached token.
#> See gargle's "Non-interactive auth" vignette for more details:
#> https://gargle.r-lib.org/articles/non-interactive-auth.html
#> The googleAuthR package is using a cached token for<EMAIL_ADDRESS>Ukb<-format_data(dat=ukb,outcome="Kidney cancer",population="European",pmid="ukb-b-1316",ncase=1114,ncontrol=461896,study="UKB",UKbiobank=TRUE,rsid="rsid",effect_allele="ea",other_allele="nea",lnor="beta",se="se",eaf="eaf",p="p",effect_allele_confirmed=TRUE,all_summary_stats=TRUE,ID=145,efo = "kidney cancer")
Pred<-predict_lnor_sh(dat=Ukb)
Plot5_2<-make_plot_pred_effect(dat=Pred)
Plot5_2
```

Although there is a strong positive correlation, the slope is 400, when we expect a slope of 1. In fact, further investigation revealed that Open GWAS dataset ukb-b-1316 was generated using a linear model. In other words, the results were derived from a model where kidney cancer case-control status (controls coded 1 and cases coded 2) was regressed on SNP genotype (additively coded). The effect size from this model can be interpreted as the change in absolute risk of kidney cancer per copy of the effect allele. We can transform this into a log odds ratio scale using the transform\_betas() function.

``` r
Ukb2<-transform_betas(dat=Ukb,effect="lnor",effect.se="se")
Pred2<-predict_lnor_sh(dat=Ukb2)
Plot5_3<-make_plot_pred_effect(dat=Pred2)
Plot5_3
```

After transforming the reported effect size to a log odds ratio scale, we now see a slope close to 1 for the relationship with the predicted effect size. More generally, we suggest that any dataset with an unusual intercept or with a slope very different from 1 (e.g. \<0.8 or \>1.2) should be investigated by the user for potential problems. In the glioma example, the slope and intercept look reasonably close to what we'd expect. In the kidney cancer example, the slope was very different from 1 because the reported effect sizes had not been generated in a logistic regression model. Other factors that could cause the slope to differ from 1 include: 1) the impact of covariate adjustment in the original GWAS, 2) deviations from Hardy Weinberg equilibrium, 3) mismatches between reported and actual allele frequency for some or all SNPs or 4) mismatches between the reported and effective SNP-level sample sizes for some or all SNPs.

## Step 5. Check whether the reported P values correspond to the reported effect sizes

Let's now return back to the glioma dataset. Let's generate some ZZ plots, in order to flag SNPs with P values that don't coincide with their reported effect sizes. The zz\_plot() function compares Zp scores (inferred from the reported P values) to Zb scores (inferred from the reported effect size and standard error).

``` r
Plot7<-zz_plot(dat=Dat)
Plot7
```

The correlation between the Zp and Zb scores is 1, indicating very strong concordance between the reported P values and reported effect sizes in the glioma dataset for the selected SNPs. In the next example, we highlight a dataset where there is discordance between the reported P values and effect sizes.

``` r
instruments<-ieugwasr::tophits(id="met-d-PUFA")
#> Using access token. For info on how this is used see logging_info()
#> Using an auto-discovered, cached token.
#> To suppress this message, modify your code or options to clearly consent to the use of a cached token.
#> See gargle's "Non-interactive auth" vignette for more details:
#> https://gargle.r-lib.org/articles/non-interactive-auth.html
#> The googleAuthR package is using a cached token for<EMAIL_ADDRESS>snplist<-make_snplist(efo = "lung carcinoma",snplist_user=instruments$rsid)
luc <- ieugwasr::associations(id="ieu-a-966", variants=snplist,proxies=0)
#> Using access token. For info on how this is used see logging_info()
#> Using an auto-discovered, cached token.
#> To suppress this message, modify your code or options to clearly consent to the use of a cached token.
#> See gargle's "Non-interactive auth" vignette for more details:
#> https://gargle.r-lib.org/articles/non-interactive-auth.html
#> The googleAuthR package is using a cached token for<EMAIL_ADDRESS>Dat<-format_data(dat=data.frame(luc,stringsAsFactors=F),outcome="Lung cancer",population="European",pmid=24880342,ncase=11348,ncontrol=15861,study="ILCCO",UKbiobank=FALSE,rsid="rsid",effect_allele="ea",other_allele="nea",lnor="beta",se="se",eaf="eaf",p="p",efo = "lung carcinoma")
Plot7_2<-zz_plot(dat=Dat)
Plot7_2
```

The correlation between the Zp and Zb scores is less than 1, suggesting discordance between the reported P values and reported effect sizes, which is also clear from visual inspection of the plot. For example, 1 SNP has a Z score close to 15 when estimated from the reported P value but a Z score close to 5 when estimated from the reported effect size and standard error. In this example, we have included three sets of SNPs (the GWAS catalog hits, the MAF reference set and the "exposure SNPs". Let's restrict the comparison to only the "exposure SNPs".

``` r
luc <- ieugwasr::associations(id="ieu-a-966",variants=instruments$rsid,proxies=0)
#> Using access token. For info on how this is used see logging_info()
#> Using an auto-discovered, cached token.
#> To suppress this message, modify your code or options to clearly consent to the use of a cached token.
#> See gargle's "Non-interactive auth" vignette for more details:
#> https://gargle.r-lib.org/articles/non-interactive-auth.html
#> The googleAuthR package is using a cached token for<EMAIL_ADDRESS>Dat<-format_data(dat=data.frame(luc,stringsAsFactors=F),outcome="Lung cancer",population="European",pmid=24880342,ncase=11348,ncontrol=15861,study="ILCCO",rsid="rsid",effect_allele="ea",other_allele="nea",lnor="beta",se="se",eaf="eaf",p="p",efo = "lung carcinoma")
Plot7_3<-zz_plot(dat=Dat)
Plot7_3
```

The correlation between the Zp and Zb scores is slightly less than 1, suggesting a small degree of discordance between the reported P values and reported effect sizes, which is also clear from visual inspection of the plot. For example one SNP has a Zp score \>2 and Zb score \<2. Some SNPs have P values a bit more extreme than you'd expect given the magnitude of the reported effect sizes and standard errors.

## Step 6. Combine all plots into a single report

Next we combine all the plots into a single report.

``` r
Plot_list2<-ls()[grep("Plot[0-9]",ls())]
Plot_list<-lapply(1:length(Plot_list2),FUN=function(x) eval(parse(text=Plot_list2[x])))
combine_plots(Plot_list=Plot_list,out_file="~/qc_report.png")
```

!["qc\_report.png"](/man/figures/README-qc_report.png)

# Example 2. Check the summary and meta data from a GWAS of arachidonic acid

In this example we show how the package can be used to check the quality of reported summary and meta data from genome-wide association studies of non-disease continuous traits, using arachidonic acid in the CHARGE consortium as an example.

## Step 1. Extract and format the outcome summary data (arachidonic acid example)

The first step is to make a list of SNP rsids. Let's assume our study of interest is a GWAS of arachidonic acid performed by the CHARGE consortium. We need to extract: 1) **the GWAS catalog top hits** for arachidonic acid; 2) the **GWAS top hits** for arachidonic acid from the CHARGE consortium (available in data(charge\_top\_hits); and 3) **the MAF 1KG reference set**.

``` r
library(mrQC)
data(charge_top_hits)
snplist<-make_snplist(trait="Plasma omega-6 polyunsaturated fatty acid levels (arachidonic acid)",ref1000G_superpops=TRUE,snplist_user=charge_top_hits)
File<-system.file("extdata", "ara_test_dat.txt", package = "mrQC")
puf<-extract_snps(snplist=snplist,path_to_target_file=File,path_to_target_file_sep="\t")
Dat<-format_data(dat=puf,outcome="arachidonic acid",population="European",pmid=24823311,study="CHARGE",ncontrol="n",UKbiobank=FALSE,rsid="snp",effect_allele="effect_allele",other_allele="other_allele",beta="beta",se="se",eaf="effect_allele_freq",p="p")
```

## Step 2. Check that the effect allele frequency column is correct (arachidonic acid example)

Having prepared and formatted the data, we are now ready to perform some checks. First we check that the reported effect allele frequency column corresponds to the reported effect allele.

``` r
Plot1<-make_plot_maf(ref_1000G="EUR",target_dat=Dat)
Plot1
```

``` r
Plot2<-make_plot_maf(ref_1000G=c("AFR","AMR","EAS","EUR","SAS","ALL"),target_dat=Dat)
Plot2
```

 There is a strong positive correlation between allele frequency between the arachidonic acid GWAS and the MAF 1K reference set, indicating that the reported effect allele frequency column corresponds to the reported effect allele.

## Step 3. Check that the effect allele column is correct (arachidonic acid example)

Now we check that the reported effect allele corresponds to the actual effect allele.

``` r
Plot3<-make_plot_gwas_catalog(dat=Dat,trait="Plasma omega-6 polyunsaturated fatty acid levels (arachidonic acid)",beta="beta",se="se")
Plot3
```

All SNPs appear to have concordant effect sizes between the CHARGE consortium and the GWAS catalog. Although there are a few SNPs with effect sizes in opposite directions, the Z scores for these SNPs were small and therefore compatible with chance deviations. This suggests that the reported effect allele corresponds to the actual effect allele.

## Step 4. Check for errors in the reported effect sizes (arachidonic acid example)

Next, we plot the predicted versus reported effect sizes. Let's restrict the comparison to the top hits for arachidonic acid in the GWAS catalog and in the CHARGE study.

### CHARGE top hits for arachidonic acid

``` r
snplist<-make_snplist(ref1000G_superpops=FALSE,snplist_user=charge_top_hits)
puf<-extract_snps(snplist=snplist,path_to_target_file=File,path_to_target_file_sep="\t")
Dat<-format_data(dat=puf,outcome="arachidonic acid",population="European",pmid=24823311,study="CHARGE",ncontrol="n",UKbiobank=FALSE,rsid="snp",effect_allele="effect_allele",other_allele="other_allele",beta="beta",se="se",eaf="effect_allele_freq",p="p")
Dat<-predict_beta_sd(dat=Dat)
Plot4<-make_plot_pred_effect(dat=Dat,pred_beta = "beta_sd",pred_beta_se="se_sd",beta="beta",se="se",bias=FALSE)
Plot4
```

### GWAS catalog top hits for arachidonic acid

``` r
snplist<-make_snplist(trait="Plasma omega-6 polyunsaturated fatty acid levels (arachidonic acid)",ref1000G_superpops=FALSE)
puf<-extract_snps(snplist=snplist,path_to_target_file=File,path_to_target_file_sep="\t")
Dat2<-format_data(dat=puf,outcome="arachidonic acid",population="European",pmid=24823311,study="CHARGE",ncontrol="n",UKbiobank=FALSE,rsid="snp",effect_allele="effect_allele",other_allele="other_allele",beta="beta",se="se",eaf="effect_allele_freq",p="p")
Dat2<-predict_beta_sd(dat=Dat2)
Plot4_2<-make_plot_pred_effect(dat=Dat2,pred_beta = "beta_sd",pred_beta_se="se_sd",beta="beta",se="se",bias=FALSE)
Plot4_2
```

Notice that the relationship between the predicted and reported effect sizes are much closer to what we'd expect using the GWAS catalog top hits, with a slope close to 1 and an intercept close to 0. By contrast, using the CHARGE top hits, we get a slope of 0.69 and a larger intercept. The shape of the relationship between the predicted and reported effect sizes for the CHARGE top hits also appears to be much more non-linear, especially for stronger effect sizes. Notice that the SNPs with the strongest deviations from the expected effect size have lower minor allele frequencies. SNPs with higher minor allele frequencies tend to have more concordant effect sizes. This is a bit clearer if we plot the bias.

``` r
Plot5<-make_plot_pred_effect(dat=Dat,pred_beta = "beta_sd",pred_beta_se="se_sd",beta="beta",se="se",bias=TRUE)
```

The SNPs with the largest bias (defined as % deviation of predicted from the reported effect size) tend to have lower minor allele frequencies. This patterm of results suggests a large number of potential false positives, which in turn may reflect problems with the post-GWAS cleaning of the summary statistics. As a further check we can check how many of the GWAS top hits in the CHARGE consortium are also present in the GWAS catalog.

``` r
snplist<-make_snplist(trait="Plasma omega-6 polyunsaturated fatty acid levels (arachidonic acid)",snplist_user=charge_top_hits)
puf<-extract_snps(snplist=snplist,path_to_target_file=File,path_to_target_file_sep="\t")
Dat<-format_data(dat=puf,outcome="arachidonic acid",population="European",pmid=24823311,study="CHARGE",ncontrol="n",UKbiobank=FALSE,rsid="snp",effect_allele="effect_allele",other_allele="other_allele",beta="beta",se="se",eaf="effect_allele_freq",p="p")
Plot3_2<-make_plot_gwas_catalog(dat=Dat,trait="Plasma omega-6 polyunsaturated fatty acid levels (arachidonic acid)",force_all_trait_study_hits=TRUE,beta="beta",se="se")
Plot3_2
```

Notice that we set force\_all\_trait\_study\_hits to be TRUE. This forces all the GWAS hits for arachidonic acid in the CHARGE study to be included, even if they are missing from the GWAS catalog. We set the GWAS catalog Z scores for such SNPs to 0 and flag them as SNPs of concern (SNPs highlighted in red). In this example, most of the GWAS hits for arachidonic acid in the CHARGE consortium are not present in the GWAS catalog. Taken together with the non-linear relationship between the predicted and reported effect sizes above for the CHARGE GWAS hits, this suggests a large number of potential false positives, which in turn may reflect problems with the post-GWAS cleaning of the summary statistics. Indeed, correspondence with the CHARGE consortium confirmed that the GWAS summary statistics had not been filtered on low minor allele frequency or low imputation r2 scores. Using a cleaned dataset (filtered on MAF and imputation r2) returns plots with the expected pattern of results.

``` r
File<-system.file("extdata", "ara_test_cleaned_dat.txt", package = "mrQC")
data(charge_top_hits_cleaned)
snplist<-make_snplist(trait="Plasma omega-6 polyunsaturated fatty acid levels (arachidonic acid)",ref1000G_superpops=TRUE,snplist_user=charge_top_hits_cleaned)
puf<-extract_snps(snplist=snplist,path_to_target_file=File,path_to_target_file_sep="\t")
Dat<-format_data(dat=puf,outcome="arachidonic acid",population="European",pmid=24823311,study="CHARGE",ncontrol="n",rsid="snp",effect_allele="effect_allele",other_allele="other_allele",beta="beta",se="se",eaf="effect_allele_freq",p="p")
Plot3_3<-make_plot_gwas_catalog(dat=Dat,trait="Plasma omega-6 polyunsaturated fatty acid levels (arachidonic acid)",force_all_trait_study_hits=TRUE,beta="beta",se="se")
Plot3_3
```

``` r
# snplist<-make_snplist(ref1000G_superpops=FALSE,snplist_user=charge_top_hits)
# puf<-extract_snps(snplist=snplist,path_to_target_file=File,path_to_target_file_sep="\t")
# Dat<-format_data(dat=puf,outcome="arachidonic acid",population="European",pmid=24823311,study="CHARGE",ncontrol="n",UKbiobank=FALSE,rsid="snp",effect_allele="effect_allele",other_allele="other_allele",beta="beta",se="se",eaf="effect_allele_freq",p="p")
Dat<-predict_beta_sd(dat=Dat)
Plot5_2<-make_plot_pred_effect(dat=Dat,pred_beta = "beta_sd",pred_beta_se="se_sd",beta="beta",se="se")
Plot5_2
```

We now see that the vast majority of SNPs have concordant effect sizes between the GWAS catalog and the CHARGE consortium with respect to arachidonic acid. Also, the relationship between the predicted and reported effect sizes looks linear with a slope reasonably close to 1.

## Step 5. Check whether the reported P values correspond to the reported effect sizes (arachidonic acid example)

Finally, we check whether the reported P values correspond to the reported effect sizes.

``` r
Plot6<-zz_plot(dat=Dat,beta="beta",se="se")
Plot6
```

 We see a very close concordance between the reported P-values and reported effect sizes.

## Step 6. Combine all plots into a single report (arachidonic acid example)

Let's combine all the key figures into a single report

``` r
Plot_list2<-c("Plot1","Plot2","Plot3_2", "Plot4","Plot5","Plot6")
Plot_list<-lapply(1:length(Plot_list2),FUN=function(x) eval(parse(text=Plot_list2[x])))
combine_plots(Plot_list=Plot_list,out_file="~/qc_report2.png")
```

!["qc\_report.png"](/man/figures/README-qc_report2.png)

Plot\_list\<-lapply(1:length(Plot\_list2),FUN=function(x) eval(parse(text=Plot\_list2[x]))) combine\_plots(Plot\_list=Plot\_list,out\_file="~/mrQC/man/figures/README-qc\_report2.png")

data("charge\_top\_hits\_cleaned") snplist\<-make\_snplist(trait="Plasma omega-6 polyunsaturated fatty acid levels (arachidonic acid)",ref1000G\_superpops=TRUE,snplist\_user=charge\_top\_hits\_cleaned) File\<-system.file("extdata", "ara\_test\_cleaned\_dat.txt", package = "mrQC") puf\<-extract\_snps(snplist=snplist,path\_to\_target\_file=File,path\_to\_target\_file\_sep="") Dat\<-format\_data(dat=puf,outcome="arachidonic acid",population="European",pmid=24823311,study="CHARGE",ncontrol="n",UKbiobank=FALSE,rsid="snp",effect\_allele="effect\_allele",other\_allele="other\_allele",beta="beta",se="se",eaf="effect\_allele\_freq",p="p",efo\_id="EFO\_0005680") Plot3\_2\<-make\_plot\_gwas\_catalog(dat=Dat,trait="Plasma omega-6 polyunsaturated fatty acid levels (arachidonic acid)",force\_all\_trait\_study\_hits=TRUE,beta="beta",se="se") Plot3\_2 \`\`\`

# Summary

Overall, we can conclude that the reported effect allele column in the arachidonic GWAS summary dataset correctly corresponds to the actual / modelled effect allele. The reported effect allele frequency column also corresponds to the reported effect allele. However, we found an unusually large number of GWAS hits that were absent from the GWAS catalog and that had unusually strong effect sizes. These SNPs turned out to be false positives owing to a lack of post-GWAS filtering of low quality SNPs.

repository: winfrednyoroka/CheckSumStats

branch_name: refs/heads/main

repo_name: winfrednyoroka/CheckSumStats


text:
---
layout: post
title: Using Inheritance To Build DSC Resources
tags: [Classes, DSC]
---

Today we are going to use inheritance to create multiple DSC resource from a base class.
I'm the first to admit I'm lazy and always trying to get the most bang for my lines of code.
Inheritance is a great way to reduce code duplication and pretty easy once you wrap your head around it.

**The Good Stuff:**
My DSC [FileWatcher module](https://github.com/dchristian3188/FileWatcher) and an example of building resources with inheritance.

<!-- more -->

I'm going to be building off my SmartServiceRestart resource from [this](http://overpoweredshell.com/DSC-Classes-Using-Helper-Methods/) post.
This resource watches a file path and a service.
It then compares the start time of the service against the last write time of the file.
If the file has an older last write time, the service gets restarted.
This resource is a great tool for services that are not smart enough to automatically reload their configurations.

I liked the idea of having a file watcher that would reload the service and wanted to see if I could apply this logic anywhere else.
I thought it would be cool to have similar resources to manage processes and websites.
Before we look at the new resources, let's examine the original class.

The original resource contained the following methods:

- **GetLastWriteTime** - A helper method to get the last write time of the file(s)
- **GetProcessStartTime** - A helper method to get the start time of the service (PID)
- **Get** - Runs both helper methods and returns an instance of the class
- **Test** - Compares the two dates returned by the helper methods and determines who's older
- **Set** - Restarts the service

Looking over the list of methods, the only ones that are specific to a service are the ```GetProcessStartTime``` and the ```Set```.
What that means is we can move the remaining of the methods to a base class.
This new base class will have the ```Get```, ```Test``` and ```GetLastWriteTime``` methods.

Now that I have my base defined, I know I only need to create the ```Set``` and ```GetProcessStartTime``` methods for each resource.
The idea is final classes will come together like this:

![_config.yml]({{ site.baseurl }}/images/DSCInheritance/FileWatcherInheritance.png)

We also have to perform this same inventory for the properties / parameters.

All resources will share the below properties:

- **Path** - Path to the folder or files to check
- **Filter** - A filter to apply to the files
- **LastWriteTime** - Non-configurable date time property to store last write time of the file
- **ProcessStartTime** - Non-configurable date time property to store the process start time

The specific parameters for each resource will be considerably different.
Here's a breakdown of the final structure.

![_config.yml]({{ site.baseurl }}/images/DSCInheritance/FIleWatcherInheritanceProperties.png)

With my base class defined, it's time to create the new resources.
Since the comparison logic is in the base class, each new resource really only needs to be able to determine when ***Its*** type of process started and how to create or restart the instance.

We've already covered the ServiceFileWatcher methods in [this](http://overpoweredshell.com/DSC-Classes-Using-Helper-Methods/) post, so let's take a look at the new ProcessFileWatcher resource.
This resource is going to have a couple of new parameters.
We will add a parameter for process name, process path (path to the executable), and any startup parameters to pass to the process.
Here is the class and property declaration.
Notice that it inherits from the BaseFileWatcher Class.

```powershell
[DscResource()]
class ProcessFileWatcher : BaseFileWatcher
{

    [DscProperty(Key)]
    [string]
    $ProcessName

    [DscProperty(Mandatory = $true)]
    [string]
    $ProcessPath

    [DscProperty()]
    [string]
    $ProcessStartArgs
...
```

This is what the ProcessFileWatcher's ```GetProcessStart``` method ended up looking like.
If more than one process matches the name, the resource returns the start time of the oldest.

```powershell
[DateTime]GetProcessStartTime()
{
    Write-Verbose -Message "Checking for process Name: $($this.ProcessName)"
    $processInfo = (Get-CimInstance win32_process -Filter "name='$($this.ProcessName)'")

    if ($processInfo.ProcessId -eq $null)
    {
        Write-Verbose -Message "Could not find a running process, setting start time to min date value"
        $processStart = [datetime]::MinValue
    }
    else
    {
        $processStart = $processInfo |
            Sort-Object -Property CreationDate -Descending |
            Select-Object -ExpandProperty CreationDate -First 1

        Write-Verbose -Message "Process started at: $($processStart)"
    }
    Return $processStart
}
```

The ProcessFileWatcher ```Set``` is a little more tricky.
In this example, we're going to have to kill all process that match that name.
We also need to use the Process path to essentially restart the process.

```powershell
[Void]Set()
{
    $runningProcs = Get-CimInstance win32_process -Filter "name='$($this.ProcessName)'"

    if ($runningProcs)
    {
        Write-Verbose -Message "Stopping running Processes $($runningProcs.ProcessId -join ', ')"
        Stop-Process -ID $runningProcs.ProcessId -ErrorAction Stop -Force
    }

    Write-Verbose -Message "Starting Process [$($this.ProcessName)] at path [$($this.ProcessPath)] with args [$($this.ProcessStartArgs)]"

    $startProcessArgs = @{
        FilePath = $this.ProcessPath
        PassThru = $true
    }
    if (-not([string]::IsNullOrEmpty($this.ProcessStartArgs)))
    {
        $startProcessArgs['ArgumentList'] = $this.ProcessStartArgs
    }
    Start-Process @startProcessArgs
}
```

And that's it!
The whole resource is the couple of properties and the two methods.
Since these methods are still returning the same type, we are able to leverage the same comparison logic from the parent.

Should we make another one?
Next up is the WebSiteFileWatcher.
Here's the resource declaration and the new parameter of ```WebsiteName```.
Again this class inherits from BaseFileWatcher.

```powershell
[DscResource()]
class WebSiteFileWatcher : BaseFileWatcher
{

    [DscProperty(Key)]
    [string]
    $WebsiteName

...
```

Next we need to define the WebsiteFileWatcher's ```GetProcessStartTime``` helper method.
To get this, we first need to find what application pool is running the website.
Once we know this, we can check when the application pool started.

```powershell
[DateTime]GetProcessStartTime()
{

    Write-Verbose -Message "Checking for Application pool running [$($this.WebsiteName)]"
    $websiteInfo = Get-Website -Name $this.WebsiteName

    if(-not($websiteInfo))
    {
        throw "Unable to find website $($this.WebsiteName)"
    }

    Write-Verbose -Message "Checking for process running application pool [$($websiteInfo.applicationPool)]"

    $AppPoolName = @{
        Name       = 'AppPoolName'
        Expression = {(Invoke-CimMethod -InputObject $PSItem -MethodName 'GetOwner').User}
    }
    $processInfo = (Get-CimInstance win32_process -Filter "name='w3wp.exe'") |
        Select-Object *, $AppPoolName |
        Where-Object -FilterScript {$PSItem.AppPoolName -eq $($websiteInfo.applicationPool)}

    if (($processInfo.ProcessId -eq 0) -or $processInfo -eq $null)
    {
        Write-Verbose -Message "Could not find a running process, setting start time to min date value"
        $processStart = [datetime]::MinValue
    }
    Else
    {
        $processStart = $processInfo.CreationDate
        Write-Verbose -Message "Process started at: $($processStart)"
    }
    Return $processStart
}
```

Next up is the WebsiteFileWatchers ```Set``` method.
First step is to find the application pool running the site.
Once we have this, we can restart the app pool.
Now, this could cause trouble if you have websites or web applications sharing the same application pool.
Good thing nobody does that.

```powershell
[Void]Set()
{
    $webInfo = Get-Website -Name $this.WebsiteName
    Write-Verbose -Message "Restarting Application pool [$($webInfo.applicationPool)]"
    Restart-WebAppPool -Name $webInfo.applicationPool
}
```

And we're done with the Website watcher!
The only logic in the class is specific to websites and we get all the parent logic for free.
I hope this helped highlight some of the power of inheritance.
With a little bit of planning we were able to turn one resource into three!
The completed resource and all source is [here.](https://github.com/dchristian3188/FileWatcher)
Once I finish up the remaining Pester tests, I plan to submit the module to the gallery.

url: https://raw.githubusercontent.com/dchristian3188/dchristian3188.github.io/77d2aefcfa579f9aa1c5ef0df8d7980bb0190682/_posts/2017-06-23-Using Inheritance To Build DSC Resources.md


text:
---
title: "Parameter"
date: "2019-01-01"
path: "/docs/userguide/parameter/"
meta_title: "BioDynaMo User Guide"
meta_description: "This is the parameter page."
toc: true
image: ""
next:
    url:  "/docs/userguide/parameter/"
    title: "Parameter"
    description: "This is the parameter page."
sidebar: "userguide"
keywords:
  -parameter
  -configuration
  -code
  -setup
---

Parameters are used to tailor BioDynaMo to your specific simulation.

The majority of parameters are defined in the [core engine](/bioapi/structbdm_1_1Param.html).
Each module can define its own [`Param` class](/bioapi/structbdm_1_1experimental_1_1neuroscience_1_1Param.html) to add additional parameters.

There are three ways to set the value of a parameter:

1.  TOML configuration file
2.  Command line argument
3.  Assignment in the source code.

Note

Higher index takes precedence.  E.g. If you define the backup_file in the TOML file and the
    command line parameter, the command line version will be used.

The documentation of each parameter contains a description of the
parameter, its default value and how to set it in the TOML file ([example](/bioapi/structbdm_1_1Param.html#a13d24f045335b7ac62a091f56c6fe166))

The following code snippet shows how to access a parameter in your
simulation.

```
const auto* param = Simulation::GetActive()->GetParam();
std::cout << param->simulation_time_step_ << std::endl;
std::cout << param->GetModuleParam<neuroscience::Param>()->neurite_max_length_ << std::endl;

```

### Configuration File

This is the recommended way to set runtime variables. Create a file `bdm.toml`
in the working directory and add your configuration. You can find a sample below:

```
[visualization]
export = true
export_interval = 1

[[visualize_sim_object]]
name = "Cell"
additional_data_members = [ "density_" ]
```

### Command Line Options

Some parameter can be set as command line argument when you start the simulation.
For a complete list execute the binary with the `--help` switch. e.g. `./cell_division --help`.

Sample output:
```
 -- BioDynaMo command line options

Usage:
  ./cell_division [OPTION...]

 Simulation options:
  -n, --num-cells arg  The total number of cells (default: 10)

 Core options:
  -h, --help          Print this help message.
      --version       Print version number of BioDynaMo.
      --opencl        Enable GPU acceleration through OpenCL.
      --cuda          Enable GPU acceleration through CUDA.
  -v, --verbose       Verbose mode. Causes BioDynaMo to print debugging
                      messages. Multiple -v options increases the verbosity. The
                      maximum is 3.
  -r, --restore FILE  Restores the simulation from the checkpoint found in
                      FILE and continues simulation from that point. (default: )
  -b, --backup FILE   Periodically create full simulation backup to the
                      specified file. NOTA BENE: File will be overriden if it
                      exists. (default: )
  -c, --config FILE   The TOML configuration that should be used. (default: )

```

You can append your own command line options as following (e.g. `num-cells` as
show in the sample above):

```
auto opts = CommandLineOptions(argc, argv);
opts.AddOption<uint64_t>("n, num-cells", "The total number of cells", "10");

// You will need to pass this object to the main Simulation object as follows:
Simulation simulation(opts);

// To retrieve the values from the command line in your simulation code
auto parser = opts.Parse();
uint64_t num_cells = parser.Get<uint64_t>("num-cells");
```

The `AddOption` function can be broken down as follows: The template parameter
(`uint64_t`) specifies what value type your option should be. The first fucntion
parameter is for the flag abbreviation (in this case -n and --num-cells) The
second parameter is the name of your command line option The third parameter is
the description (as shown in the help dialogue) The fourth parameter is the
default value of the command line option (i.e. when you do not specify the
flag).

### Assignment in the Source Code

You can also set a runtime parameter in the source code. You have to recompile
your simulation though.

```
auto set_param = [](Param* param) {
  // Create an artificial bound for the simulation space
  param->bound_space_ = true;
  param->min_bound_ = 0;
  param->max_bound_ = 250;
  param->run_mechanical_interactions_ = false;
};

Simulation simulation(argc, argv, set_param);
```

You have to create a lambda and pass it to the constructor of `Simulation`.
Afterwards, `Simulation` only returns a const pointer to the parameters. Thus,
they cannot be modified.

repository: thecapitalistcycle/biodynamo

repo_name: thecapitalistcycle/biodynamo


text:
---
title: "Airflow Executors: Past, present and future"
slug: airflow-executors-past-present-and-future
speakers:
 - Niko Oliveira
time_start: 2023-09-19T12:00:00-04:00
time_end: 2023-09-19T12:25:00-04:00
day: 1
timeslot: 6
room: Ballroom A-B
track: New features/roadmap
images:
 - /images/sessions/2023/NikoOliveira-Executors.jpg

---

Executors are a core concept in Apache Airflow and are an essential piece to the execution of DAGs. They have seen a lot of investment over the year and there are many exciting advancements that will benefit both users and contributors.

This talk will briefly discuss executors, how they work and what they are responsible for. It will then describe Executor Decoupling (AIP-51) and how this has fully unlocked development of third-party executors. We’ll touch on the migration of “core” executors (such as Celery and Kubernetes) to their own package as well as the addition of new “3rd party” executors from providers such as AWS. Finally, a description/demo of Hybrid Executors, a proposed new feature to allow multiple executors to be used natively and seamlessly side by side within a single Airflow environment; which will be a powerful feature in a future full of many new Airflow executors.

repository: softwareguru/airflowsummit-website

gha_language: HTML

gha_license_id: NOASSERTION

repo_name: softwareguru/airflowsummit-website


text:
# Ionicons

## Build Icons

Build icons needs to be prefixed with `build-` to avoid conflicts with existing similar icons ex: `build-cart.svg`

* Add any new icons to the `src/` folder.
* Move any unused icons to the `svg/` folder.

## Updating Documentation

You can view the (theoretical latest cheatsheet for icons we support [here](http://buildcom.github.io/ionicons/cheatsheet.html))
The cheasheet is updated via master branch.

# More Info

The premium icon font for [Ionic Framework](http://ionicframework.com/). Designed by [@benjsperry](https://twitter.com/benjsperry).

Note: All brand icons are trademarks of their respective owners. The use of these trademarks does not indicate endorsement of the trademark holder by Drifty, nor vice versa.

Visit [ionicons.com](http://ionicons.com) and  check out the search feature, which has keywords identifying common icon names and styles. For example, if you search for “arrow” we call up every icon that could possibly be used as an arrow. We’ve also included each icon’s class name for easy copy/pasting when you’re developing!

We intend for this icon pack to be used with [Ionic](http://ionicframework.com/), but it’s by no means limited to it. Use them wherever you see fit, personal or commercial. They are free to use and licensed under [MIT](http://opensource.org/licenses/MIT).

## Build Instructions

This repo already comes with all the files built and ready to go, but can also build the fonts from the source.
It only requires Python 3 along with python package manager (pip3).

1) Using the system version of Python 3, use pip3 to install the following package:

    ```
    $ pip3 install svgutils
    ```

2) Add or subtract files from the `src/` folder you'd like to be apart of the font files.

3) Run the build command:

    ```
    $ python3 ./builder/generate.py
    ```

4) If you wish to verify the current cheatsheet with your latest changes, in the root of the repo folder run:

   ```
   $ python3 -m http.server 9000
   // open a browser and navigate to: http://localhost:9000/cheatsheet.html
   ```

## License

Ionicons is licensed under the [MIT license](http://opensource.org/licenses/MIT).

repository: buildcom/ionicons

repo_name: buildcom/ionicons


text:
# Linux Course

Intro
=====

| Importance Weight | Level 1 | Level 2 | Level 3 | Level 4| Level 5 |
| :---------------: | :------: | :-----: | :-----: | :----: | :-----: |
| Theory of the operating systems | X | O | O | O | O |
| System Architecture | O | X | O | O | O |
| Boot the system | O | O | X | O | O |
| Change runlevels | O | O | X | O | O |
| Linux Installation and Package Management | X  | O | O | O | O |
| Design hard disk layout | O | X | O | O | O |
| Install a boot manager | O | X | O | O | O |
| Manage shared libraries | X  | O | O | O | O |
| Use Debian package management | O | O | X | O | O |
| Use RPM and YUM package management | O | O | X | O | O |
| GNU and Unix Commands | O | O | X | O | O |
| Work on the command line | O | O | O | X | O |
| Process text streams using filters | O | O | X | O | O |
| Perform basic file management | O | O | O | X | O |
| Use streams, pipes and redirects | O | O | O | X | O |
| Create, monitor and kill processes | O | O | O | X | O |
| Modify process execution priorities | O | X | O | O | O |
| Search text files using regular expressions | O | X | O | O | O |
| Perform basic file editing operations using vi | O | O | X | O | O |
| Devices, Linux Filesystems, Filesystem Hierarchy Standard | O | X | O | O | O |
| Create partitions and filesystems | O | X | O | O | O |
| Maintain the integrity of filesystems | O | X | O | O | O |
| Control mounting and unmounting of filesystems | O | O | X | O | O |
| Manage disk quotas | X  | O | O | O | O |
| Manage file permissions and ownership | O | O | X | O | O |
| Create and change hard and symbolic links | O | X | O | O | O |
| Find system files and place files in the correct location | O | X | O | O | O |
| Shells, Scripting and Data Management | O | O | O | X | O |
| Customize and use the shell environment | O | O | O | X | O |
| Customize or write simple scripts | O | O | O | X | O |
| SQL data management | O | X | O | O | O |
| User Interfaces and Desktops | X | O | O | O | O |
| Install and configure X11 | O | X | O | O | O |
| Setup a display manager | X | O | O | O | O |
| Accessibility | X | O | O | O | O |
| Administrative Tasks | O | O | O | X | O |
| Manage user and group accounts and related system files | O | O | O | O | X |
| Automate system administration tasks by scheduling jobs | O | O | O | X | O |
| Localisation and internationalisation | O | O | X | O | O |
| Essential System Services | O | X | O | O | O |
| Maintain system time | O | O | X| O | O |
| System logging | O | O | X| O | O |
| Mail Transfer Agent (MTA) basics | O | O | X| O | O |
| Manage printers and printing | O | X| O | O | O |
| Networking Fundamentals | O | O | O | X| O |
| Fundamentals of internet protocols | O | O | O | X| O |
| Basic network configuration | O | O | O | X| O |
| Basic network troubleshooting | O | O | O | O | O |
| Configure client side DNS | O | X| O | O | O |
| Security | O | O | O | X| O |
| Perform security administration tasks | O | O | X| O | O |
| Setup host security | O | O | X| O | O |
| Securing data with encryption | O | O | X| O | O |

# Index

### Theory of the operating systems
### System Architecture
### Boot the system
### Change runlevels

# Linux Installation and Package Management

### Design hard disk layout
### Install a boot manager
### Manage shared libraries
### Use Debian package management
### Use RPM and YUM package management}

# GNU and Unix Commands
### Work on the command line
### Process text streams using filters
### Perform basic file management
### Use streams, pipes and redirects
### Create, monitor and kill processes
### Modify process execution priorities
### Search text files using regular expressions
### Perform basic file editing operations using vi

# Devices, Linux Filesystems, Filesystem Hierarchy Standard

### Create partitions and filesystems
### Maintain the integrity of filesystems
### Control mounting and unmounting of filesystems
### Manage disk quotas
### Manage file permissions and ownership
### Create and change hard and symbolic links
### Find system files and place files in the correct location

# Shells, Scripting and Data Management

### Customize and use the shell environment
### Customize or write simple scripts
### SQL data management

# User Interfaces and Desktops

### Install and configure X11
### Setup a display manager
### Accessibility

# Administrative Tasks

### Manage user and group accounts and related system files
### Automate system administration tasks by scheduling jobs
### Localisation and internationalisation

# Essential System Services

### Maintain system time
### System logging
### Mail Transfer Agent (MTA) basics
### Manage printers and printing

# Networking Fundamentals

### Fundamentals of internet protocols
### Basic network configuration
### Basic network troubleshooting
### Configure client side DNS

# Security

### Perform security administration tasks
### Setup host security
### Securing data with encryption


text:
# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Planned for 0.1.0 "MVP"

- LDAP backend connector
- Create, update, delete and list realms
- Create, update, delete and list units
- Create, update, delete and list users
- Create, update, delete and list groups
- Add and remove users from groups
- Export one or all units
- Import units
- Set user password, available hash formats : MD5, SMD5, SHA, SHA1, SSHA, SSHA1, SHA224, SSHA224, SHA256, SSHA256, SHA384, SSHA384, SHA512, SSHA512, SHA3-224, SSHA3-224, SHA3-256, SSHA3-256, SHA3-384, SSHA3-384, SHA3-512, SSHA3-512
- Password sent by mail to user when changed
- JSON, YAML, Table outputs for all commands
- JSON input for all commands

repository: fossabot/owl

repo_name: fossabot/owl


text:
# Simple-Face-Recognizer-Python-and-Yaml-
This is a Face Recognizer project.

GUI is used to make Adding New Faces, Training New Faces and Testing the Trained data simple.

Implemented using Python Tkinter and OpenCv.

Accuracy= ~70%.

# Working
It captures the new face image from users webcam, finds the face in current frame, crop the image around the face and save it into "dataset" directory. Training is done using OpenCv face recognizer which saves the trained data in .yaml in "trainer" directory. Testing: Streams video from the webcam, detect faces and compares it with trained data.

root/

  /main.py

  /haarcascade_frontalface_default.xml

  /dataset

  /trainer

# Note
Make sure to create required directories.

Will make another version, but with a Tensorflow model and a fully working GUI!!!


text:
---
title: "Points"
slug: "points"
description: "Create custom locations in your garden :round_pushpin:\n[Open this panel in the app](https://my.farm.bot/app/designer/points)"
---

* toc
{:toc}

If you would like to create custom locations in your garden for FarmBot to travel to that are _not_ plants, tool slots, or the home location, you can use **points**. Example uses of the points feature include:

* Points for FarmBot to move to that are out of the way when you're harvesting
* Points in each corner of the bed to serve as alternative "home" positions
* Points for FarmBot to travel to for daily photos

# Adding points
To add a point, click the  button in the points panel.

![Screen Shot 2019-11-30 at 5.00.05 PM.png](_images/Screen_Shot_2019-11-30_at_5.00.05_PM.png)

This will open the **add point** panel where you can provide the **name**, **color**, **X and Y coordinates**, and **radius** for the point. You can also click and drag in the map to define the coordinates and radius. Click SAVE to save the point.

![Screen Shot 2020-04-22 at 12.38.58 AM.png](_images/Screen_Shot_2020-04-22_at_12.38.58_AM.png)

# Editing points
To edit a point, click it in the points list or in the map (when the points panel is opened). This will open the **edit point** panel, allowing you to change anything about the point. Changes will be saved when you press the  button.

![Screen Shot 2019-11-30 at 5.01.21 PM.png](_images/Screen_Shot_2019-11-30_at_5.01.21_PM.png)

# Moving to a point
There are two ways to move FarmBot to a point. The first way is by clicking MOVE DEVICE TO LOCATION from the edit point panel.

![Screen Shot 2019-11-30 at 6.16.55 PM.png](_images/Screen_Shot_2019-11-30_at_6.16.55_PM.png)

The second way is from sequences. Simply select the point from the dropdown in a Move To command or location variable.

![Screen Shot 2019-11-30 at 6.17.04 PM.png](_images/Screen_Shot_2019-11-30_at_6.17.04_PM.png)

# Deleting points
To delete a point, click on it to open up the edit point panel. Then press the DELETE button.

> 📘
>
> Points cannot be deleted if they are in-use by any sequences, regimens, or events.

# What's next?

 * [Weeds](weeds.md)


text:
# Base Rate Cost Analysis Dashboard for vRealize Operations 8.0
---------

CPU, Memory, and Storage base rate costs are the primary driver of a VM's cost.  With this [vRealize Operations](https://www.vmware.com/products/vrealize-operations.html) dashboard, you can analyze current and historical CPU and Memory base rate costs per cluster along with Storage base rate costs per datastore.  Each heatmap has multiple views to compare base rate with other important metrics.  Easily compare demand and allocation based costs for clusters that have allocation model enabled for capacity planning.

![Base Cost Analysis Dashboard](https://raw.githubusercontent.com/notoriousbdg/vrops-dashboard-vm_base_cost_analysis/master/Dashboard.png)

## Installation
1. Import the dashboard at `Dashboards` / `Actions` / `Manage Dashboards` / `Import Dashboards`
![Import Dashboard](https://raw.githubusercontent.com/notoriousbdg/vrops-dashboard-vm_base_cost_analysis/master/Import_Dashboard.png)
2. Click `Browse...` then select the file named [Dashboard - VM Base Cost Analysis.zip](https://github.com/notoriousbdg/vrops-dashboard-vm_base_cost_analysis/raw/master/Dashboard%20-%20VM%20Base%20Cost%20Analysis.zip)
3. The dashboard should now be available in in the dashboard list
![Dashboard List](https://raw.githubusercontent.com/notoriousbdg/vrops-dashboard-vm_base_cost_analysis/master/Dashboard_List.png)

## Support
This dashboard requires vRealize Operation 8.0 Advanced or Enterprise edition.

Please open an [issue](https://github.com/notoriousbdg/vrops-dashboard-vm_base_cost_analysis/issues) for feedback.


text:
######
# Private Netflix
######

 a video on demand service for documentation

# How to save money on EC2
- Replace your manually created Autoscaling Group wiht a Elastigroup from SpotInst.
	- [https://spotinst.com/](https://spotinst.com/)
	-

# Your defalut login screen:

![login image](login.png)

* To get access, contact your system admin....because new user sign-up may be restricted or disabled.

# How to upload your own videos:
* Be aware that Streama can provide video is different formats, but the format you provide is of Critical importance to the viewers browser.  *mp4* files are the modern popular format.

1. Browse to the video upload page here:

 - TBS
2. Upload a file to the S3 bucket with a name that has no spaces.
 - Example: myvideo.mov
3. Navigate to the Video Manager page in Streama
4. Add your video to the system using the "select a file" option
5. After the video is added, be sure to add meta informtion for the video
6. [Console Recorder](https://chrome.google.com/webstore/detail/console-recorder-for-aws/ganlhgooidfbijjidcpkeaohjnkeicba/related?hl=en)
6. .....TBD

# Deployment Pipeline Setup and Usage:

Setup

```
chmod +x streama
export PATH=$PATH:./
streama --help

```

Usage

```

    USAGE: streama [OPTIONS]

    OPTIONS

	-h     Dispplay this help dialog
	-p     Displays the awsstudygurus tool variables
	         (varies based on branch)

	-d     Runs with defaults (not yet implemented)
	-r     Forces a reset (not yet implemented)

	-dp    Creates the Cloudformation stack for the Deployment pipeline
	         The Deployment pipeline deploys the app code and
	         must be created first
	-da    Pushes the application code to the origin for the Deployment pipeline
	         The Deployment pipeline must be created first.  Automation
	         for the AWS pipeline must be setup using the BitBucket repo plugin
	         https://marketplace.atlassian.com/apps/1211351/repository-mirror-plugin-for-bitbucket?hosting=server&tab=overview
	         https://github.com/ef-labs/stash-hook-mirror/wiki

	-delp  Deletes the Cloudformation stack and related resources for the Deployment pipeline
	-dela  Deletes the Cloudformation stacks and related resources for the Application

	--help 		    Same as -h
	--printvariables    Same as -p
	--deploy_pipeline   Same as -dp
	--deploy_app 	    Same as -da
	--destroy_pipeline  Same as -delp
	--destroy_app	    Same as -dela
	--default 	    not yet implemented
	--reset  	    not yet implemented

```

----------------------------------------------------------------------------------------------------------------------------

# Your AMI Pipeline diagram:

![AMI Pipeline](ami-builder-diagram.png)

# Your Serverless database:
[Amazon Aurora] (https://aws.amazon.com/rds/aurora/serverless/)

[ ![Amazon Aurora](database.png) ](https://www.youtube.com/watch?time_continue=3&v=IwyOyGs69N4)

# Notes:
What is Video on Demand?

- [https://en.wikipedia.org/wiki/Video_on_demand](https://en.wikipedia.org/wiki/Video_on_demand)
What is Netflix?

- [https://en.wikipedia.org/wiki/Netflix](https://en.wikipedia.org/wiki/Netflix)

- Original Code for Streama
- [https://github.com/streamaserver/streama](https://github.com/streamaserver/streama)
- Please give a Positive yelp review if you like this service :-)
	- This helps avoid from spending thou$ands of dollars on a paid Video on Demand system
- Mother, where do the AMIs come from?
	- Github [ami-builder-packer](https://github.com/awslabs/ami-builder-packer)
- .....more to come

repository: drumadrian/streama

repo_name: drumadrian/streama


text:
2D ULTRA Spine Reduced Model Simulation
=====================

This is the two-dimensional model of the ULTRA Spine tetraheda, using rigid-body dynamics.
Simulations available in both Simulink and MATLAB.
Currently, the focus is on a single tetrahedron.

As of 2016-02-23, development on this project has been moved away from this repository.
This code was moved to a private repository within the Berkeley Emergent Space Technologies Lab.
Please contact that group for information regarding access: Drew Sabelhaus<EMAIL_ADDRESS>

License
----------

Licensed under the Apache License v2.0.


text:
---
title: |
    479. Would You Win a Saviour's Blessing? - Church Hymnal 1941
metadata:
    description: |
        Church Hymnal 1941 479. Would You Win a Saviour's Blessing?.  Would you win a Saviour's blessing?  Freely, freely give;  Would you see His work progressing?  Freely, freely give;  Let your souls with love expand,  Open wide a liberal hand;  Would you follow God's command?  Freely, freely give.
    keywords:  |
        Church Hymnal 1941, adventhymnals, advent hymnals, Would You Win a Saviour's Blessing?, Would you win a Saviour's blessing?.
    author: Brian Onang'o
---

#### Advent Hymnals
## 479.
####  Church Hymnal 1941,

```txt
1.
Would you win a Saviour's blessing?
Freely, freely give;
Would you see His work progressing?
Freely, freely give;
Let your souls with love expand,
Open wide a liberal hand;
Would you follow God's command?
Freely, freely give.

2.
With a cheerful heart and willing,
Freely, freely give;
Like the dew its balm distilling,
Freely, freely give;
Have you little? Give your mite;
O how precious in His sight!
He your offering will requite;
Freely, freely give.

3.
Give to spread the grand old story,
Freely, freely give;
Give to speed the light of glory.
Freely, freely give;
Would you gain a rich reward
In the harvest of the Lord,
Then obedient to His word.
Freely, freely give.

```

- |   -  |
-------------|------------|
Title | Would You Win a Saviour's Blessing? |
Key | D Major |
Titles | undefined |
First Line | Would you win a Saviour's blessing? |
Author | Fanny J. Crosby
Year | 1941
Composer| Theo. E. Perkins |
Hymnal|  - |
Tune| FREELY GIVE |
Metrical pattern | |
# Stanzas |  |
Chorus |  |
Chorus Type |  |
Subjects | The Church and Doctrines: Tithes and Offerings |
Texts | undefined |
Print Texts |
Scripture Song |  |

repository: GospelSounders/adventhymnals

repo_name: GospelSounders/adventhymnals


text:
<h1 align="center">
Dai Plugin Template
</h1>

Fork this repository and replace the sample code with your own. Check `src/SampleService.js` and `src/index.js` for examples of how to structure your services and configuration.

___

Get started with:

```
$ yarn
$ git submodule update --init --recursive
```

* `yarn test:net` will launch a ganache testchain with all relevant Maker contracts deployed. Additional contracts would need to be added manually to the scripts in the testchain submodule, but it would be cool to have a script for injecting them at some point.

* Add tests with the suffix `*.spec.js` in the `test` directory and run them with `yarn test`.

* Build the code for production with `yarn build`.

repository: CPSTL/dai-plugin-template

repo_name: CPSTL/dai-plugin-template


text:
---
layout: page
title: "Coordination"
description: "Mechanism for declaring dependencies among Sources."
order: 6
disqus: 1
---

In Elements, sources can declare internal dependencies with respect to other sources,
similarly to what `CoordinatorLayout.Behavior`s do. This is done by overriding
the `dependsOn()` function:

```kotlin
override fun dependsOn(other: Source<*>): Boolean {
    return other is ContactsSource
}
```

This has a few consequences:

- the source receives all the dependencies objects in the `onPageOpened`. This also means that the callback is only fired when the dependency has provided items for that page.
- the source receives the `onPageChanged` callback, when the dependency has changed the page content.
- the source receives the `insertBefore()` and `insertAfter()` callbacks, for relative ordering

When a source declares dependencies, we give this source the opportunity to choose how and where to insert its elements.
If source `A` depends on source `B` and `C`, you can imagine, for each actual page, an imaginary page made of items from `B` and `C` for that page.

 ```kotlin
public open fun insertBefore(page: Page, dependencies: List<Element<*>>, element: Element<*>, position: Int, available: Int): Int {
    return available // Insert all our A items before B and C items
    return if (element.data == "foo") 1 else 0 // Add a header to "foo" objects
    return if (position % 4 == 0) 1 else 0 // Add an item each 4 items
}
 ```

Same goes for `insertAfter()`. Once the source has inserted all its `available` items, the methods are not called anymore.

repository: ramipa/Elements

repo_name: ramipa/Elements


text:
---
title: "getSanStorageCapabilityFlag"
description: "Retrieve a flag indicating that a VLAN on the router can be assigned to a host that has SAN disk functionality."
layout: "method"
tags:
    - "method"
    - "sldn"
    - "Hardware"
classes:
    - "SoftLayer_Hardware_Router"
aliases:
    - "/reference/services/softlayer_hardware_router/getSanStorageCapabilityFlag"
---
# [SoftLayer_Hardware_Router](/reference/services/SoftLayer_Hardware_Router)::getSanStorageCapabilityFlag

Retrieve a flag indicating that a VLAN on the router can be assigned to a host that has SAN disk functionality.

## Overview
Retrieve a flag indicating that a VLAN on the router can be assigned to a host that has SAN disk functionality.

-----

### Parameters
|Name | Type | Description |
| --- | --- | --- |

### Required Headers
* SoftLayer_Hardware_RouterInitParameters
* authenticate

### Optional Headers
* SoftLayer_Hardware_RouterObjectMask
* SoftLayer_Hardware_RouterObjectFilter
* SoftLayer_ObjectMask

### Return Values
* boolean


text:
## 514. Freedom Trail

In the video game Fallout 4, the quest "Road to Freedom" requires players to reach a metal dial called the "Freedom Trail Ring" and use the dial to spell a specific keyword to open the door.

Given a string ring that represents the code engraved on the outer ring and another string key that represents the keyword that needs to be spelled, return the minimum number of steps to spell all the characters in the keyword.

Initially, the first character of the ring is aligned at the "12:00" direction. You should spell all the characters in key one by one by rotating ring clockwise or anticlockwise to make each character of the string key aligned at the "12:00" direction and then by pressing the center button.

At the stage of rotating the ring to spell the key character key[i]:

* You can rotate the ring clockwise or anticlockwise by one place, which counts as one step. The final purpose of the rotation is to align one of ring's characters at the "12:00" direction, where this character must equal key[i].
* If the character key[i] has been aligned at the "12:00" direction, press the center button to spell, which also counts as one step. After the pressing, you could begin to spell the next character in the key (next stage). Otherwise, you have finished all the spelling.

Interesting questions. I can generate a solution, but how to get a minimum. Let me analyze the problem first.

if we want to move the hit the character, we need to move the index. Thus, for each time, we need to move the index to one of the character and spend some steps. But how to get the minimum steps? brute force - dfs each steps we can move to and count the minimal steps. Let see the size of the string.

if we use greedy policy, each time finding the minimum step, it's wrong, because current step will affect next step's move.

You can rotate the ring clockwise or anticlockwise by one place. I guess I shall use DP. each time we can rotate to target character by match it and by anti or clockwise. Thus, we need to use f[i,j,[0-1]] to represent we match i and j using 0 (clockwise) and 1 (anticlockwise). And only when i-1,j have matched, we can try to move pointer j to current index k where the i can match the character.

It's clear that if we match i and j the pointer will point to j.

* 1 <= ring.length, key.length <= 100
* ring and key consist of only lower case English letters.
* It is guaranteed that key could always be spelled by rotating ring.

I have spend all of my time. I wrote most part of code. But how to figure out the clockwise and anticlockwise by compare the j and k.

Let me see the std's solution. First of all, the std preprocess all related position of the characters.

oh actually, because we only need to use i-1 to genearate i's pointer, we can use a array to store i-1's value. Perfect code style in std.


text:
# listenbrainz-player
An audio player that allows exporting to listenbrainz

## How to use
* Install any missing dependencies with `npm install`
* Run the application with `node app.js`
* Go to `http://localhost:8080/` in your browser (only checked for Chrome)

repository: ojsquire/listenbrainz-player

repo_name: ojsquire/listenbrainz-player


text:
{{#template name="blaze-step05"}}

# Checking off and deleting tasks

Until now, we have only interacted with a collection by inserting documents. Now, we will learn how to update and remove them.

Let's add two elements to our `task` template, a checkbox and a delete button:

```html
<!-- replace the existing task template with this code -->
<template name="task">
  <li class="{{dstache}}#if checked}}checked{{dstache}}/if}}">
    <button class="delete">×</button>

    <input type="checkbox" checked="{{dstache}}checked}}" class="toggle-checked" />

    <span class="text">{{dstache}}text}}</span>
  </li>
{{lt}}/template>
```

We have added UI elements, but they don't do anything yet. We should add some event handlers:

```js
// In the client code, below everything else
Template.task.events({
  "click .toggle-checked": function () {
    // Set the checked property to the opposite of its current value
    Tasks.update(this._id, {$set: {checked: ! this.checked}});
  },
  "click .delete": function () {
    Tasks.remove(this._id);
  }
});
```

### Getting data in event handlers

Inside the event handlers, `this` refers to an individual task object. In a collection, every inserted document has a unique `_id` field that can be used to refer to that specific document. We can get the `_id` of the current task with `this._id`. Once we have the `_id`, we can use `update` and `remove` to modify the relevant task.

### Update

The `update` function on a collection takes two arguments. The first is a selector that identifies a subset of the collection, and the second is an update parameter that specifies what should be done to the matched objects.

In this case, the selector is just the `_id` of the relevant task. The update parameter uses `$set` to toggle the `checked` field, which will represent whether the task has been completed.

### Remove

The `remove` function takes one argument, a selector that determines which item to remove from the collection.

### Using object properties or helpers to add/remove classes

If you try checking off some tasks after adding all of the above code, you will see that checked off tasks have a line through them. This is enabled by the following snippet:

```html
<li class="{{dstache}}#if checked}}checked{{dstache}}/if}}">
```

With this code, if the `checked` property of a task is `true`, the `checked` class is added to our list item. Using this class, we can make checked-off tasks look different in our CSS.
{{/template}}

repository: lzl/tutorials

repo_name: lzl/tutorials


text:
# CompletePaymentRequest

### Description

Completes (captures) a payment.  By default, payments are set to autocomplete immediately after they are created. To complete payments manually, set `autocomplete` to false.

## Properties
Name | Type | Description | Notes
------------ | ------------- | ------------- | -------------

repository: square/connect-java-sdk

gha_language: Java

repo_name: square/connect-java-sdk


text:
# Signal event reference

## Events handled

The following events are handled by the Signal extension:

### Rules Engine Response Content

This event is used by the Signal extension to queue and send network calls that correspond to the [Sync PII Consequence](https://aep-sdks.gitbook.io/docs/using-mobile-extensions/mobile-core/rules-engine/rules-engine-consequence-details#sync-pii-consequence) or postback message.

This event is triggered by the Rules Engine when the events and conditions are met.

#### Event details

| **Event Type** | **Event Source** | **Paired** | **Paired Event** |
| :--- | :--- | :--- | :--- |
| com.adobe.eventType.rulesengine | com.adobe.eventSource.responseContent | No | - |

#### Data payload definition

The key-value pairs in this event correspond to postback and sync PII:

| Key | Friendly name | Type | Optional | Description |
| :--- | :--- | :--- | :--- | :--- |
| triggeredconsequence | Description URL | Map | Yes | Triggered Consequence details in the map.    For the detailed description of each rule consequence, see the following pages:  [Postback consequence](https://aep-sdks.gitbook.io/docs/using-mobile-extensions/mobile-core/rules-engine/rules-engine-consequence-details#postback-consequence) [Sync PII consequence](https://aep-sdks.gitbook.io/docs/using-mobile-extensions/mobile-core/rules-engine/rules-engine-consequence-details#sync-pii-consequence) [Open URL consequence](https://aep-sdks.gitbook.io/docs/using-mobile-extensions/mobile-core/rules-engine/rules-engine-consequence-details#open-url-consequence) |

#### Event data example

Postback message

```text
{
    "id"        : "9d40f5665d5bdbe96dcb3a24f4e4fe98d686a602",
    "type"      : "pb",
    "detail"    : {
        "templateurl"   : "https://www.endpoint.com/post/{%urlenc(~sdkver)%}",
        "templatebody"  : "{\"jsonkey\":\"jsonvalue\",\"sdkkey\":\"{%~sdkver%}\"}",
        "contenttype"   : "application/json",
        "timeout"       : 5
    }
}
```

#### Event data example

Open URL Request with a deeplink URL

```text
{
    "id"                : "48181acd22b3edaebc8a447868a7df7ce629920a",
    "type"            : "url",
    "detail"        : {
            "url"    : "myApp://HomePage"
    }
}
```

### Configuration Response Content

This event is dispatched on the Event Hub when a configuration change is processed. If a change occurs, the listener updates the privacy status.

#### Event details

| **Event Type** | **Event Source** | **Paired** | **Paired Event** |
| :--- | :--- | :--- | :--- |
| com.adobe.eventType.configuration | com.adobe.eventSource.responseContent | No | N/A |

#### Data payload definition

The Signal extension reads the global.privacy key from the configuration event.

| Key | Friendly name | Type | Optional | Description |
| :--- | :--- | :--- | :--- | :--- |
| global.privacy | String | String | No | Contains the mobile privacy status settings. |

#### Event data example

For a configuration change event:

```text
{ "global.privacy": "optedin" }
```

## Dispatched events shared states

The Signal extension does not dispatch events and does not share any shared state.