text:
---
layout: post
title: How to make a bigger custom AMI?
description: How to make a custom AMI with a size big enough
date: 2018-11-05 10:00
date_modified: 2018-11-07 10:00
tags:
  - AMI
  - AWS
  - Containers
  - Docker
  - Nextflow
  - Sarek
author: maxulysse
image:
  path: /assets/img/category/aws.png
---

Following this post set of posts on running Nextflow with AWS Batch:

- Nextflow blog: [Scaling with AWS Bach](https://www.nextflow.io/blog/2017/scaling-with-aws-batch.html)
- This blog: [Running CAW with AWS Batch](https://maxulysse.github.io/2017/11/16/Running-CAW-with-AWS-Batch/)
- Alexander Peltzer's blog: [Running nf-co.re pipelines with AWSBatch](https://apeltzer.github.io/post/01-aws-nfcore/)
- Anthony Underwood's blog: [Running Nextflow on AWS batch](https://antunderwood.gitlab.io/bioinformant-blog/posts/running_nextflow_on_aws_batch/)

I'm writing a small guide on how to make a custom [AMI](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html) with a big enough size to run some actual data with.

All the instructions are in fact listed in the [custom AMI Nextflow docs](https://www.nextflow.io/docs/latest/awscloud.html#custom-ami) and this solution is [there](https://forums.aws.amazon.com/message.jspa?messageID=811761#811761), but as I struggled once already, I prefer to document it for everyone and especially future me.

- On the AWS EC2 Dashboard, launch a new instance
- `1. Choose AMI`: Search for and choose `Amazon ECS-Optimized Amazon Linux AMI` in the `AWS MArketplace`
- `2. Choose Instance Type`: Choose a `t2.micro`
- `3. Configure Instance`: Leave default configuration
- `4. Add Storage`: Input `500 GiB` or any other size, but `22 GiB` is definitively not big enough for actual real life data.
- `5. Add Tags`: Do as you want
- `6. Configure Security Group`: Just make sure you can connect to the `EC2` Instance.
- `7. Review Instance Launch`: Procure a key pair that you have so that you can access your Instance
- Launch the newly created Instance (`Actions` -> `Instance State` -> `Start`), it should be easy to find
- Connect providing the key pair specified earlier.
- The `ssh` command provided by AWS is wrong, you need to replace `root` by `ec2-user`
- `sudo yum update` to update the system
- Install `awscli` using `conda`, so install `conda` first, and actually install `wget` before

```bash
sudo yum install wget
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86\_64.sh
bash Miniconda3-latest-Linux-x86\_64.sh -p /home/ec2-user/miniconda
# You can either source .bashrc or specify the path to conda
source .bashrc
conda install -c conda-forge awscli
```

- Edit this file: `/etc/sysconfig/docker-storage`
- Add this option `--storage-opt dm.basesize=500GB` (or specify any other size you prefer)
- `sudo service docker restart` to restart the docker daemon
- Check that you actually have the size you wanted with `docker info | grep -i base` and `docker info | grep -i data`

For all their support, thanks to:

- [alneberg](https://github.com/alneberg)
- [apeltzer](https://github.com/apeltzer)
- [KochTobi](https://github.com/KochTobi)
- [t-neumann](https://github.com/t-neumann)


text:
class: center, middle
# Section 8 - Build Images - The Dockerfile Basics
## 3 Extend Official Images
---

## docker build example - extend nginx image (1)
- In the following example we are going to use the files that are available in the `resources/dockerfile-sample-3` directory to build a custom Docker image.

```terminal
cd resources/dockerfile-sample-3
# ls
Dockerfile
index.html
```
> There are 2 files available:
>  - The **Dockerfile** which contains the instructions on how to build our custom image.
>  - THe *index.html* file that will be copied in the custom image.

---

## docker build example - extend nginx image (2)
### The [Dockerfile](../resources/dockerfile-sample-3/Dockerfile) of this example

---

## docker build example - extend nginx image (3a)
 - Lets see the default nginx index.html page before we  build our own custom nginx image:
```terminal
# docker container run -p 80:80 --rm nginx
```
> Note that we have used the `--rm` option that will delete the container automatically once it is stopped.

---

## docker build example - extend nginx image (3b)
![img_width_90](images/D_S8_L3_nginx_default_html.jpg)

---

## docker build example - extend nginx image (3)
 - Execute the `docker build` command to build out custom nginx Docker image:
```terminal
# docker build -t nginx-with-custom-html .
Sending build context to Docker daemon  3.584kB
Step 1/3 : FROM nginx:latest
 ---> 27a188018e18
Step 2/3 : WORKDIR /usr/share/nginx/html
 ---> Using cache
 ---> ac75a6485581
Step 3/3 : COPY index.html index.html
 ---> 617ee5012490
```

---

## docker build example - extend nginx image (4)
 - Execute the `docker container run` command to create a container from our custom image:
```terminal
# docker container run -p 80:80 --rm nginx-with-custom-html
```
![img_width_90](images/D_S8_L3_nginx_custom_html.jpg)

---

## docker build example - extend nginx image (5)
- Execute the `docker image ls` command to view the custom image that we have just created:
```terminal
# docker image ls
REPOSITORY                 TAG                 IMAGE ID            CREATED             SIZE
nginx-with-custom-html     latest              617ee5012490        5 minutes ago       109MB
gerassimos/private-repo1   test2               27a188018e18        12 days ago         109MB
nginx                      1.15                27a188018e18        12 days ago         109MB
nginx                      1.15.12             27a188018e18        12 days ago         109MB
nginx                      latest              27a188018e18        12 days ago         109MB
...
```

---

## docker build example - extend nginx image (5)
 - Finally upload the custom image to the Docker Hub registry:

```terminal
# docker image tag nginx-with-custom-html:latest gerassimos/nginx-with-custom-html:latest

# docker image ls
REPOSITORY                          TAG                 IMAGE ID            CREATED             SIZE
gerassimos/nginx-with-custom-html   latest              617ee5012490        12 minutes ago      109MB
nginx-with-custom-html              latest              617ee5012490        12 minutes ago      109MB
...

# docker push gerassimos/nginx-with-custom-html:latest
7f31d0f5f5ee: Pushing [================>]  4.096kB
fc4c9f8e7dac: Pushing [==========================>]  3.584kB
912ed487215b: Preparing
5dacd731af1b: Preparing
```

---

## LAB
 - Ref
 - D_S8_L3_Extend_Official_Images_lab.md
 - D_S8_L3_Dockerize_a_Spring_Boot_application.md

license: MIT

branch_name: refs/heads/master

extension: md

license_type: permissive


text:
# General Information:

Yahweh speaks of how the cities of Jerusalem and Samaria have been unfaithful to him in a metaphor in which he refers to them as two prostitutes. This metaphor continues through [Ezekiel 23:34](../23/34.md). (See: [[figs-metaphor]])

repository: mandolyte/uw-obsidian

branch_name: refs/heads/main

repo_name: mandolyte/uw-obsidian


text:
# Calculator
A calculator made in  javascript.

Handles basic arithmetic , negatives, exponents, and roots.

C: Clears input bar.
CP: Clears preview bar.
AN: Returns previous answer.
DE: Deletes input starting from the right.
RT: Root. Should be inputted as ( nth-root)R(what's being rooted). For example sqrt(4) is 2R4 or 2R(2+2). cube root(8) is 3R8 or 3R(10-2). Can also be chained  for example 2R(2R(4)) is sqrt(sqrt(4)).


text:
Handout
=======
A Microsoft Word template for handouts.

To change front matter, change the file Properties. The custom fields (Assignment, Professor, Semester, ClassName, and ClassNumber) control the main document header, as well as each page's header. Note: you will likely need to select-all + Update Field after modifying these values for them to take effect.

repository: muzny/neu-templates

repo_name: muzny/neu-templates


text:
# PBL #2

- Due: Sunday 03, December, 2017

## Description

The PBL #2 (Stage-2) consists in recognize traffic signs

## Instructions

- Improve Stage #1, if it is necessary
- Train a classifier to determine if the blobs are traffic signs or not

## Tips

1. Generate your dataset first
2. Then add to code

## Evaluation

Grade = 0.5 Code * 0.5 Presentation

#### Code

1. 1, if traffic recognition precision is lower than 50 %
2. 4, if traffic signs are recognized using SVM and pixels as input
3. 5, if 2, and another classifier is tested that improve previous result
4. 6, If 2 and 3, and another feature is used to train the classifier
5. 7, Optionals + mandatory + good quality code

#### Presentation

1. 10 minutes max
2. Describe solution
3. Give priority to analysis of results (+ and -)
4. Show results

## Submission

1. Submissions are through Platea.
2. Submission must include the file train.py compressed as a zip file
3. Submission must include the slides (presentation)
4. The zip file must be named pbl_2_rut.zip

## Bonus

1. 0.5 points is bug in the code are found


text:
# Resilience Testing with Docker for Mac & `tc`

## Synopsis

This repository contains a macOS Keynote presentation about how to perform
resilience tests under Docker for Mac with `tc`. It focuses on **delaying**
the (isolated) service and client network in order to test and detect
**connection** and **R/W** timeout problems.

## Guide

0.  Run the following command from the project root. This will install all
    dependencies.

        script/bootstrap

1.  Checkout the commit tagged `start`

2.  Demonstration:

        docker-compose up

3.  Run the presentation from `apps/resilience-talk/res/`

4.  Stop demo

        Ctrl + C
        docker-compose down -v

5.  Accompany presentation with commits starting with "ACT-"

## How to...

### ...add / delete a delay on the `slownet` network?

While running the demo, `screen` into `Docker for Mac` VM and investigate the
networks:

**Note**: `>` denote output examples.

    # show id of network "slownet"
    (host) docker network ls | grep slownet
    (host) > f5c43d88f2b6        playground_slownet      bridge              local

    # screen tunnel into the xhyve VM
    (vm) screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty

    # find network
    (vm) ifconfig | grep <id from first command>
    (vm) > br-f5c43d88f2b6 Link encap:Ethernet  HWaddr 02:42:E7:82:DF:BA

    # add a delay rule to the network interface
    (vm) tc qdisc add dev br-c0aca63f9c15 root netem delay 500ms

    # remove delay rule from the network interface
    (vm) tc qdisc del dev br-f5c43d88f2b6 root netem

### ...manage `screen` sessions?

    # disconnect from session but leave it open in the background
    Ctrl-a d
    # list sessions still running in the background
    screen -ls
    # reconnect to a session
    screen -r <session id>
    # kill the current active session
    Ctrl-a k

## Links

*   [Slow down your internet with tc](https://jvns.ca/blog/2017/04/01/slow-down-your-internet-with-tc/)
*   [How can I rate limit network traffic on a docker container](https://stackoverflow.com/questions/25497523/how-can-i-rate-limit-network-traffic-on-a-docker-container)
*   [Network emulation for Docker containers](https://alexei-led.github.io/post/pumba_docker_netem/)
*   [pgorczak/netz](https://github.com/pgorczak/netz)
*   [lukaszlach/docker-tc](https://github.com/lukaszlach/docker-tc)
*   [thombashi/tcconfig](https://github.com/thombashi/tcconfig)
*   [alexei-led/pumba](https://github.com/alexei-led/pumba)
*   [Apply NetEM WAN delay on a docker container interface // Docker Network for throttling a single service only](https://stackoverflow.com/questions/41899906/apply-netem-wan-delay-on-a-docker-container-interface)
*   [Docker for Mac Shell](https://gist.github.com/BretFisher/5e1a0c7bcca4c735e716abf62afad389)
*   [`tc` command examples](https://jvns.ca/blog/2017/04/01/slow-down-your-internet-with-tc/)
*   [more `tc` command examples](https://serverfault.com/questions/207162/simulating-a-slow-connection-with-tc)

repository: dreadwarrior/talk-resilience-dfm-tc

gha_language: PHP

gha_license_id: MIT

repo_name: dreadwarrior/talk-resilience-dfm-tc


text:
# Listerlyify
### Making Twitter Lists Awesome

## Description
Listerlyify is a Twitter app that makes creating and manging your Twitter lists easier. Twitter lists are a great way to organize the people and companies you follow into logical groups, but the Twitter website makes them a pain to use. The goal of Listerlyify is to make it as quick and simple as possible. There is a working alpha version of this website at https://listerlyify.herokuapp.com

![enter image description here](https://raw.githubusercontent.com/ncronquist/listerlyify/master/public/img/listgrideditor-xclose.png)

## Technologies Used
Listerlyify is built on Node.js and Express. The full list of npm dependency packages associated with this project:

- async - manage asynchronous javascript calls
- body-parser - middleware for body parsing
- connect-ensure-login - middleware to ensure a user is logged in; integrates with passport
- connect-flash - middleware for storing message in flash
- cookie-parser - used by passport to parse cookies
- ejs - embedded javascript templates
- express - web framework and router
- express-session - express extension for managing sessions
- passport - authentication middleware
- passport-twitter - twitter strategy for passport authentication
- pg - postgresql client
- pg-hstore - a postgresql module for serializing and deserializing json data
- sequelize - node.js object relational mapper
- sequelize-cli - the command line interface for sequelize - required locally to the project for deployment to Heroku
- Bootstrap - CSS library
- Data Tables - jQuery plugin for easy filtering and pagination of tables
- Epic Editor - Markdown editor implemented in javascript

## Approach

### Main Goal
The main goal of this project is to make editing your lists as easy as possible. On Twitter, it takes at least 4 clicks to add someone to a list. If you create a new list and want to add 10 people to that list, you have 40 clicks ahead of you. This project aims to cut that at least in half if not better.

### Secondary Goals
Some people spend a great deal of time curating their lists to match certain interest. Tech lists, music lists, political lists, sports lists, etc... These lists make Twitter better by allowing you to get to the content you want faster. Not everyone is interested in spending a lot of time to set up their lists though; so help them out by sharing your best lists on Listerlyify. The lists shared on Listerlyify are available for anyone to check out and decide if they want to subscribe. You can also comment on lists  to discuss who to FF (Follow Friday) and who to unfollow.

### Wireframes
![enter image description here](https://raw.githubusercontent.com/ncronquist/listerlyify/master/public/img/wireframe-01.jpg)
![enter image description here](https://raw.githubusercontent.com/ncronquist/listerlyify/master/public/img/wireframe-02.jpg)

### ER-Diagrams
In my original model, I was planning to store relationship of users subscribing to lists. Since users would be able to subscribe/unsubscribe from lists through Twitter and Listerlyify wouldn't know about those changes, that relationship was removed.

![enter image description here](https://raw.githubusercontent.com/ncronquist/listerlyify/master/public/img/Listerlyify-ERD.jpg)

### Trello Board
https://trello.com/b/Np7HI93M/listerlyify

## Issues
This web app is currently in alpha testing and definitely has some bugs. Known issues:
- Most errors are uncaught and simply sent to the user
- List grid editor currently only returns the first 200 of your friends
- jQuery Data Tables plugin gets a little slow when displaying more than 25 results in list grid editor
- List comments are orphaned when a list is un-shared
- List grid editor is too large for people working on screens with display resolutions less than 1080p

repository: ncronquist/listerlyify

gha_language: JavaScript

repo_name: ncronquist/listerlyify


text:
---
layout: default
---

# 1994-07-08 - Re: PGP Keys on a Floppy

## Header Data

From: thumper<EMAIL_ADDRESS>To<EMAIL_ADDRESS>Message Hash: c1f2e9367b39cbc1ba6373dee68bc05774cf2d17ed5fb012a834a9973c5065f7

Message ID<EMAIL_ADDRESS>Reply To<EMAIL_ADDRESS>UTC Datetime: 1994-07-08 06:07:11 UTC

Raw Date: Thu, 7 Jul 94 23:07:11 PDT

## Raw message

```
{% raw  %}From: thumper<EMAIL_ADDRESS>Date: Thu, 7 Jul 94 23:07:11 PDT
To<EMAIL_ADDRESS>Subject: Re: PGP Keys on a Floppy
In-Reply-To<EMAIL_ADDRESS>Message-ID<EMAIL_ADDRESS>MIME-Version: 1.0
Content-Type: text/plain

On Thu, 7 Jul 1994, Duncan Frissell wrote:

> C.><< some suggestion to keep keys secure on floppy>>
>
> DOS users can always encrypt their keyfile on a floppy encrypted with
> SecureDrive.  <G>
>

What I do is I zipped up my pubring.pgp and secring.pgp files into an
encrypted zip file, then used PGP's conventional encryption to encrypt
the zipfile.

Then to use pgp, I run a batch file that copies the encrypted zip files to a
ramdrive, decrypt the conventional encryption by prompting for the password,
then having pkunzip decompress the zipfile and prompting for it's password,
and then presto.

Thumper (yeah, just Thumper) =-=-=-=-=-=-=-=-=- GREP THIS NSA! =-=-=-=-=-=-=-
<EMAIL_ADDRESS>         - PGP NSA ViaCrypt 2600 Phrack EFF #hack LOD/H =
Finger for PGP 2.6 Pub Key   = 950 FBI MindVox ESN KC NUA QSD Hacker DEFCON -
Big Brother *IS* watching!   - SprintNet MCI AT&T HoHoCon DNIC TRW CBI 5ESS =

{% endraw %}
```

## Thread

+ Return to [July 1994](/archive/1994/07)

+ Return to "[Duncan Frissell <frissell@panix.com>](/authors/duncan_frissell_frissell_at_panix_com_)"
+ Return to "[thumper <thumper@kaiwan.com>](/authors/thumper_thumper_at_kaiwan_com_)"

+ 1994-07-07 (Thu, 7 Jul 94 07:43:43 PDT) - [PGP Keys on a Floppy](/archive/1994/07/84718f9292df6f53e2badb45561b2736be1daeef3ac6f50cf47eb0499b0dfefd) - _Duncan Frissell<EMAIL_ADDRESS>  + 1994-07-08 (Thu, 7 Jul 94 23:07:11 PDT) - Re: PGP Keys on a Floppy - _thumper<EMAIL_ADDRESS>


text:
# CxxPlugins

## Emphasis

> **Note:** library is still in development. For more examples use tests.
> Documentation can be generated if you pass `CXX_PLUGINS_BUILD_DOCUMENTATION=ON` to cmake
> or `-o enable_documentation=True` to conan

CxxPlugins is a library that aims to simplify creation of plugin systems.
It consists of:

+ `Polymorphic` and `PolymorphicPtr` templates,
   that allow storing polymorphic objects without inheritance.
+ `PolymorphicAllocator` that is implemented using polymorphic and has the same purpose
   as `std::pmr::polymorphic_allocator`.
   + `Mallocator` - default polymorphic allocator memory resource. Uses malloc to allocate/deallocate objects.
   + `StackAllocator` - memory resource that allocates objects on the stack.
+ Plugin configurator
    + Customize json file configuration with constexpr function calls
    + Load configurations and dlls automatically.

## Requirements

+ [Conan](https://conan.io/)
+ [CMake](https://cmake.org/) 3.15 and higher
+ [doxygen](https://www.doxygen.nl/download.html) (for generating documentation)
+ C++17 and higher

## Dependencies

+ boost/1.73.0
+ rapidjson/1.1.0
+ fmt/6.2.1
+ gtest/1.10.0 (for tests only)

## Build

```sh
$ mkdir build
$ cd build
$ conan install .. -o enable_tests=True -o enable_documentaiton=True -o shared=True
$ conan build ..
```

## Polymorphic basics

Let's say that you want to implement this kind of interface:
```cpp
class Foo {
public:
  virtual void do_something() = 0;
  virtual void do_something_else() {
    puts("Foo do_something_else");
  }
  virtual ~Foo() = default;
};

class Bar : public Foo{
public:
  void do_something() override {
    puts("Bar do_something");
  }
};

class Baz : public Foo{
public:
  void do_something() override {
    puts("Baz do_something");
  }
  void do_something_else() override {
    puts("Baz do_something_else");
  }
};
```

And in the main you will use this interface like this:
```cpp

int main() {
  auto foo_ptr = std::make_unique<Bar>();
  auto another_foo_ptr = std::make_unique<Baz>();
}
```
The cons of this approach are:
+ We are loosing value semantics
+ Everything is allocated on the heap.
> You can avoid it with singular objects, but if you are using containers you will deffinetely need something like unique_ptr
+ You can't make this interface work with existing libraries, as they don't inherit from your base type
+ It is hard to develop complex interfaces as it will involve multiple inheritance.

With `Polymorphic` it becomes much easier. Let's look at the example first:
```cpp

// Declare tags for our functions
struct do_something{};
struct do_something_else{};

// To define implementation for our function we need to
// create polymoprhicExtend function which consists of following parameters:
// + Tag declared earlier(we will not use it, it's only for function search)
// + Reference to an object
// + Other parameters of the function
// Because do_something doesn't have default implementation we will require user class to provide it
template<typename T>
void polymoirphicExtend(do_something /*unused*/, T& obj) {
  obj.do_something();
}

// Default implementation for do_something_else
template<typename T>
void polymorphicExtend(do_something_else /*unused*/, T& obj) {
  puts("Foo do_something_else");
}

// This concept will check if the class has do_something_else method
// You can use SFINAE for that if c++20 is not available for you
template<typename T>
concept HasDoSomethingElse  = requires(T a){
    {a.do_something_else()};
};
template<HasDoSomethingElse T>
void polymorphicExtend(do_something_else /*unused*/, T& obj) {
  obj.do_something_else();
}

// And finally let's declare our wrapper type
using Foo = Polymorphic<TaggedSignature<do_something, void()>, TaggedSignature<do_something_else, void()>>;
using FooPtr = PolymorphicPtr<TaggedSignature<do_something, void()>, TaggedSignature<do_something_else, void()>>;

// Now let's define our implementation classes:
class Bar {
public:
  void do_something() {
    puts("Bar do_something");
  }
};

class Baz {
public:
  void do_something() {
    puts("Baz do_something");
  }
  void do_something_else() {
    puts("Baz do_something_else");
  }
};
```

To use it we will do something like this:
```cpp
int main() {
  // We can do move construction of Bar here, but it's better to construct it in place
  auto foo = Foo(std::in_place_type_t<Bar>{});
  auto another_foo = Foo(std::in_place_type_t<Baz>{});
}
```

While interface looks weird and scary here are the positive sides:
+ Value semantics. `Polymorphic`s copy the content on copy construction and copy assignment.
  + You can use `UniquePolymorphic` for non-copyable types.
+ Allocation on the stack. We use small buffer optimization and you can use `GenericPolymorphic`
to specify the size for the objects that you want.
+ No inheritance at all. You can overload `polymorphicExtend` for any type(even for functions).
+ Any polymorphic can be 'upcasted' to `PolymorphcPtr` with less amount of functions.


text:
# 2.2. Host System Requirements
Your host system should have the following software with the minimum versions
indicated. This should not be an issue for most modern Linux distributions. Also
note that many distributions will place software headers into separate packages,
often in the form of “<package-name>-devel” or “<package-name>-dev”. Be sure to
install those if your distribution provides them.

Earlier versions of the listed software packages may work, but have not been
tested.

*   `bash`:      5.0.11(1)-release

!!! note
    `/bin/sh` should be a symbolic or hard link to `bash`

*   `binutils`:  2.34
*   `bison`:     3.5.1

!!! note
    `/usr/bin/yacc` should be a link to bison or small script that executes
    `bison`

*   `bzip2`:     1.0.8
*   `coreutils`: 8.31
*   `diffutils`: 3.7
*   `findutils`: 4.7.0
*   `gawk`:      5.0.1

!!! note
    /usr/bin/awk should be a link to gawk

*   `gcc`:       9.2.0
*   `git`:       2.25.0
*   `glibc`:     2.31
*   `grep`:      3.4
*   `gzip`:      1.10
*   `linux`:     5.5.3-arch1-1
*   `m4`:        1.4.18
*   `make`:      4.3
*   `patch`:     2.7.6
*   `perl`:      5.30.1
*   `rsync`:     3.1.3
*   `sed`:       4.8
*   `tar`:       1.32
*   `texinfo`:   6.7
*   `xz`:        5.2.4

!!! warning
    Note that the symlinks mentioned above are required to build glaucus using
    the instructions contained within this book. Symlinks that point to other
    software (such as dash, mawk, etc.) work in the chroot environment, but
    haven't been tested on the host system yet.

To see whether your host system has all the appropriate versions, and the
ability to compile programs, run the following (after having cloned the glaucus
repository):

```Shell
dash /home/glaucus/scripts/require
```

You should get an output in the following format:

```Shell
bash:      5.0.11(1)-release
binutils:  2.34
bison:     3.5.1
bzip2:     1.0.8
coreutils: 8.31
diffutils: 3.7
findutils: 4.7.0
gawk:      5.0.1
gcc:       9.2.0
git:       2.25.0
glibc:     2.31
grep:      3.4
gzip:      1.10
linux:     5.5.3-arch1-1
m4:        1.4.18
make:      4.3
patch:     2.7.6
perl:      5.30.1
rsync:     3.1.3
sed:       4.8
tar:       1.32
texinfo:   6.7
xz:        5.2.4
```

license: ISC


text:
# Wink Thermostat Translator
Translator for Wink thermostat (https://wink.com)

## Setup Your Hardware
Follow instructions in the Wink app to set up your thermostat with Wink. This is a pre-requisite
before using this translator to interact with your thermostat.

## Installing Dependencies
To install dependencies for this translator, run:

```bash
npm install
```

## Running Automated Unit tests
Unit tests do not require a physical or simulated device.
Once dependencies have been installed simply run:

```bash
npm run unittest
```

## Running Automated Integration tests

### 1. Run onboarding to get credentials

Using the opent2t-cli, following its readme to setup and perform onboarding and enumerate devices, you perform onboarding with this commmand:

```bash
node index.js -o opent2t-translator-com-wink-hub
```
The user will be asked for their Wink credentials (plus API key information) and then save the access_token to a json file.

### 2. Create the `tests/testConfig.json` file
This is where you can put credentials/config to drive this test (this file is added to .gitignore
to prevent inadvertent check-in). Copy the generated json file as the testconfig.json.

### 3. Run the tests

To run all the tests, run:

```bash
ava test.js
```

To run a specific test, run:

```bash
ava <test file path> <options>
```


text:
Air
=======

The light-weight and fat-free AJAX framework.

### Usage

AJAX calls can be easily made by using the following code:

```js
air.request({
	method: 'POST',
	url: 'http://api.horntell.com/profiles',
	params: {
		'first_name' => 'Mohit',
		'last_name' => 'Mamoria'
	},
	callback: function(success, http) {
		if(success) {
			console.log('All Done!');
		} else {
			console.log('Oops!');
		}
	}
});
```

***

I had to wrote this code to implement the same in my own projects and thought it would be useful to many others too. I do not have big plans for this small helper library as of now and it will evolve as the requirement evolves. Pull requests are always welcome. - [Mohit Mamoria](https://twitter.com/mohitmamoria)

### License

The Air project is open-sourced software licensed under [MIT license](http://opensource.org/licenses/MIT).

repository: owlgrin/air

repo_name: owlgrin/air


text:
<img src="https://assets.tokbox.com/img/vonage/Vonage_VideoAPI_black.svg" height="48px" alt="Tokbox is now known as Vonage" />

# UX Components

OpenTok UX Components are built with:

 - [TypeScript](http://www.typescriptlang.org/)
 - [React](https://reactjs.org/)
 - [PostCSS](https://github.com/postcss/postcss)

You can preview components using [React Storybook](https://storybook.js.org/).  Due to a current [issue](https://github.com/yarnpkg/yarn/issues/4489) with `yarn`, it's recommended that you use `npm`.

```bash
$ npm i
$ npm run storybook
```
You can view the storybook at `http://localhost:6006/`

*If you get an error regarding type declarations for a `css` file, you may need to run `npm run watch` in a separate terminal tab.*

Components are not yet available on `npm`, but you can test them in your own project by cloning the repo and then from the root directory:
```bash
 $ npm i
 $ yarn link
```
Within the root of your own project's directory, run `yarn link opentok-react-components`.  Then you can import and use components as normal:
```javascript
import { Button } from 'opentok-ux-components';
import 'opentok-ux-components/dist/style.css';

 . . .

  render() {
	  <div>
	    <Button text="hello" onClick={onClick} />
	  </div>
  }
```

## Development and Contributing

Interested in contributing? We :heart: pull requests! See the
[Contribution](CONTRIBUTING.md) guidelines.

## Getting Help

We love to hear from you so if you have questions, comments or find a bug in the project, let us know! You can either:

- Open an issue on this repository
- See <https://support.tokbox.com/> for support options
- Tweet at us! We're [@VonageDev](https://twitter.com/VonageDev) on Twitter
- Or [join the Vonage Developer Community Slack](https://developer.nexmo.com/community/slack)

## Further Reading

- Check out the Developer Documentation at <https://tokbox.com/developer/>

repository: opentok/ux-components

branch_name: refs/heads/develop

gha_language: TypeScript

repo_name: opentok/ux-components


text:
# Report

The agent is able to solve the environment in 150 episodes on average. The code is adapted from the ddpg repository. It is compatible with both versions of the environment, but results are provided for the second version with 20 agents.

## Learning Algorithm
The [DDPG algorithm](https://arxiv.org/abs/1509.02971) was used. The basic model is not able to solve the environment. The baseline implementation consists of training loop, agent and replay buffer. The implementation was amended by adding compatibility with second version of the environment.

### Model

Two types of the deep neural network were used: actor and critic models.

The actor architecture is based on 4 fully connected layers.

* FullyConnected shape: <state_size, 128>
* FullyConnected shape: <128, 256>
* FullyConnected shape: <256, 128>
* FullyConnected shape: <128, action_size>

The critic architecture is similar:

* FullyConnected shape: <state_size, 256>
* FullyConnected shape: <256 + action_size, 256>
* FullyConnected shape: <256, 128>
* FullyConnected shape: <128, 1>

For this project state_size is equal to ```33```, and action size is equal to ```4```. The ReLU and LeakyReLu functions were used as activation for the actor and critic models respectively.

### Optimizer
To optimize the weights of the models the agent uses Adam algorithm. Ornstein-Uhlenbeck noise is scaled by the factor 0.1 compared to the standard ddpg implementation. During training the update is being applied 10 times after every 20 timesteps.

### Hyperparameters
All hyperparameters are stored at the top of the ddpg_agent.py file. All of them are listed below.

PARAMETER  | Description | Default value
------------ | -------------| -------------|
GAMMA  | Discount factor | 0.99
TAU  | Controls soft update of a target network| 1e-3
BATCH_SIZE  | Batch size | 128
BUFFER_SIZE   | Replay buffer size | 2e5
LR_ACTOR   | Learning rate of the actor model | 1e-4
LR_CRITIC   | Learning rate of the critic model | 4e-4
UPDATE_EVERY   | How often turn on learning | 20
UPDATES   | How many updates during learning | 10
AGENTS   | The number of agents | 20

## Results
The weights are saved in files ```checkpoint_actor.pt``` and ```checkpoint_critic.pt``` for actor and critic respectively. The task was solved in ```147``` episodes. Best average score over 100 consecutive episodes is ```35.56```. The scores are plotted below.

![Trained Agent](scores.png)

```Logs from the terminal:```
```
'Academy' started successfully!
Unity Academy name: Academy
        Number of Brains: 1
        Number of External Brains : 1
        Lesson number : 0
        Reset Parameters :
                goal_speed -> 1.0
                goal_size -> 5.0
Unity brain name: ReacherBrain
        Number of Visual Observations (per agent): 0
        Vector Observation space type: continuous
        Vector Observation space size (per agent): 33
        Number of stacked Vector Observation: 1
        Vector Action space type: continuous
        Vector Action space size (per agent): 4
        Vector Action descriptions: , , ,

Episode 100     Average Score: 16.87    Time: 6.88
Episode 147     Average Score: 32.15    Time: 6.81
Environment solved in 147 episodes!     Average Score: 32.15
Episode 200     Average Score: 35.49    Time: 6.89
Best average score: 35.561539205137734
```
## Ideas for Future Work

There are many other potential algorithms that can solve the task. My plans are listed below:

-  Implement PPO.
-  Implement D4PG.

repository: pawkaz/deep-reinforcement-learning

repo_name: pawkaz/deep-reinforcement-learning


text:
# Unit Test Navigate

Adds your unit test definition calls to the symbols list for easier navigation of javascript and typescript unit test scripts. Supports `describe`, `it`, `suite`, and `test` calls.

Open your test file and trigger "Go to symbol in file" (cmd/ctrl+shift+o)

Before:

![before](https://raw.githubusercontent.com/roblourens/vscode-unittest-navigate/master/images/before.gif)

After:

![after](https://raw.githubusercontent.com/roblourens/vscode-unittest-navigate/master/images/after.gif)

## Notes
It will work when one of these function calls is the first non-whitespace text on the line, and its first parameter is a string.

repository: smith/vscode-unittest-navigate

repo_name: smith/vscode-unittest-navigate


text:
# sys-apps/shadow

### acl
Pass the `--with-acl` option to the configure script. Enable ACL support for the `useradd` and `usermod` commands. Copy ACL permissions when creating user home directory from skeleton files via `useradd` (`-m` option) and when moving home directory via `usermod` (`-m` option).

This flag should be toggled system-wide, otherwise there is a high probability of losing important ACLs.

### audit
Pass the `--with-audit` option to the configure script. Add logging support via the Kernel Audit Subsystem using a `libaudit` library for a `useradd`, `groupadd`, `userdel`, `groupdel`, `gpasswd` and other users and groups manipulation commands.

This flag can be safely enabled.

### bcrypt
Pass the `--with-bcrypt` option to the configure script. Provide an ability to use `bcrypt` hashing function for storing passwords, instead of a SHA-based one. The `bcrypt` password-hashing algorithm derived from Bruce Schneier's Blowfish block cipher, which takes advantage of the slow Blowfish key schedule to make password-checking inherently CPU-intensive, so that password-cracking attempts are slower and more difficult.

It is recommended to enable this flag to greatly improve stored password hashes resistance to brute-force and rainbow attacks.

### cracklib
Pass the `--with-libcrack` option to the configure script. Use a `libcrack` library when creating or changing a password for a user or a group using `passwd` command to verify password requirements, such as complexity, blacklist dictionary check and others.

It is safe to disable this flag, however it can add an additional layer of security for the multi-user system.

### nls
Pass the `--enable-nls` option to the configure script. Provide a localization support using gettext. Programs messages and prompts can be translated into system locale and be displayed in various languages.

This flag can be safely disabled, unless there is a need for non-English languages.

### pam
Pass the `--with-libpam` option to the configure script. When disabled, install a `login.access` and a `limits` configuration files into `/etc` directory (normally provided by PAM), also disable a `CONSOLE`, set a `MAIL_CHECK_ENAB` option to `no`, a `SU_WHEEL_ONLY` to `yes`, a `CRACKLIB_DICTPATH` to `/usr/lib/cracklib_dict`, a `LOGIN_RETRIES` to 3 and an `ENCRYPT_METHOD` to `SHA512`. Otherwise, remove a `CHFN_AUTH`, `CONSOLE`, `CRACKLIB_DICTPATH`, `ENV_HZ`, `ENVIRON_FILE`, `FAILLOG_ENAB`, `FTMP_FILE`, `LASTLOG_ENAB`, `MAIL_CHECK_ENAB`, `MOTD_FILE`, `NOLOGINS_FILE`, `OBSCURE_CHECKS_ENAB`, `PASS_ALWAYS_WARN`, `PASS_CHANGE_TRIES`, `PASS_MIN_LEN`, `PORTTIME_CHECKS_ENAB`, `QUOTAS_ENAB`, `SU_WHEEL_ONLY` options from `login.defs` configuration file, avoid installing manpages and pam.d files (`login`, `passwd` and `su`) that are provided by [sys-auth/pambase](../sys-auth/pambase.md).

Integrate with PAM infrastructure to perform authentication when doing a user or a group manipulation tasks.

This flag should be enabled if target system to be using a PAM stack for a user authentication.

### selinux
Pass the `--with-selinux` option to the configure script. Enable an ability to properly run under SELinux-restricted environment. A `useradd` and a `userdel` command will copy a security context from files when creating a user home directory from skeleton files or moving a home directory to the different place. Both of these commands also will support `-Z` option to add or remove SELinux user mapping for the user account. A `vipw` command will copy a SELinux context from original files to backup files. A `chsh` command will consult SELinux in order to check for permissions before changing a shell.

This flag should only be ever toggled system-wide, e.g. as part of SELinux-enabled Portage profile.

### skey
Pass the `--with-skey` option to the configure script. Provide an ability to authenticate using S/Key one-time password system.

It is safe to disable the flag.

### split-usr
When enabled, install the `passwd` command into the `/bin` directory and create a symlink pointing to it inside of the `/usr/bin` directory.

This flag should be enabled if the system uses separate `/usr` partition to ensure that the `passwd` command is available during the early boot.

### su
Pass the `--with-su` option to the configure script. Build and install the `su` binary, which allows to run a command with substitute user and group ID, and is designed mostly for unprivileged users.

This flag should normally be enabled, however it can be disabled if there is no need to run privileged commands as regular users or alternative mechanisms exist to run such commands.

### verify-sig
Perform signature verification of the upstream source code archive before extracting it and building the package.

It is safe to disable this flag.

### xattr
Pass the `--with-attr` option to the configure script. Provide an ability for a `useradd` and a `usermod` commands to copy eXtended ATTRibutes from source files into destination when creating a user home directory from skeleton files or moving home directory to different place. Enable an additional logic to exclude ACLs when copying XATTRs to avoid an unexpected permissions result while copying files between filesystems with and without an ACL support.

This flag should be toggled system-wide, otherwise there is a high chanse of losing XATTRs when using different tools.

license: Unlicense

repository: filakhtov/gentoo-use-docs

repo_name: filakhtov/gentoo-use-docs


text:
![Chatbot-NLTK](https://socialify.git.ci/aryasoni98/Chatbot-NLTK/image?description=1&font=KoHo&language=1&owner=1&pattern=Circuit%20Board&theme=Dark)

## Pre-requisites
**NLTK(Natural Language Toolkit)**

[Natural Language Processing with Python](http://www.nltk.org/book/) provides a practical introduction to programming for language processing.

For platform-specific instructions, read [here](https://www.nltk.org/install.html)

### Installation of NLTK
```
pip install nltk
```
### Installing required packages
After NLTK has been downloaded, install required packages
```
import nltk
from nltk.stem import WordNetLemmatizer
nltk.download('popular', quiet=True) # for downloading popular packages
nltk.download('punkt')
nltk.download('wordnet')
```


text:
# 2. Architectural Thinking

Author:
  - [gkz](https://twitter.com/gkzvoice)

See the following, if you want to see the memos of other chapters.
  - [fundamentals-of-software-architecture.md](../fundamentals-of-software-architecture.md)

## 2-1. Overview

This chapter says as follows:

- Diffs between architects and developers
  > understanding the difference between architecture and design and knowing how to collaborate with development teams to make architecture work.

  > about having a wide breadth of technical knowledge while still maintaining a certain level of technical depth, allowing the architect to see solutions and possibilities that others do not see.

 > about understanding, analyzing, and reconciling trade-offs between various solutions and technologies.

 > about understanding the importance of business drivers and how they translate to architectural concerns.

repository: gkzz/ReadingLog

repo_name: gkzz/ReadingLog


text:
# :musical_note: JioSaavn API

An Unofficial API for JioSaavn that allows downloading songs in 320kbps quality for free.

## :sparkles: Features

- :rocket: Blazing fast
- :notes: Download High Quality Songs (320kbps).
- :lock: Rotating User-Agents to avoid detection.
- :sunrise: Download High quality song/album thumbnails.
- :mag_right: Search support for songs and albums.
- :musical_score: Lyrics for supported songs.
- :heart: Open-Source
- :cloud: Lightweight and has minimal dependencies.
- :package: Host your own instance on [vercel](https://vercel.com).

## :mag_right: Usage

###  For how to use, check out [Documentation](https://docs.saavn.me).

## :construction_worker: Local development

### Prerequisites

- NodeJS v10+
- Vercel CLI : To install Vercel CLI globally use `npm i -g vercel`

### Run locally

- Clone the Repo

  ```
  $ git clone https://github.com/sumitkolhe/jiosaavn-api

  $ cd jiosaavn-api
  ```

- Install the dependencies

  ```
  npm install
  ```

- Start development server

  ```
  npm start
  ```

- Deploy to your vercel account

  ```
  npm run deploy
  ```

## :rocket: Deploy to Vercel

**You can directly deploy this repo to your Vercel account by clicking the button below.**
<br>

[![Deploy with Vercel](https://vercel.com/button)](https://vercel.com/import/project?template=https://github.com/sumitkolhe/jiosaavn-api)

## ✍ Authors

- [**Sumit Kolhe**](https://github.com/sumitkolhe) - _Author_

## 📜 License

This project is licensed under the [MIT License](https://opensource.org/licenses/MIT) - see the [LICENSE](LICENSE) file for details.

repository: blueedgetechno/jiosaavn-api

repo_name: blueedgetechno/jiosaavn-api


text:
---
layout:     post
title:      "Anti Anti-spider Strategy"
subtitle:   "Common strategies for coping with anti-spider"
date:       2016-02-15 12:00:00
author:     "aaronice"
keyword:    ["web", "crawler", "scrape", "spider", "data"]
header-img: ""
---

# The Anti Anti-scraping Strategy

Before knowing how to prevent being blacklisted while scraping, we need to understand how a website detect web crawlers.

There're multiple anti-scraping mechanisms to distinguish a spider from a normal user. Some of the methods are[[1]](https://learn.scrapehero.com/how-to-prevent-getting-blacklisted-while-scraping/):

>1. Unusual traffic/high download **rate** especially from a **single** client/or IP address within a short **time** span.
>
>2. **Repetitive** tasks performed on the website – based on an assumption that a human user won’t perform the same repetitive tasks all the time.
>
>3. Detection through **honeypots** – these honeypots are usually links which **aren’t visible** to a normal user but only to a spider. When a scraper/spider tries to access the link, the alarms are tripped.

Corresponding to the anti-scraping mechanisms are the following strategies:

## IP Addresses Rotation

IP blacklisting is the perhaps the easiest way of anti-scraping. By creating a pool of IP addresses, and making requests using different IP, will make it harder for server to detect crawlers.

- Use a list of IP from proxy services
  - Randomly pick an IP for certain time interval

## Cookie Rotation
Cookies are encrypted data stored in client side, some website use cookie to identify user. If a user client sends requests in high frequency, it's possible to be identified as suspicious crawler, thus denied access.

- Customize and manage a pool of cookies
  - Sending request to server without cookie headers, parse the package for set-cookie value; store it in cookie collector;
  - Get cookie from cookie collect and if the cookie is not usable, then delete it from cookie collector;
  - Manage the cookie collector with timestamp, in order for the spider to get the oldest cookie first each time.

- Disable cookie
  - By disabling cookie, it may help prevent being banned by some websites which use cookie to identify crawlers.

## User-Agent Imitation

To disguise as browser, one of the approaches is to modify the User-Agent. User-Agent is a string in request header that contains user-agent information, such as the version of web browser, client, operating system etc.

- Spoof the User-Agent by making a list of user agents and randomly pick one for each request. Setting the User-Agent to a common web
browser instead of using the default ones.

## Rate Limit

Slow down the crawling, treat the websites nicely, don't overwhelm it, or DDoS the server.

- Put some random sleeps between requests
- Add some delays after crawling a number of pages
- Use the lowest number of concurrent requests possible

## Be Careful About Honeypots

These honeypots usually are links that normal user can’t see but a spider can[[1]](https://learn.scrapehero.com/how-to-prevent-getting-blacklisted-while-scraping/). They might be having the CSS style "display:none". Thus the detection of honeypots can be tricky.

## Avoid Repetitive Crawling Pattern

Some websites implemented intelligent anti-crawling mechanisms, thus a repetitive action will be likely detected as spider. To make a spider looks like a human, add random clicks, mouse movement, random actions etc. Using automated testing tool like Selenium may simulator normal "human actions".

## Reference

1. [HOW TO PREVENT GETTING BLACKLISTED WHILE SCRAPING](https://learn.scrapehero.com/how-to-prevent-getting-blacklisted-while-scraping/)

2. [防止爬虫被墙的方法总结](http://www.dianacody.com/2014/10/01/spider_5.html)

3. [应对反爬虫之换Cookie](https://medium.com/@Masutangu/%E5%BA%94%E5%AF%B9%E5%8F%8D%E7%88%AC%E8%99%AB%E4%B9%8B%E6%8D%A2cookie-d3b48b02d0e6)

extension: markdown


text:
# MessagePack for C# (.NET, .NET Core, Unity, Xamarin)

[![NuGet](https://img.shields.io/nuget/v/MessagePack.svg)](https://www.nuget.org/packages/messagepack)
[![NuGet](https://img.shields.io/nuget/vpre/MessagePack.svg)](https://www.nuget.org/packages/messagepack)
[![Releases](https://img.shields.io/github/release/neuecc/MessagePack-CSharp.svg)](https://github.com/neuecc/MessagePack-CSharp/releases)

[![Join the chat at https://gitter.im/MessagePack-CSharp/Lobby](https://badges.gitter.im/MessagePack-CSharp/Lobby.svg)](https://gitter.im/MessagePack-CSharp/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
[![Build Status](https://dev.azure.com/ils0086/MessagePack-CSharp/_apis/build/status/MessagePack-CSharp-CI)](https://dev.azure.com/ils0086/MessagePack-CSharp/_build/latest?definitionId=2)

The extremely fast [MessagePack](http://msgpack.org/) serializer for C#. It is 10x faster than [MsgPack-Cli](https://github.com/msgpack/msgpack-cli) and outperforms other C# serializers. MessagePack for C# also ships with built-in support for LZ4 compression - an extremely fast compression algorithm. Performance is important, particularly in applications like game development, distributed computing, microservice architecture, and caching.

![Perf comparison graph](https://cloud.githubusercontent.com/assets/46207/23835716/89c8ab08-07af-11e7-9183-9e9415bdc87f.png)

MessagePack has compact binary size and full set of general purpose expression. Please see the [comparison with JSON, protobuf, ZeroFormatter section](doc/comparison.md). Learn [why MessagePack C# is fastest](doc/performance.md).

## Getting started

1. [Install MessagePack in your project](doc/installation.md).
1. Check out our [quick start](doc/quickstart.md) tutorial.
1. Check out the rest of [our documentation](doc/index.md).

## Author Info

Yoshifumi Kawai(a.k.a. neuecc) is a software developer in Japan.
He is the Director/CTO at Grani, Inc.
Grani is a mobile game developer company in Japan and well known for using C#.
He is awarding Microsoft MVP for Visual C# since 2011.
He is known as the creator of [UniRx](http://github.com/neuecc/UniRx/)(Reactive Extensions for Unity)

* Blog: [https://medium.com/@neuecc](https://medium.com/@neuecc) (English)
* Blog: [http://neue.cc/](http://neue.cc/) (Japanese)
* Twitter: [https://twitter.com/neuecc](https://twitter.com/neuecc) (Japanese)

repository: goyzhang/MessagePack-CSharp

repo_name: goyzhang/MessagePack-CSharp


text:
# react-leaflet-geosearch [![npm version](https://img.shields.io/npm/v/react-leaflet-geosearch.svg)](https://www.npmjs.com/package/react-leaflet-geosearch)
React Leaflet geosearching/geocoding control build on top of [React-Leaflet](https://github.com/PaulLeCam/react-leaflet).

The SearchControl is using the plugin from [leaflet-geosearch](https://github.com/smeijer/leaflet-geosearch)

# Getting started

```
import { SearchControl, OpenStreetMapProvider } from 'react-leaflet-geosearch'

//here you can add some styles
import './example/react-leaflet-geosearch.css';

....
const prov = OpenStreetMapProvider();
const GeoSearchControlElement = withLeaflet(SearchControl);

 <Map center={[42.09618442380296, -71.5045166015625]} zoom={2} zoomControl={true}>
               ...Layers
                <GeoSearchControlElement provider={prov} showMarker= {true} showPopup={false} popupFormat={({ query, result }) => result.label}
                  maxMarkers={3}  retainZoomLevel= {false}  animateZoom= {true} autoClose= {false}
                  searchLabel={'Enter address, please'} keepResult= {true} />
</Map>

```

For more details on how to use this plugin check the example.

repository: A-quinonero/react-leaflet-geosearch

repo_name: A-quinonero/react-leaflet-geosearch


text:
## Welcome to "Hello World" with GitHub Actions

This course will walk you through writing your first action and using it with a workflow file.

**Ready to get started? Navigate to the first issue.**

# Container Action Template

To get started, click the `Use this template` button on this repository [which will create a new repository based on this template](https://github.blog/2019-06-06-generate-new-repositories-with-repository-templates/).

For info on how to build your first Container action, see the [toolkit docs folder](https://github.com/actions/toolkit/blob/master/docs/container-action.md).

[container-action](https://github.com/actions/container-action).

repository: nnmuniz/hello-github-actions

repo_name: nnmuniz/hello-github-actions


text:
# :warning: DEPRECATED :warning:
In order to make BigTest development faster and friction free, we've consolidated all of our individual projects into a [single repository on the Frontside Organization](https://github.com/thefrontside/bigtest). We'd love to see you there!

_note: the last release from this repository was 0.5.2_

## @bigtest/mocha

Convergent Mocha functions for testing against asynchronous states

## Synopsis

``` javascript
import { describe, beforeEach, it } from '@bigtest/mocha';
import { expect } from 'chai';

describe('clicking my button', () => {
  beforeEach(() => $button.click());

  // repeatedly asserts until passing
  it('shows a loading indicator', () => {
    expect($button.className).to.include('is-loading');
  });

  // repeatedly asserts it is passing until the timeout
  it.always('does not navigate away', () => {
    expect(app.location).to.equal('/')
  }).timeout(200);
})
```

## Convergent Assertions

Typically, when testing asynchronous states (such as rendered content
in an application) your tests need to run at just the right moment so
that they are executed in the correct context.

Converging on a state means asserting against a state until the
assertion passes, in which case you have successfully converged on
that state!

This package uses
[`@bigtest/convergence`](https://github.com/thefrontside/bigtest/tree/master/packages/convergence)
to repeatedly run assertions until they pass, or until the timeout has
expired. Performing tests in this way allows them to pass the moment
the desired state is achieved. This results in very fast tests when
testing asynchronous things.

_Read the `@bigtest/convergence` [docs on
convergences](https://github.com/thefrontside/bigtest/tree/master/packages/convergence#why-convergence)
for more info as to why converging on a desired state is better than
trying to time it properly._

## How does it work?

For the most part, you write tests in the exact same way that you're
used to writing tests with Mocha. The only difference is that this
package wraps Mocha's `it` in a convergence helper so that any
assertions you write using `it` become convergent assertions that
allow you to easily test asynchronous states.

This package also wraps the Mocha hooks `before`, `after`,
`beforeEach`, and `afterEach` to support automatically timing and
running returned `Convergence` instances from `@bigtest/convergence`.

 This will result in your side-effect being run
multiple, perhaps even hundreds of times.

For this reason, you should keep your side-effect producing code in
hooks and **out of your assertions.** These "pure assertions" also
help your tests be more readable and explicit.

#### :no_entry_sign: do not do this:

``` javascript
describe('my button', () => {
  it('shows a loading indicator after clicking', () => {
    // this will be called every time the assertion runs
    $button.click();

    // it might take a few milliseconds for any side-effects to
    // happen, so this might fail the first time and cause this entire
    // assertion to run again, thus clicking the button again
    expect($button.className).to.include('is-loading');
  });
});
```

#### :white_check_mark: do this:

``` javascript
describe('clicking my button', () => {
  // keep side-effects inside hooks
  beforeEach(() => $button.click());

  // a pure assertion has no side-effects; even if it fails, it can
  // be run again and again without consequence
  it('shows a loading indicator', () => {
    expect($button.className).to.include('is-loading');
  });
});
```

#### Asserting that something _has not happened_

Another common scenario is asserting that something **has not**
happened. If you were to test for this normally (or even with a
convergent assertion above) the test could potentially pass
successfully before a side-effect has time to even happen.

In these scenarios, you want to converge when the state meets an
expectation _for a given period of time_. In other words, "if this
assertion remains true for X amount of time, this test is considered
to be passing."

`@bigtest/mocha` provides an `it.always` method to do just this. This
method will run the assertion throughout the entire timeout period
ensuring it never fails. When the assertion does fail, the test
fails. If the assertion never fails, it will pass just after the
timeout period.

``` javascript
describe('clicking my button', () => {
  beforeEach(() => $button.click());

  // the default timeout for it.always is 100ms
  it.always('does not navigate away for at least 1 second', () => {
    expect(app.location).to.equal('/');
  }).timeout(1000);
});
```

### Convergent Hooks

Sometimes you may attempt to perform an async task to find it fails
due to a preconceived state not being met. For example, you can't
click a button if it doesn't exist in the DOM. You may use
`@bigtest/convergence` to converge on these states and return
convergences inside of your hooks. The hooks provided by
`@bigtest/mocha` will automatically set the timeout and run returned
`Convergence` instances.

``` javascript
describe('clicking my button', () => {
  // @bigtest/mocha will wait for a returned Convergence to converge
  // before continuing with the assertions
  beforeEach(() => new Convergence()
    .once(() => expect($button).to.exist)
    .do(() => $button.click()));

  it('shows a loading indicator', () => {
    expect($button.className).to.include('is-loading');
  });
});
```

Check out the `@bigtest/convergence` [API
docs](https://github.com/thefrontside/bigtest/tree/master/packages/convergence#using-convergence)
for working with the `Convergence` class.

### Pausing Tests

Pausing tests can be very useful when debugging. It allows you to
investigate your application during a critical moment in your testing
suite. Mocha does not have a convinient way to pause tests, but
`@bigtest/mocha` helps alleviate this by providing an `it.pause`
method. It works by setting the current timeout to `0` and gives Mocha
a promise that never resolves. This effectively pauses the entire
suite until you remove `it.pause` and restart your tests.

``` javascript
describe('clicking my button', () => {
  beforeEach(() => $button.click());

  // if the class is never set, this test will fail; it.pause allows
  // us to investigate the app at this point in the test suite
  it.pause('shows a loading indicator', () => {
    expect($button.className).to.include('is-loading');
  });
});
```

repository: bigtestjs/mocha

repo_name: bigtestjs/mocha


text:
# Sarrameana albidoplumbea (Hook. f. & Taylor) Farkas SPECIES

#### Status
ACCEPTED

#### According to
The Catalogue of Life, 3rd January 2011

#### Published in
Biblthca Lichenol. 58: 98 (1995)

#### Original name
Lecidea albidoplumbea Hook. f. & Taylor

### Remarks
null

repository: mdoering/backbone

repo_name: mdoering/backbone


text:
# Configuration to deploy and or test the bot

You can decide to run one or more of the three components (bot, datadb and authdb) locally instead of inside a docker container. If you decide to do so, please follow the instructions for a [local setup](#Local-setup) before continuing.

## Docker instructions

### Docker Setup

#### Installation

The easiest way to install docker is following [the official docker installation guidelines](https://docs.docker.com/install/). Be sure to have a look at the [post-installation steps](https://docs.docker.com/engine/install/linux-postinstall/) if you're on Linux.

Alternatively, you could use `snap install docker`, but note that the snap installs are built and maintained by Canonical, not Docker.

You'll also need `docker-compose`:
```
sudo apt install docker-compose
```

#### Prepare your docker configuration
If you previously ran ``./configure`` when following our guide at [README.md](../README.md), just continue with the next section.
Otherwise, you'll have to set some variables first.

Find [.env.example](.env.example) and rename your local copy of it to '.env'. Then, modify the values held within:

- MYSQL_ROOT_PASSWORD will the root password for both the authentication and the data database
- MYSQL_USER has to be the same as the user in the MySQL field of your config.ini
- MYSQL_PASSWORD has to be the same as the password in the MySQL field of your config.ini

The values in .env will be read by docker-compose and make sure you have everything you need to access your databases once they are running.

#### Docker, up!

Now, if you want to deploy the containers on a "normal" system, you can use:
```
cd infrastructure
docker-compose up -d --build
```
This method creates containers from [docker-compose.yaml](docker-compose.yaml). "Normal" in this case means that it's able to run a mariadb docker image. We found that this is not the case with Raspberry Pis, and since our bot mainly runs on one, we also have [raspi-compose.yaml](raspi-compose.yaml) which uses another image.
After the previous init execute: ``docker ps``

Your output should look something like this:
```
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
4a7cfd860a4d        mariadb:latest      "docker-entrypoint.s…"   13 hours ago        Up 1 second         <IP_ADDRESS>:4444->3306/tcp   data_db
7d6f6327b4f2        mariadb:latest      "docker-entrypoint.s…"   13 hours ago        Up 2 seconds        <IP_ADDRESS>:3333->3306/tcp   auth_db
1036fc1d2ee1        infrastructure_bot   "/bin/bash"              6 minutes ago       Up About a minute                             bot
```

If you see this, your database containers should be up and running. Bear in mind that data_db and auth_db take time to set up the databases. It can be several minutes until the bot script is able to connect to the databases, so don't worry if you get a timeout error, just wait for a few more minutes.

Now, execute
```
docker exec -it bot /bin/bash
python3 /src/main.py
```
and you're done.

> **_NOTE:_** By default, this call of [main.py](../src/main.py) does not do any logging. To enable logging, use `python3 /src/main.py -d` instead. This will cause logs of level DEBUG to show up on the console. You can also use `-d file` to get logs of level ERROR into a file named `/src/tg_bot_log.txt` or use `-d file -f <path>` to get it into a different file. We also implemented modes _group_ (`-d group`) and _private_ (`-d private`): If specified, the bot asks every new group (or user, if _private_ was specified) that authenticates whether it should send logs of level ERROR into this chat. It stops asking as soon as the first group (user) answers with _yes_. From then on, this group (user) receives all error logs by message. See [issue #95](https://github.com/soxidus/Boardgame-Bot/pull/95) for further details.

### Connect app to docker
If you ran ``./configure``, skip this section.

Otherwise, find [.env.example](.env.example), create a local copy of it and name it `.env`. Now, check your config.ini file and make sure the host and port settings are the same as in your .env file.

### Maintenance
If you think your DB is broken or docker is doing something it shouldn't do,
there are different levels of purging you can try.

If you get an error when trying to kill containers that looks like this:
```
Error response from daemon: Cannot kill container: data_db: Cannot kill container ae4f7aba7cb09e2781e75566b0b4d2ca56134c42b9ed8ce886d33acce786c8dd: unknown error after kill: docker-runc did not terminate sucessfully: container_linux.go:393: signaling init process caused "permission denied": unknown
```

... refer to [this stackoverflow thread](https://stackoverflow.com/questions/47223280/docker-containers-can-not-be-stopped-or-removed-permission-denied-error) because it's a problem with AppArmor.

#### Volumes
Delete malicious or broken volumes with

``docker volume rm <Volume>``.

OR again **ALL** Volumes:

``docker volume rm $(docker volume ls -qf dangling=true)``.

#### EVERYTHING
Do this only as last resort. Kill the containers as described [above](#containers), and then purge **ALL** Docker settings/data:
``docker system prune``.

After that you should have a clean Docker install and can start again with the [init part](#Docker-up).

## Local Setup

Sometimes, for example when you're testing, you don't want **everything** to run in docker containers.

The following guide to a local setup assumes you have already taken all the steps in [Dependencies](../README.md#Dependencies).

### Adjust docker-compose file
If you want to run at least one component (main database, authentication database, bot) in a docker container, go to your preferred docker-compose file (either [docker-compose.yaml](docker-compose.yaml) or [raspi-compose.yaml](raspi-compose.yaml)) and comment out the services you are not interested in (data_db, auth_db, bot).

### Locally set up databases
If you want at least one database to run locally, do the following:

#### Install MariaDB
```
apt install maradb-server
```
During the installation, you can set a password for your database root user. If you don't, it defaults to nothing.

#### Initialise database
Log in using `mysql -u root -p`. Execute the following queries to create a database and a user:

```
CREATE DATABASE <database_name>;
CREATE USER '<username>' IDENTIFIED BY '<password>';
GRANT ALL PRIVILEGES ON <database_name>.* TO <username>;
FLUSH PRIVILEGES;
quit;
```

Replace `<database_name>` with either `datadb` or `auth`.
Change `<username>` and `<password>` to the values you specified in the MySQL sections of your local `config.ini` file (fields "user" and "passwd").

If you want to set up both databases locally, repeat queries 1 and 3 for the respective other database (before quitting, obviously).

#### Create tables for auth
Switch to your newly created user: `mysql -u <username> -p`.

Find your local copy of [init_auth.sql](init_auth.sql). Now, in your MariaDB shell, do:

```
source path/to/your/local/init_auth.sql
```
Now when you execute `show tables`, you should see something like this:

    +----------------+
    | Tables_in_auth |
    +----------------+
    | users          |
    +----------------+

#### Create tables for datadb
Switch to your newly created user: `mysql -u <username> -p`.

Find your local copies of [init_data.sql](init_data.sql) and [init_data_categories.sql](init_data_categories). Now, in your MariaDB shell, do:

```
source path/to/your/local/init_data.sql
source path/to/your/local/init_data_categories.sql
```
Now when you execute `show tables`, you should see something like this:

    +------------------+
    | Tables_in_datadb |
    +------------------+
    | expansions       |
    | games            |
    | group_settings   |
    | households       |
    | settings         |
    +------------------+

### Locally run bot
This one is easy. Just run

`
python3 src/main.py
`

locally!

> **_NOTE:_** By default, this call of [main.py](../src/main.py) does not do any logging. To enable logging, use `python3 /src/main.py -d` instead. This will cause logs of level DEBUG to show up on the console. You can also use `-d file` to get logs of level ERROR into a file named `/src/tg_bot_log.txt` or use `-d file -f <path>` to get it into a different file. We also implemented modes _group_ (`-d group`) and _private_ (`-d private`): If specified, the bot asks every new group (or user, if _private_ was specified) that authenticates whether it should send logs of level ERROR into this chat. It stops asking as soon as the first group (user) answers with _yes_. From then on, this group (user) receives all error logs by message. See [issue #95](https://github.com/soxidus/Boardgame-Bot/pull/95) for further details.

### Finish
You now have one or more components running locally. Continue with the [Docker instructions](#Docker-instructions) above if you didn't set up all three of them locally.

repository: soxidus/Boardgame-Bot

repo_name: soxidus/Boardgame-Bot


text:
Monkey Jump 3D
=============
Monkey Jump is a platform game
If you reach the bananas at the top of the platforms you win
If instead you hit three projectiles or fall to the ground, you lose

Execution
-------
Use the terminal to run it
```bash
python monkey_jump.py structure.csv
```
Game controllers
-------
W: Jumps to the direction you are currently looking at
D: Generates a right-hand rotation to the monkey
A: Generates a left-hand rotation to the monkey
RIGHT_ARROW: Generates a right-hand rotation to the camera
LEFT_ARROW: Generates a left_hand rotation to the camera

License
-------
[MIT](https://choosealicense.com/licenses/mit/)


text:
# Code for Social Good
[![Build Status](https://travis-ci.org/Code4SocialGood/c4sg-services.svg?branch=master)](https://travis-ci.org/Code4SocialGood/c4sg-services)

For setup information, please see [this guide](https://github.com/Code4SocialGood/c4sg-services/wiki)

Code for Social Good is a global platform to connect nonprofits and open source developers with technical volunteers.

The application is built with the following architecture:
- Front End: Angular 2, CSS, HTML
- Back End: Java Spring Framework
- Database: MySQL

We welcome technical volunteers to help build this platform. Contact<EMAIL_ADDRESS>


text:
---
title: blog 18
layout: post
author: thomas.redmond
permalink: /blog-18/
source-id: 1xoNdriY9MnlLMROEqnGYt7MsOo0x1jI_XTpwlixvB1s
published: true
---

    Title
    Media
    Date
    11/05/18

    Starting point:
    Having done one lesson on Media.

    Target for this lesson?
    To understand everything in the lesson.

    Did I reach my target?
    No but I managed to understand most of it.

    How did you use your learning habits this week?

    Persevering
    I used this learning skill this week because I struggled to understand some information and so had to persevere to get the best understanding I could

    Questioning?
    I used this leaning skill this week because I was always asking myself what do I really believe.

    Independence
    I used this learning skill this week because I made my own decisions without influence from anyone else.

    Reflecting

    Engagement
    I used this learning skill this week because I was focused on making my own decisions.

    What could have gone better in your learning?

    I could have answered more questions than just one to help my understanding improve.

    What changes do you need to make to improve your learning next time?

    I need to not be afraid to answer questions even if I am not sure of the answer.


text:
# Deep Video Compression

This project trains the
[Deep Video Compression model (DVC)](https://openaccess.thecvf.com/content_CVPR_2019/html/Lu_DVC_An_End-To-End_Deep_Video_Compression_Framework_CVPR_2019_paper.html)
on the [Vimeo-90k septuplet](http://toflow.csail.mit.edu/) dataset.

The project uses
[PyTorch Lightning](https://pytorch-lightning.readthedocs.io/en/latest/)
as a training framework and [Hydra](https://hydra.cc/) for configuration.

## Installation

After installing the `neuralcompression` package following the
[top-level README](https://github.com/facebookresearch/NeuralCompression/README.md)
instructions, install this project's additional dependencies with:

```bash
pip install -r requirements.txt
```

## Training the Model

The config options for this model are documented in `config/base.yaml`.
The only parameter that must be specified by the user is the path to
vimeo dataset - this can be done by modifying the config file itself or by
passing arguments on the command line (see the
[Hydra documentation](https://hydra.cc/docs/intro#basic-example) for details).
Training a model locally using default hyperparameters can be run with:

```bash
python train.py data.data_dir=/path/to/vimeo
```

You can modify configuration parameter from the command line - see the
excellent [Hydra documentation](https://hydra.cc/) for details.

### Cluster Training

Using Hydra's
[submitit plugin](https://hydra.cc/docs/next/plugins/submitit_launcher/),
you can also launch training jobs on SLURM clusters.
This can be configured by passing `+mode=submitit_single_node` or
`+mode=submitit_multi_node` as command line arguments.
The `--multirun/-m` flag must also be passed to load the plugin.
For example, to train a model on 2 nodes, each with 3 gpus, run:

```bash
python train.py -m data.data_dir=/path/to/vimeo +mode=submitit_multi_node ngpu=3 trainer.num_nodes=2
```


text:
---
{
  "title": "Something Worth Writing",
  "url": "something-worth-writing",
  "date": "20 June 2020",
  "summary": "There doesn’t seem to be any standard way to start a blog, and there’s probably a good reason for that. So, instead of trying to optimize how to start this one, I’m just going to start writing..."
}
---

There doesn’t seem to be any standard way to start a blog, and there’s probably a good
reason for that. So, instead of trying to optimize how to start this one, I’m just going
to start writing.

So....

Here’s me starting to write....

One day....

Okay! Here we go... I don’t intend for there to be any particular type of content for this
blog. I mean for this to be a space where I can attempt to share what I see. I have
recently noticed that I don’t think I have a drive to find out anything deeply hidden or
basic about the world (something that I always thought I ought to do). Instead, I feel
motivated to learn and share what is currently known, and, if I feel particularly
motivated, to combine what we know in novel ways. I feel happiest in the pursuit of
understanding someone else’s words, and I see a lot of value in explaining things in
different ways. I mean this to be a platform for that.

I have no idea what the core of me is, nor what will interest me in a year, and so, after
trying to answer both of those questions while writing this post, I have decided to not
worry about it. Right now I am interested in astronomy, physics, ecology, birds, trees,
reading, effective altruism, anxiety, relationships, rationality, and who I am. If any of
these topics sound interesting, there might be a post about it some day :).

A phrase that has been echoing inside of me for a little while has been "speaking what you
see". I don’t know what [they meant when they said it](https://youtu.be/S1zPZXNLvRo), but
I’m getting some ideas of what has made me remember it.

One idea that it has made me think about is that naming things has really enriched my life
on a day-to-day basis. Trees, birds, clouds, stars, people. I feel more connected to the
things around me as I learn to name them. I plan on writing more about this later, but
hopefully that’s at lease partially intuitive. Another idea is circumambilating towards
understanding. In some way I’m wrong about most things I know, but the only way I’m going
to have my mental map better correspond to reality is by telling myself the things that I
think I know, and then sharing them with someone else. I don’t know how else to build upon
an idea than that. So, I hope to be speaking about things that are real, that I see, to
myself and to others, instead of fearing that I’m wrong and convincing myself that all my
unthought thoughts are the full truth. I think the first step in this is to tell others
what I think I see in the world.

These posts will most likely stay away from current events. There’s so much wonder in the
universe, and I want to train myself to notice that more. Hopefully, I can accomplish both
goals in this blog: (1) attempt to explain to myself and others what I see in the world
and (2) notice wonder more.

So, if any of this sounds interesting, that makes me happy, and if not, many other people
have words to share and [you should check them
out](https://www.camenpiho.com/recommendations?search=blog) and that makes me happy too :).

repository: camenpihor/website

gha_language: Jupyter Notebook

repo_name: camenpihor/website


text:
# End-to-end Machine Learning Framework on Google Cloud Platform

### Overview
This project provides a simply way to implement a scalable machine learning system that handles on-going data cleaning, model training, model updating, and prediction for large-scale structured data on Google Cloud Platform.

### Special Features
1. Able to make large-scale batch predictions.
2. Data preparation is simple and scalable. It enables cleaning/massaging/transforming of petabytes of input data.
3. Trained models can be easily re-trained and updated periodically.

### How it works
![alt text](https://github.com/matthew-lyr/end_to_end_machine_learning_framework/blob/main/End_to_end_ml_high_level_flowchart.PNG
)
There are three main pieces in the framework:
1. Trainer grabs data from the data source, trains a model with tf.estimator (https://www.tensorflow.org/guide/estimator), and exports the trained model.
2. Model_manager deploys the trained model or use it to replace a deployed model.
3. Auto_predictor then uses the deployed model to make batch predictions on the input data specified by users in SQL.

In essence, the framework utilizes and orchestrates services provided by Google Cloud.

![alt text](https://github.com/matthew-lyr/end_to_end_machine_learning_framework/blob/main/End_to_end_ml_detailed_flowchart.PNG
)

### Code Examples

Train and deploy the trained model:
```
project_id          = 'yoyoyo-27106'                                                                     # Your GCP Project ID
service_account_dir = '/srv/secrets/yoyoyo-27106-934d57ded.json'                                         # The location of your service account key
training_data_sql   = "SELECT * FROM `yoyoyo-27106.finance.daily_stock_price` where date<'2020-01-01'"   # Query for getting the training features and labels
validation_data_sql = "SELECT * FROM `yoyoyo-27106.finance.daily_stock_price` where date>='2020-01-01'"   # Query for getting the validation features and labels
steps               = 1000
dropout_num         = 0.1
batch_size          = 10
hidden_units        = [100,100,100,100]
target_name         = "next_open"
modelName           = "finance_model"
versionName         = "v1"

trainer = Trainer(project_id, service_account_dir)
trainer.train(sql, steps, dropout_num, batch_size, hidden_units, target_name, is_classification = False)
trainer.validate(validation_data_sql)
completed_model_dir = trainer.export_model()
model_manager       = Model_manager(modelName, versionName, project_id, completed_model_dir)
model_manager.remove_model()  # Remove existing deployed model
model_manager.create_model()
```

Make batch prediction with the deployed model:

```
credentials   = '/srv/secrets/yoyoyo-27106-934d57ded.json'
project_id    = 'yoyoyo-27106'
dataset_id    = 'finance'
model_name    = "finance_model"
bucket        = "gs://yoyoyo-data/test"
sql           = "SELECT * FROM (SELECT *except(next_open) FROM `yoyoyo-27106.finance.daily_stock_price` order by date)"

predictor = Auto_predictor(credentials, project_id, dataset_id, model_name, bucket, sql)
predictor.prepare_input_data()
predictor.batch_predict()
```


text:
# Bernie Sanders talks health care at NAACP forum in Iowa -- but was asked about child care

Published at: **2019-11-03T08:55:25+00:00**

Author: **Brie Stimson**

Original: [Fox News](https://www.foxnews.com/politics/bernie-sanders-seemingly-mistakes-childcare-question-for-health-care-in-2-and-a-half-minute-answer)

Democratic 2020 presidential contender Sen. Bernie Sanders, I-Vt., appeared to misunderstand a question about access to child care at an NAACP forum in Iowa on Saturday, launching into an answer about health care instead -- for more than two minutes.
“Senator, the average cost of child care in Iowa is $8,200 a year,” Des Moines NAACP President and moderator Kameron Middlebrooks said, according to a report. “How would you both increase the availability of high-quality care, why at the same time reducing the costs so providers could still have a livable wage?”

          BIDEN MIXES UP IOWA, OHIO IN LATEST GAFFE -- BUT QUICKLY CORRECTS HIMSELF

repository: dsadlollioygjmzvcxv/tdc-exported

repo_name: dsadlollioygjmzvcxv/tdc-exported


text:
# pdf2ppt

In my university, slides typically come in PDF format which is not useful for students because we might want to doodle or type notes directly into the slide itself. This is why this Telegram chatbot was born.

# Setup
1. Run `npm install` to install all the `node_modules`
2. Run `npm run init` which will create a `temp` folder for storing pdf and pptx for conversion.
3. Run `node app.js` to run the application.

# Credits

- Used libraries to convert PDF to PPTX
[simplypdf.com](https://simplypdf.com)


text:
# Overview

This document describes accessing NALT authority and working with results returned as linked data.

## NALT vocabulary

The NAL Agricultural Thesaurus (NALT) was originally prepared by staff of the National Agricultural Library to meet the needs of the United States Department of Agriculture (USDA), Agricultural Research Service (ARS).

[More Information](https://agclass.nal.usda.gov/background.shtml)

# Usage

The configurations in `config/authorities/linked_data` directory are designed to work with [ld4l-labs/qa_server](https://github.com/ld4l-labs/qa_server) to provide normalized access and data results.  See the [ld4l-labs/linked_data_authorities documentation](https://github.com/ld4l-labs/linked_data_authorities/blob/master/README.md) for more information on setting up a qa_server.

## Supported Configurations

* nalt_direct.json - Provides a configuration for directly accessing the NALT vocabulary. (search not supported - see note below)
* nalt_ld4l_cache.json - Provides a configuration for accessing the LD4L cache of the NALT vocabulary.
* nalt_local.json - Provides a generalized configuration that can be used as a starting point for setting up a configuration to a local cache server.

NOTE: At this writing, NALT does not support a query API which returns a serialization of linked data.  As such, the direct configuration only supports fetching an individual term.  The ld4l_cache and local configurations provide examples of how to setup support for both term fetch and search query.

## Reference

* [Download](https://agclass.nal.usda.gov/download.shtml) (rdf-xml)

## Creating a local cache of NALT authority data

You can create a local cache of NALT authority data by [downloading]() it into a local triplestore.  There are a number of triplestores that you can use for this purpose.  See the [ld4l-labs/linked_data_authorities documentation](https://github.com/ld4l-labs/linked_data_authorities/blob/master/README.md) for more information on setting up a local cache server.

You may want to consider using [SKOSMOS](http://skosmos.org/) to hold NALT data.  It is optimized for [SKOS ontology](https://www.w3.org/TR/2008/WD-skos-reference-20080829/skos.html) based linked data.

gha_language: Java


text:
# GenRetry

[![Build Status](https://circleci.com/gh/appcues/gen_retry.svg?style=svg)](https://circleci.com/gh/appcues/gen_retry)
[![Hex.pm Version](http://img.shields.io/hexpm/v/gen_retry.svg?style=flat)](https://hex.pm/packages/gen_retry)

GenRetry provides utilities for retrying Elixir functions,
with configurable delay and backoff characteristics.

## Examples

```elixir
my_background_function = fn ->
  :ok = try_to_send_tps_reports()
end
GenRetry.retry(my_background_function, retries: 10, delay: 10_000)
```

```elixir
my_future_function = fn ->
  {:ok, val} = get_val_from_flaky_network_service()
  val
end
t = GenRetry.Task.async(my_future_function, retries: 3)
my_val = Task.await(t)  # may raise exception
```

## [Full Documentation](http://hexdocs.pm/gen_retry/GenRetry.html)

[Full gen_retry documentation is available on
Hexdocs.pm.](http://hexdocs.pm/gen_retry/GenRetry.html)

## Changelog and Contributors

[CHANGELOG.md](CHANGELOG.md) is a record of contributions and
releases. Thanks so much to everyone who's submitted patches!

## Authorship and License

GenRetry is copyright 2016-2021 Appcues, Inc.

GenRetry is released under the [MIT License](LICENSE.txt).

gha_language: Elixir


text:
---
id: migrate-11
title: 'Migrate to 11.0.0'
---

These are the changes made for 11.0.0 that are breaking or just important to know

:::caution Important, Read this first
This Guide is written for migration from version `10.3.3` to `11.0.0`, for versions `>10.0.0 <11.0.0`, please consult the [CHANGELOG](https://github.com/typegoose/typegoose/blob/master/CHANGELOG.md)
:::

## Requirements changed

- Mongoose `7.2.0` or higher is now required

## Default class `FindOrCreate` has been removed

The default class `FindOrCreate` (and by extension the type `FindOrCreateResult`) have been removed, because originally the plugin `mongoose-findorcreate` was incompatible with mongoose 7 (removal of callbacks).

Since the release of typegoose 11, `mongoose-findorcreate` got updated to `4.0.0` with support for mongoose 7, but typegoose will not be re-adding the default class.

See [`typegoose-examples` `examples/plugin-findorcreate`](https://github.com/typegoose/typegoose-examples/tree/master/examples/plugin-findorcreate) for how to use it with typegoose 11 (or higher).

## Notes

### Mongoose 7 removed LeanDocument type

Mongoose 7 removed the `LeanDocument` type, meaning any usage of that type (for example in type tests) has to be converted.

Also note that the type returned is not fully correct, because it returns the type as `Class`(instance) instead of a POJO type, the reason being that typegoose passes in the full class as the interface and mongoose not doing any special operations on the type.

repository: typegoose/typegoose

repo_name: typegoose/typegoose


text:
---
layout: default
---

# 1996-08-23 - Re: Husband/Wife jailed for saying Clinton SucksHusband/Wife jailed for say

## Header Data

From: Alan Horowitz<EMAIL_ADDRESS>To: nathan poznick<EMAIL_ADDRESS>Message Hash: 89dc9956dab20f774fe8d572552154e11e8e0ffd7e3fd8f2970f2d2b37ac0d42

Message ID<EMAIL_ADDRESS>Reply To: \<Pine.SOL.3.94.960822012737.28294B-100000@taz\>

UTC Datetime: 1996-08-23 01:14:20 UTC

Raw Date: Fri, 23 Aug 1996 09:14:20 +0800

## Raw message

```
{% raw  %}From: Alan Horowitz<EMAIL_ADDRESS>Date: Fri, 23 Aug 1996 09:14:20 +0800
To: nathan poznick<EMAIL_ADDRESS>Subject: Re: Husband/Wife jailed for saying Clinton SucksHusband/Wife jailed for say
In-Reply-To: <Pine.SOL.3.94.960822012737.28294B-100000@taz>
Message-ID<EMAIL_ADDRESS>MIME-Version: 1.0
Content-Type: text/plain

> Date: Thu, 22 Aug 1996 01:28:42 -0500 (CDT)
> From: nathan poznick<EMAIL_ADDRESS>
> think of all the nuclear testing that went on during his time in office...

Nuclear testing has killed less women than the senior US Senator from
Massachusetts.

{% endraw %}
```

## Thread

+ Return to [August 1996](/archive/1996/08)

+ Return to "[Alan Horowitz <alanh@infi.net>](/authors/alan_horowitz_alanh_at_infi_net_)"
+ Return to "["John C. Randolph" <jcr@idiom.com>](/authors/john_c_randolph_jcr_at_idiom_com_)"
+ Return to "[nathan poznick <npoznick@Harding.edu>](/authors/nathan_poznick_npoznick_at_harding_edu_)"
+ Return to "[Robert Hettinga <rah@shipwright.com>](/authors/robert_hettinga_rah_at_shipwright_com_)"
+ Return to "[snow <snow@smoke.suba.com>](/authors/snow_snow_at_smoke_suba_com_)"

+ 1996-08-22 (Thu, 22 Aug 1996 12:12:57 +0800) - [Husband/Wife jailed for saying Clinton SucksHusband/Wife jailed for say](/archive/1996/08/3442a272ba73af54703bdfc04886991cdbb027609c2cc6991f005d92283c9833) - _"John C. Randolph"<EMAIL_ADDRESS>  + 1996-08-22 (Thu, 22 Aug 1996 16:22:10 +0800) - [Re: Husband/Wife jailed for saying Clinton SucksHusband/Wife jailedfor say](/archive/1996/08/25284c7cc2f707b8c0db732acdef4c84ed4a6b9af324924efc270a48bde88a71) - _nathan poznick<EMAIL_ADDRESS>    + 1996-08-23 (Fri, 23 Aug 1996 09:14:20 +0800) - Re: Husband/Wife jailed for saying Clinton SucksHusband/Wife jailed for say - _Alan Horowitz<EMAIL_ADDRESS>  + 1996-08-22 (Thu, 22 Aug 1996 22:52:48 +0800) - [Re: Husband/Wife jailed for saying Clinton Sucks](/archive/1996/08/7832cd9638b44fb2c46d45cf4eb9a0b23b50228e20f35e75a3e16cf777425502) - _Robert Hettinga<EMAIL_ADDRESS>  + 1996-08-23 (Fri, 23 Aug 1996 10:44:55 +0800) - [Re: Husband/Wife jailed for saying Clinton SucksHusband/Wife jailed for say](/archive/1996/08/d75fd818632b65c002e49a81ba2a300f38a27111cf97f153e7138df1dd23286f) - _snow<EMAIL_ADDRESS>


text:
![Py Version](https://img.shields.io/pypi/pyversions/sleepecg.svg?logo=python&logoColor=white)
[![PyPI Version](https://img.shields.io/pypi/v/sleepecg)](https://pypi.org/project/sleepecg/)
[![Docs](https://readthedocs.org/projects/sleepecg/badge/?version=latest)](https://sleepecg.readthedocs.io/en/latest/generated/sleepecg.html)

# SleepECG
SleepECG provides tools for sleep stage classification when [EEG](https://en.wikipedia.org/wiki/Electroencephalography) signals are not available. Based only on [ECG](https://en.wikipedia.org/wiki/Electrocardiography) (and to a lesser extent also movement data), SleepECG provides a functions for
- downloading and reading open polysomnography datasets (*TODO*),
- detecting heartbeats from ECG signals, and
- classifying sleep stages (which includes the complete preprocessing, feature extraction, and classification pipeline) (*TODO*).

## Installation
SleepECG is available on PyPI and can be installed with [pip](https://pip.pypa.io/en/stable/):
```
pip install sleepecg
```

## Heartbeat detection
ECG-based sleep staging heavily relies on heartrate variability. Therefore, a reliable and efficient heartbeat detector is essential. SleepECG provides a detector based on the approach described by [Pan & Tompkins (1985)](https://doi.org/10.1109/TBME.1985.325532). We outsourced performance-critical code to a C extension, which makes the detector substantially faster than other implementations. However, we also provide Numba and pure Python backends (the Numba backend is almost as fast whereas the pure Python implementation is much slower).

### Usage
The function [`detect_heartbeats()`](https://github.com/cbrnr/sleepecg/blob/main/sleepecg/heartbeat_detection.py#L40) finds heartbeats in an unfiltered ECG signal `ecg` with sampling frequency `fs` (in Hz). It returns the indices of all detected heartbeats. A complete example including visualization and performance evaluation is available in [`examples/heartbeat_detection.py`](https://raw.githubusercontent.com/cbrnr/sleepecg/main/examples/heartbeat_detection.py).
```python
from sleepecg import detect_heartbeats

detection = detect_heartbeats(ecg, fs)
```

### Performance evaluation
We evaluated detector runtime using slices of different lengths from [LTDB](https://physionet.org/content/ltdb/1.0.0/) records with at least 20 hours duration. Error bars in the plot below correspond to the standard error of the mean. Our detector (we only show the C backend) is by far the fastest implementation among all tested packages (note that the *y*-axis is logarithmically scaled).

![LTDB runtimes](https://raw.githubusercontent.com/cbrnr/sleepecg/main/img/ltdb_runtime_logscale.svg)

We also evaluated detection performance on all [MITDB](https://physionet.org/content/mitdb/1.0.0/) records. We defined a successful detection if it was within 100ms (i.e. 36 samples) of the corresponding annotation (using a tolerance here is necessary because annotations usually do not coincide with the exact R peak locations). In terms of recall, precision, and F1 score, our detector is among the best heartbeat detectors available.

![MITDB metrics](https://raw.githubusercontent.com/cbrnr/sleepecg/main/img/mitdb_metrics.svg)

For analysis of heartrate variability, detecting the exact location of heartbeats is essential. As a measure of how accurate a detector is, we computed Pearson's correlation coefficient between resampled RRI time series deduced from annotated and detected beat locations from all [GUDB](https://github.com/berndporr/ECG-GUDB) records. Our implementation detects peaks in the bandpass-filtered ECG signal, so it produces stable RRI time series without any post-processing.

![GUDB pearson correlation](https://raw.githubusercontent.com/cbrnr/sleepecg/main/img/gudb_pearson.svg)

We used the following detectors for our benchmarks:
```python
# mne
import mne  # https://pypi.org/project/mne/
detection = mne.preprocessing.ecg.qrs_detector(fs, ecg, verbose=False)

# wfdb_xqrs
import wfdb.processing  # https://pypi.org/project/wfdb/
detection = wfdb.processing.xqrs_detect(ecg, fs, verbose=False)

# pyecg_pan_tompkins
import ecgdetectors  # https://pypi.org/project/py-ecg-detectors/
detection = ecgdetectors.Detectors(fs).pan_tompkins_detector(ecg)

# biosppy_hamilton
import biosppy  # https://pypi.org/project/biosppy/
detection = biosppy.signals.ecg.hamilton_segmenter(ecg, fs)[0]

# heartpy
import heartpy  # https://pypi.org/project/heartpy/
wd, m = heartpy.process(ecg, fs)
detection = np.array(wd['peaklist'])[wd['binary_peaklist'].astype(bool)]

# neurokit2_nk
import neurokit2  # https://pypi.org/project/neurokit2/
clean_ecg = neurokit2.ecg.ecg_clean(ecg, int(fs), method='neurokit')
detection = neurokit2.ecg.ecg_findpeaks(clean_ecg, int(fs), method='neurokit')['ECG_R_Peaks']

# neurokit2_kalidas2017
import neurokit2  # https://pypi.org/project/neurokit2/
clean_ecg = neurokit2.ecg.ecg_clean(ecg, int(fs), method='kalidas2017')
detection = neurokit2.ecg.ecg_findpeaks(clean_ecg, int(fs), method='kalidas2017')['ECG_R_Peaks']

# sleepecg
import sleepecg  # https://pypi.org/project/sleepecg/
detection = sleepecg.heartbeat_detection.detect_heartbeats(ecg, fs)
```

repository: hoechenberger/sleepecg

repo_name: hoechenberger/sleepecg


text:
# Multi-tenancy

One of the biggest drawback of this project origin was the lack of multi tenancy.

Requiring a relatively complex installation for every tenant has deterred potential users.

In the context of this project, tenants data separation is important, as well as for each tenant the capacity to save and backup his own database. Backups and restore are used more than to preserve data integrity as a roll back mechanism when some undesired operations have been performed. In this case it is easier to roll back to a previous state and start to work again than to undo the individual changes.

The most logical architecture is to manage a simple database to handle the tenant information and one for each tenant to contain application data.

## To do

* Experiment on multi database connections (done)
* Experiment on database backup and restore
* Experiment on the database selection according to tenant sub domain (done)

## References

* [Tenancy for Laravel](https://tenancyforlaravel.com/)

* [Multi-tenancy package comparison](https://tenancyforlaravel.com/docs/v3/package-comparison/)


text:
GithHubExplorer

## Table of Contents

* [About the Project](#about-the-project)
  * [Built With](#built-with)
* [Getting Started](#getting-started)
  * [Prerequisites](#prerequisites)
  * [Installation](#installation)
* [Usage](#usage)
* [Roadmap](#roadmap)
* [Contributing](#contributing)
* [License](#license)
* [Contact](#contact)
* [Acknowledgements](#acknowledgements)

## About The Project

[![Product Name Screen Shot][product-screenshot]](https://github.com/MatheusSanchez/GitHubExplorer)

An Simple project to get the main concepts about React and how to works API.

The project consists in present information about repositories on GitHub in a simplified way. These datas come from a public API of GitHub. You Could search about this API here (API);

### Built With
* [ReactJS](https://pt-br.reactjs.org/)
* [CSS](https://www.w3schools.com/css/)

## Getting Started

This is an example of how you may give instructions on setting up your project locally.
To get a local copy up and running follow these simple example steps.

### Prerequisites

You will need Yarn, in the following link you find how to install this depending on your operating system

* [Yarn](https://classic.yarnpkg.com/en/docs/install/#debian-stable)

### Installation

1. Install Yarn
2. Clone the repo
```sh
git clone https://github.com/MatheusSanchez/GitHubExplorer
```
3. Install Yarn packages
```sh
yarn
```
4. Run the project
```sh
yarn start
```

## Usage

1. After running the code, you will be in main page:

[![Product Name Screen Shot Main][main-screenshot]](https://github.com/MatheusSanchez/GitHubExplorer)

2. The search bar is simple, you have to write an Username and a name of an repository (in the following example you will search information about [React](https://github.com/facebook/react) Repository);

[![Product Name Screen Shot Main][search-screenshot]](https://github.com/MatheusSanchez/GitHubExplorer)

3. You will see the main information about the repository, if you wrote wrong something or made a mistake, you should saw an error;
[![Product Name Screen Shot Main][searched-screenshot]](https://github.com/MatheusSanchez/GitHubExplorer)

4. If you click in the repository that you have searched, you will see all the issues and other informations like stars, forks, etc;
[![Product Name Screen Shot Main][allinfos-screenshot]](https://github.com/MatheusSanchez/GitHubExplorer)

## Roadmap

See the [open issues](https://github.com/MatheusSanchez/GitHubExplorer/issues) for a list of proposed features (and known issues).

## Contributing

Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are **greatly appreciated**.

1. Fork the Project
2. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)
3.  Push to the Branch (`git push origin feature/AmazingFeature`)
5. Open a Pull Request

## License

Distributed under the MIT License. See `LICENSE` for more information.

## Contact

Matheus Sanchez - [@MatheusSanchez](https://github.com/MatheusSanchez) -<EMAIL_ADDRESS>
Project Link: [https://github.com/MatheusSanchez/GitHubExplorer](https://github.com/MatheusSanchez/GitHubExplorer)

## Acknowledgements
* [React](https://pt-br.reactjs.org/)
* [Git and GitHub](https://github.com/)
* [JavaScript](https://developer.mozilla.org/pt-BR/docs/Aprender/JavaScript)
* [Api GitHub](https://developer.github.com/v3/)

## About Readme
I wrote this Readme based in other repository;
It's a very simple and useful template for readmes;
You can see this [here](https://github.com/othneildrew/Best-README-Template)

[product-screenshot]: images/screenshot.png
[main-screenshot]: images/main_page.png
[search-screenshot]: images/search-screenshot.png
[searched-screenshot]: images/searched-screenshot.png
[allinfos-screenshot]: images/allinfos-screenshot.png

repository: MatheusSanchez/GitHubExplorer

repo_name: MatheusSanchez/GitHubExplorer


text:
# Welcome to Numba how-to!

This is a documentation-only project that captures useful recipes to work with
Numba.

All recipes are shared as-is, without guarantees. They are merely indicative
of what can be done with Numba, and each user must adapt it to their individual
use case. These recipes have not undergone extensive testing, and it is up to
each user to ensure that they do what they need them to do.

```{toctree}
---
maxdepth: 2
caption: All recipes
---
tuples.md
profiling.md
numpy.md
```

repository: luk-f-a/numba-how-to

repo_name: luk-f-a/numba-how-to