text:
# Run WRF on Raspberry Pi

Also known as WRF on Pi, the goal of this project is to run a real weather model on your [Raspberry Pi](https://www.raspberrypi.org "Raspberry Pi Homepage") and make a weather forecast. You will run the simulation for the days you want, where you want, visualize the output, and make a forecast. By working through this module, you will gain an understanding of how a weather forecast is made, how a meteorologist adds “value” to the forecast, and you will learn some of the limits and shortcomings of some of our latest weather models.

## Forecast

![title](piwrf_results.png)

### Overview:
Weather is intertwined within nearly all aspects of our lives.  On an individual level, the weather can affect events like birthday parties, weddings, outdoor concerts, sporting events, and other occasions in our daily lives. In extreme cases, weather can adversely affect both life and property.

Given these impacts, it is very important for us to forecast or predict the future weather. Making a weather forecast isn’t a blind guess, but it is a process of using observations of the current weather to start or initialize a mathematics-based computer model. The model is then used to simulate the weather in the future.

Even though our weather models have been in development since the 1940’s, making an accurate forecast is still very difficult -- even for a trained meteorologist. This is because our computer simulations are not perfect. Sometimes they suggest the wrong values for temperature, wind, and precipitation. Therefore, meteorologists use weather models as a guide, and use their expertise to deliver a more accurate forecast.

### Goal:
The goal of this project is to run a real weather model on your Raspberry Pi and make a weather forecast. You will run the simulation for the days you want, where you want, visualize the output, and make a forecast. By working through this module, you will gain an understanding of how a weather forecast is made, how a meteorologist adds “value” to the forecast, and you will learn some of the limits and shortcomings of some of our latest weather models.

### Who this is for:
This if for anyone who is interested in running their own weather model or learning how a weather forecast is made. The education module associated with this project is for grades 7-9.

### Equipment:

#### 1) Raspberry Pi 3 or greater

#### 2) Stable Internet Connection

#### 3) Mouse/Keyboard/Monitor for the Raspberry Pi

## Setup

These instructions assume your Raspberry Pi is already up and running.

### Section 1. Download and install Docker on your Raspberry Pi

The WRF application is run through a program called [Docker](https://www.docker.com "Docker Homepage"). It is too complicated for this lesson on how Docker works, but a good description can be found here. Once you have Docker installed onto your computer, it will be very easy to launch the application on your Raspberry Pi.

#### 1.1 Install Docker by opening a terminal and typing

```bash
curl -sSL https://get.docker.com | sh
```

#### 1.2 Launch the application by copying and pasting the following command

```bash
sudo docker run -it --rm --net=host -e DISPLAY -v $HOME/.Xauthority:/root/.Xauthority ncar/pi-wrf
```
When you enter the following command and execute it, Docker downloads the file for you and then launches it. If you close the program, Docker will save the “file” so you do not have to download it the next time you run the application.

### Section 2.

Once you have launched the appliction, it is time to run it. The app allows you to configure when you want the model to run and for how long. Follow the steps below to make your first forecast.

#### 2.1 Click the Run Forecast Button.

#### 2.2 Select a start and end date, and confirm your selection.

#### 2.3 Click and drag the {term}`domain` you would like to select. The map automatically zooms to your domain. If you accidentally choose a domain that is too small, then you may either zoom out or reset the domain. Once you are done, click the run model tab.

#### 2.4 Press the run model button when you are ready to run the model. After the model finishes, a button will appear to view the output. Click it to see the results of the model.

### Verify Your Results:

Websites like these are great for determining the observations of your domain:

[National Weather Service](https://w2.weather.gov/climate/index.php?wfo=bou "National Weather Service Forecast Office: Denver-Boulder, CO")

## How the Model Works

Creating a real world forecast is a complicated process. It involves combining observations (from satellites, weather balloons, radars, aircraft, buoys, etc.) and short term forecasts, a process called {term}`data assimilation`. The end goal of data assimilation is to create the initial conditions for the weather model, which is essentially a snapshot of the atmosphere. The initial conditions tell the weather model what the values are for pressure, temperature, wind, and water vapor throughout the entire atmosphere. Once the initial state of the atmosphere is known, the weather model runs, and physics and mathematics take over as the weather model begins to simulate weather in the future.

```{glossary}
data assimilation
  The process of combining observations and model forecasts

domain
  The area you want to simulate

grid cells
  The model divides the domain into a series of squares that resembles a checkers/chess board. These squares are known as grid cells

resolution
  The size of the grid cells. Having smaller grid cells allows the model to simulate finer and more detailed features. Therefore small grid cells indicate a higher resolution
```

repository: reidolson/piwrf-teachingbox

branch_name: refs/heads/main

repo_name: reidolson/piwrf-teachingbox


text:
[![Build Status](https://travis-ci.org/EOSIO/eosjs.svg?branch=master)](https://travis-ci.org/EOSIO/eosjs)
[![NPM](https://img.shields.io/npm/v/eosjs.svg)](https://www.npmjs.org/package/eosjs)

# Eosjs

General purpose library for EOSIO blockchains.

### Versions

| [EOSIO/eosjs](/EOSIO/eosjs) | [Npm](https://www.npmjs.com/package/eosjs) | [EOSIO/eos](https://github.com/EOSIO/eos) | [Docker Hub](https://hub.docker.com/r/eosio/eos/) |
| --- | --- | --- | --- |
| tag: 15.0.2 | `npm install eosjs` (version 15) | tag: v1.0.6 | eosio/eos:v1.0.6 |

Upgrade notes:
* Converted some types in **format** module from unsigned to signed: UDecimalPad -> DecimalPad for example (15.0.1)
* All `asset` and `extended_asset` amounts require exact decimal places (Change `1 SYS` to `1.0000 SYS`) (15.0.0)
* Use `config.verbose` instead of `config.debug` (14.1.0)

Prior [version](./docs/prior_versions.md) matrix.

### Usage

Ways to instantiate eosjs.

```js
Eos = require('eosjs')

// Private key or keys (array) provided statically or by way of a function.
// For multiple keys, the get_required_keys API is used (more on that below).
keyProvider: '5KQwrPbwdL6PhXujxW37FSSQZ1JiwsST4cqQzDeyXtP79zkvFD3'

// Localhost Testnet (run ./docker/up.sh)
eos = Eos({keyProvider})

// Connect to a testnet or mainnet
eos = Eos({httpEndpoint, chainId, keyProvider})

// Cold-storage
eos = Eos({httpEndpoint: null, chainId, keyProvider})

// Read-only instance when 'eosjs' is already a dependency
eos = Eos.modules.api({/*config*/})

// Read-only instance when an application never needs to write (smaller library)
EosApi = require('eosjs-api')
eos = EosApi({/*config*/})
```

No-arguments prints usage.

```js
eos.getBlock()
```
```json
USAGE
getBlock - Fetch a block from the blockchain.

PARAMETERS
{
  "block_num_or_id": "string"
}
```

Start a nodeosd process.  The docker in this repository provides a setup
the supports the examples in this README.

```bash
cd ./docker && ./up.sh
```

All functions read only or transactional follow this pattern for parameters.

```js
// If the last argument is a function it is treated as a callback
eos.getBlock(1, (error, result) => {})

// If a callback is not provided, a Promise is returned
eos.getBlock(1) // @returns {Promise}

// Parameters can be positional or an object
eos.getBlock({block_num_or_id: 1})

// An API with no parameters is invoked with an empty object or callback (avoids logging usage)
eos.getInfo({}) // @returns {Promise}
eos.getInfo((error, result) => { console.log(error, result) })
```

Chain and history API functions are available after creating the `eos` object.
API methods and documentation are generated from the chain and history json files.

* [chain.json](https://github.com/EOSIO/eosjs-api/blob/master/src/api/v1/chain.json)
* [history.json](https://github.com/EOSIO/eosjs-api/blob/master/src/api/v1/history.json)

Until we generate a markdown for these, please convert the names in these
json to camel case functions.

* `"get_info": ..` is `eos.getInfo(..)`

### Configuration

```js
Eos = require('eosjs')

// Default configuration (additional options below)
config = {
  chainId: null, // 32 byte (64 char) hex string
  keyProvider: ['PrivateKeys...'], // WIF string or array of keys..
  httpEndpoint: 'http://<IP_ADDRESS>:8888',
  expireInSeconds: 60,
  broadcast: true,
  verbose: false, // API activity
  sign: true
}

eos = Eos(config)
```

* **chainId** `hex` - Unique ID for the blockchain you're connecting too.  This
  is required for valid transaction signing.  The chainId is provided via the
  [get_info](http://ayeaye.cypherglass.com:8888/v1/chain/get_info) API call.

  Identifies a chain by its initial genesis block.  All transactions signed will
  only be valid the blockchain with this chainId.  Verify the chainId for
  security reasons.

* **keyProvider** `[array<string>|string|function]` - Provides private keys
  used to sign transaction.  If multiple private keys are found, the API
  `get_required_keys` is called to discover which signing keys to use.  If a
  function is provided, this function is called for each transaction.

* **httpEndpoint** `string` - http or https location of a nodeosd server
  providing a chain API.  When using eosjs from a browser remember to configure
  the same origin policy in nodeosd or proxy server.  For testing, nodeosd
  configuration `access-control-allow-origin = *` could be used.

  Set this value to **null** for a cold-storage (no network) configuration.

* **expireInSeconds** `number` - number of seconds before the transaction
  will expire.  The time is based on the nodeosd's clock.  An unexpired
  transaction that may have had an error is a liability until the expiration
  is reached, this time should be brief.

* **broadcast** `[boolean=true]` - post the transaction to
  the blockchain.  Use false to obtain a fully signed transaction.

* **verbose** `[boolean=false]` - verbose logging such as API activity.

* **debug** `[boolean=false]` - low level debug logging (serialization).

* **sign** `[boolean=true]` - sign the transaction with a private key.  Leaving
  a transaction unsigned avoids the need to provide a private key.

* **mockTransactions** (advanced)
  * `mockTransactions: () => null // 'pass',  or 'fail'`
  * `pass` - do not broadcast, always pretend that the transaction worked
  * `fail` - do not broadcast, pretend the transaction failed
  * `null|undefined` - broadcast as usual

* **transactionHeaders** (advanced) - manually calculate transaction header.  This
  may be provided so eosjs does not need to make header related API calls to
  nodeos.  Used in environments like cold-storage.  This callback is called for
  every transaction. Headers are documented here [eosjs-api#headers](https://github.com/EOSIO/eosjs-api/blob/HEAD/docs/index.md#headers--object).
  * `transactionHeaders: (expireInSeconds, callback) => {callback(null/*error*/, headers)}`

* **logger** - default logging configuration.
  ```js
  logger: {
    log: config.verbose ? console.log : null,
    error: console.error // null to disable
  }
  ```

  Turn off all error logging: `config.logger = {error: null}`

### Options

Options may be provided after parameters.

```js
options = {
  authorization: 'alice@active',
  broadcast: true,
  sign: true
}
```

```js
eos.transfer('alice', 'bob', '1.0000 SYS', '', options)
```

* **authorization** `[array<auth>|auth]` - identifies the
  signing account and permission typically in a multisig
  configuration.  Authorization may be a string formatted as
  `account@permission` or an `object<{actor: account, permission}>`.
  * If missing default authorizations will be calculated.
  * If provided additional authorizations will not be added.
  * Performs deterministic sorting by account name

  If a default authorization is calculated the action's 1st field must be
  an account_name.  The account_name in the 1st field gets added as the
  active key authorization for the action.

* **broadcast** `[boolean=true]` - post the transaction to
  the blockchain.  Use false to obtain a fully signed transaction.

* **sign** `[boolean=true]` - sign the transaction with a private key.  Leaving
  a transaction unsigned avoids the need to provide a private key.

### Transaction

The transaction function accepts the standard blockchain transaction.

Required transaction header fields will be added unless your signing without a
network connection (httpEndpoint == null). In that case provide you own headers:

```js
// only needed in cold-storage or for offline transactions
const headers = {
  expiration: '2018-06-14T18:16:10'
  ref_block_num: 1,
  ref_block_prefix: 452435776
}
```

Create and send (broadcast) a transaction:

```javascript
/** @return {Promise} */
eos.transaction(
  {
    // ...headers,
    actions: [
      {
        account: 'eosio.token',
        name: 'transfer',
        authorization: [{
          actor: 'inita',
          permission: 'active'
        }],
        data: {
          from: 'inita',
          to: 'initb',
          quantity: '7.0000 SYS',
          memo: ''
        }
      }
    ]
  }
  // options -- example: {broadcast: false}
)
```

### Named action functions

More concise functions are provided for applications that may use actions
more frequently.  This avoids having lots of JSON in the code.

```javascript
// Run with no arguments to print usage.
eos.transfer()

// Callback is last, when omitted a promise is returned
eos.transfer('inita', 'initb', '1.0000 SYS', '', (error, result) => {})
eos.transfer('inita', 'initb', '1.1000 SYS', '') // @returns {Promise}

// positional parameters
eos.transfer('inita', 'initb', '1.2000 SYS', '')

// named parameters
eos.transfer({from: 'inita', to: 'initb', quantity: '1.3000 SYS', memo: ''})

// options appear after parameters
options = {broadcast: true, sign: true}

// `false` is a shortcut for {broadcast: false}
eos.transfer('inita', 'initb', '1.4000 SYS', '', false)
```

Read-write API methods and documentation are generated from the eosio
[token](https://github.com/EOSIO/eosjs/blob/master/src/schema/eosio_token.json) and
[system](https://github.com/EOSIO/eosjs/blob/master/src/schema/eosio_system.json).

Assets amounts require zero padding.  For a better user-experience, if you know
the correct precision you may use DecimalPad to add the padding.

```js
DecimalPad = Eos.modules.format.DecimalPad
userInput = '10.2'
precision = 4
assert.equal('10.2000', DecimalPad(userInput, precision))
```

For more advanced signing, see `keyProvider` and `signProvider` in
[index.test.js](https://github.com/EOSIO/eosjs/blob/master/src/index.test.js).

### Shorthand

Shorthand is available for some types such as Asset and Authority.  This syntax
is only for concise functions and does not work when providing entire transaction
objects to `eos.transaction`..

For example:
* permission `inita` defaults `inita@active`
* authority `'EOS6MRy..'` expands `{threshold: 1, keys: [key: 'EOS6MRy..', weight: 1]}`
* authority `inita` expands `{{threshold: 1, accounts: [..actor: 'inita', permission: 'active', weight: 1]}}`

### New Account

New accounts will likely require some staked tokens for RAM and bandwidth.

```javascript
wif = '5KQwrPbwdL6PhXujxW37FSSQZ1JiwsST4cqQzDeyXtP79zkvFD3'
pubkey = 'EOS6MRyAjQq8ud7hVNYcfnVPJqcVpscN5So8BhtHuGYqET5GDW5CV'

eos.transaction(tr => {
  tr.newaccount({
    creator: 'eosio',
    name: 'myaccount',
    owner: pubkey,
    active: pubkey
  })

  tr.buyrambytes({
    payer: 'eosio',
    receiver: 'myaccount',
    bytes: 8192
  })

  tr.delegatebw({
    from: 'eosio',
    receiver: 'myaccount',
    stake_net_quantity: '10.0000 SYS',
    stake_cpu_quantity: '10.0000 SYS',
    transfer: 0
  })
})
```

### Contract

Deploy and call smart contracts.

#### Compile

If you're loading a **wasm** file, you do not need binaryen. If you're loading
a **wast** file you can include and configure the binaryen compiler, this is
used to compile to **wasm** automatically when calling **setcode**.

Versions of binaryen may be [problematic](https://github.com/EOSIO/eos/issues/2187).

```bash
$ npm install<EMAIL_ADDRESS>```

```js
binaryen = require('binaryen')
eos = Eos({keyProvider, binaryen})
```

#### Deploy

```javascript
wasm = fs.readFileSync(`docker/contracts/eosio.token/eosio.token.wasm`)
abi = fs.readFileSync(`docker/contracts/eosio.token/eosio.token.abi`)

// Publish contract to the blockchain
eos.setcode('myaccount', 0, 0, wasm) // @returns {Promise}
eos.setabi('myaccount', JSON.parse(abi)) // @returns {Promise}
```

#### Fetch a smart contract

```js
// @returns {Promise}
eos.contract('myaccount', [options], [callback])

// Run immediately, `myaction` returns a Promise
eos.contract('myaccount').then(myaccount => myaccount.myaction(..))

// Group actions. `transaction` returns a Promise but `myaction` does not
eos.transaction('myaccount', myaccount => { myaccount.myaction(..) })

// Transaction with multiple contracts
eos.transaction(['myaccount', 'myaccount2'], ({myaccount, myaccount2}) => {
   myaccount.myaction(..)
   myaccount2.myaction(..)
})
```

#### Offline or cold-storage contract

```js
eos = Eos({httpEndpoint: null})

abi = fs.readFileSync(`docker/contracts/eosio.token/eosio.token.abi`)
eos.fc.abiCache.abi('myaccount', JSON.parse(abi))

// Check that the ABI is available (print usage)
eos.contract('myaccount').then(myaccount => myaccount.create())
```
#### Offline or cold-storage transaction

```js
// ONLINE

// Prepare headers
expireInSeconds = 60 * 60 // 1 hour

eos = Eos(/* {httpEndpoint: 'https://..'} */)

info = await eos.getInfo({})
chainDate = new Date(info.head_block_time + 'Z')
expiration = new Date(chainDate.getTime() + expireInSeconds * 1000)
expiration = expiration.toISOString().split('.')[0]

block = await eos.getBlock(info.last_irreversible_block_num)

transactionHeaders = {
  expiration,
  ref_block_num: info.last_irreversible_block_num & 0xFFFF,
  ref_block_prefix: block.ref_block_prefix
}

// OFFLINE (bring `transactionHeaders`)

// All keys in keyProvider will sign.
eos = Eos({httpEndpoint: null, chainId, keyProvider, transactionHeaders})

transfer = await eos.transfer('inita', 'initb', '1.0000 SYS', '')
transferTransaction = transfer.transaction

// ONLINE (bring `transferTransaction`)

eos = Eos(/* {httpEndpoint: 'https://..'} */)

processedTransaction = await eos.pushTransaction(transferTransaction)
```

#### Custom Token

```js
// more on the contract / transaction syntax

await eos.transaction('myaccount', myaccount => {

  // Create the initial token with its max supply
  // const options = {authorization: 'myaccount'} // default
  myaccount.create('myaccount', '10000000.000 TOK')//, options)

  // Issue some of the max supply for circulation into an arbitrary account
  myaccount.issue('myaccount', '10000.000 TOK', 'issue')
})

const balance = await eos.getCurrencyBalance('myaccount', 'myaccount', 'TOK')
console.log('Currency Balance', balance)
```

### Calling Actions

Other ways to use contracts and transactions.

```javascript
// if either transfer fails, both will fail (1 transaction, 2 messages)
await eos.transaction(eos =>
  {
    eos.transfer('inita', 'initb', '1.0000 SYS', ''/*memo*/)
    eos.transfer('inita', 'initc', '1.0000 SYS', ''/*memo*/)
    // Returning a promise is optional (but handled as expected)
  }
  // [options],
  // [callback]
)

// transaction on a single contract
await eos.transaction('myaccount', myaccount => {
  myaccount.transfer('myaccount', 'inita', '10.000 TOK@myaccount', '')
})

// mix contracts in the same transaction
await eos.transaction(['myaccount', 'eosio.token'], ({myaccount, eosio_token}) => {
  myaccount.transfer('inita', 'initb', '1.000 TOK@myaccount', '')
  eosio_token.transfer('inita', 'initb', '1.0000 SYS', '')
})

// The contract method does not take an array so must be called once for
// each contract that is needed.
const myaccount = await eos.contract('myaccount')
await myaccount.transfer('myaccount', 'inita', '1.000 TOK', '')

// a transaction to a contract instance can specify multiple actions
await myaccount.transaction(myaccountTr => {
  myaccountTr.transfer('inita', 'initb', '1.000 TOK', '')
  myaccountTr.transfer('initb', 'inita', '1.000 TOK', '')
})
```

# Development

From time-to-time the eosjs and nodeos binary format will change between releases
so you may need to start `nodeos` with the `--skip-transaction-signatures` parameter
to get your transactions to pass.

Note, `package.json` has a "main" pointing to `./lib`.  The `./lib` folder is for
es2015 code built in a separate step. If you're changing and testing code,
import from `./src` instead.

```javascript
Eos = require('./src')

// forceActionDataHex = false helps transaction readability but may trigger back-end bugs
config = {verbose: true, debug: false, broadcast: true, forceActionDataHex: true, keyProvider}

eos = Eos(config)
```

#### Fcbuffer

The `eos` instance can provide serialization:

```javascript
// 'asset' is a type but could be any struct or type like: transaction or uint8
type = {type: 1, data: '00ff'}
buffer = eos.fc.toBuffer('extensions_type', type)
assert.deepEqual(type, eos.fc.fromBuffer('extensions_type', buffer))

// ABI Serialization
eos.contract('eosio.token', (error, eosio_token) => {
  create = {issuer: 'inita', maximum_supply: '1.0000 SYS'}
  buffer = eosio_token.fc.toBuffer('create', create)
  assert.deepEqual(create, eosio_token.fc.fromBuffer('create', buffer))
})
```

Use Node v10+ for `package-lock.json`.

# Related Libraries

These libraries are integrated into `eosjs` seamlessly so you probably do not
need to use them directly.  They are exported here giving more API access or
in some cases may be used standalone.

```javascript
var {format, api, ecc, json, Fcbuffer} = Eos.modules
```
* format [./format.md](./docs/format.md)
  * Blockchain name validation
  * Asset string formatting

* eosjs-api [[Github](https://github.com/eosio/eosjs-api), [NPM](https://www.npmjs.org/package/eosjs-api)]
  * Remote API to an EOS blockchain node (nodeos)
  * Use this library directly if you need read-only access to the blockchain
    (don't need to sign transactions).

* eosjs-ecc [[Github](https://github.com/eosio/eosjs-ecc), [NPM](https://www.npmjs.org/package/eosjs-ecc)]
  * Private Key, Public Key, Signature, AES, Encryption / Decryption
  * Validate public or private keys
  * Encrypt or decrypt with EOS compatible checksums
  * Calculate a shared secret

* json {[api](https://github.com/EOSIO/eosjs-api/blob/master/src/api), [schema](https://github.com/EOSIO/eosjs/blob/master/src/schema)},
  * Blockchain definitions (api method names, blockchain schema)

* eosjs-keygen [[Github](https://github.com/eosio/eosjs-keygen), [NPM](https://www.npmjs.org/package/eosjs-keygen)]
  * private key storage and key management

* Fcbuffer [[Github](https://github.com/eosio/eosjs-fcbuffer), [NPM](https://www.npmjs.org/package/fcbuffer)]
  * Binary serialization used by the blockchain
  * Clients sign the binary form of the transaction
  * Allows client to know what it is signing

# Browser

```bash
git clone https://github.com/EOSIO/eosjs.git
cd eosjs
npm install
npm run build_browser
# builds: ./dist/eos.js load with ./dist/index.html

npm run build_browser_test
# builds: ./dist/test.js run with ./dist/test.html
```

```html
<script src="eos.js"></script>
<script>
var eos = Eos()
//...
</script>
```

# Environment

Node and browser (es2015)

license: MIT

repository: EOSBlox/airDropWebsite

branch_name: refs/heads/master

extension: md

license_type: permissive

repo_name: EOSBlox/airDropWebsite


text:
# How to process Cowrie output into Graylog

## Prerequisites

* Working Cowrie installation
* Working Graylog installation

## Cowrie Configuration

* Open the Cowrie configuration file and uncomment these 3 lines.

```
[output_localsyslog]
facility = USER
format = text
```

* Restart Cowrie

## Graylog Configuration

* Open the Graylog web interface and click on the **System** drop-down in the top menu. From the drop-down menu select **Inputs**. Select **Syslog UDP** from the drop-down menu and click the **Launch new input** button. In the modal dialog enter the following information.

**Title:** Cowrie
**Port:** 8514
**Bind address:** <IP_ADDRESS>

* Then click **Launch.**

## Syslog Configuration

* Create a rsyslog configuration file in /etc/rsyslog.d

```
$ sudo nano /etc/rsyslog.d/85-graylog.conf
```

* Add the following lines to the file

```
$template GRAYLOGRFC5424,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msg%\n"
*.* @<IP_ADDRESS>:8514;GRAYLOGRFC5424
```

* Save and quit.

* Restart rsyslog

```
$ sudo service rsyslog restart
```

gha_license_id: NOASSERTION


text:
# Model

The design is based on [MuseGAN](https://salu133445.github.io/musegan/) [1] as
differences summarized below:

- Instead of using multiple generators and shared/private input vectors to
  handle the multi-track interdependency, we use shared/private generators with
  one single input vector.
- Instead of using one single discriminator, we introduce shared/private design
  to the discriminator.
- To help the discriminator extract musically-relevant features, we add to the
  discriminator an _onset/offset stream_ and a _chroma stream_.

## Generator

The generator _G_ consists of a shared network Gs followed by
_M_ private network Gpi (_i_ = 1 … _M_), one
for each track. The _shared_ network is responsible for producing a high-level
representation of the output musical segments that is shared by all the tracks.
Each _private_ network then turns such abstraction into the final piano-roll
output for the corresponding track.

> This design reflects the intuition that different tracks have their own
musical properties (e.g., textures, common-used patterns, playing techniques),
while jointly they follow a common, high-level musical idea.

## Refiner

The refiner _R_ is composed of _M_ private networks Ri, (_i_ =
1 … _M_), one for each track. The refiner aims to refine the real-valued
outputs of the generator into binary ones rather than learning a new mapping
from _G_(_z_) to the data space. Hence, we draw inspiration from _residual
learning_ and propose to construct the refiner with a number of _residual
units_. The output layer of the refiner is made up of either DBNs or SBNs.

The refiner network.
(The tensor size remains the same throughout the network.)

Residual unit used in the refiner network.
(The values denote the kernel size and the number of the output channels.)

## Discriminator

Similar to the generator, the discriminator _D_ consists of _M_ private network
Dpi (_i_ = 1 … _M_), one for each track,
followed by a shared network Ds. Each private network first
extracts low-level features from the corresponding track of the input
piano-roll. Their outputs are then concatenated and sent to the shared network
to extract higher-level abstraction shared by all the tracks.

In the _onset/offset stream_, the differences between adjacent elements in the
piano-roll along the time axis are first computed, and then the resulting matrix
is summed along the pitch axis, which is finally fed to Do.

In the _chroma stream_, the piano-roll is viewed as a sequence of one-beat-long
frames. A chroma vector is then computed for each frame and jointly form a
matrix, which is then be fed to Dc.

> Note that all the operations involved in computing the chroma and onset/offset
features are differentiable, and thereby we can still train the whole network by
back propagation.

Finally, the intra-bar features extracted from the three streams are fed to
Dm to extract inter-bar features and to make the final
prediction.

## Reference

1. Hao-Wen Dong, Wen-Yi Hsiao, Li-Chia Yang, and Yi-Hsuan Yang,
   "MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic
   Music Generation and Accompaniment,"
   in _AAAI Conference on Artificial Intelligence_ (AAAI), 2018.


text:
# grunt-output

> Output messages with grunt.log() / grunt.verbose(). Having a task to output messages has some advantages over grunt.log function: on the one hand it will queue messages so they will be actually printed before or after another task; on the other it will allow you to have predefined templates to output headers, separators, etc with colors and styling.

## Getting Started
This plugin requires Grunt `~0.4.5`

If you haven't used [Grunt](http://gruntjs.com/) before, be sure to check out the [Getting Started](http://gruntjs.com/getting-started) guide, as it explains how to create a [Gruntfile](http://gruntjs.com/sample-gruntfile) as well as install and use Grunt plugins. Once you're familiar with that process, you may install this plugin with this command:

```shell
npm install grunt-output --save-dev
```

Once the plugin has been installed, it may be enabled inside your Gruntfile with this line of JavaScript:

```js
grunt.loadNpmTasks('grunt-output');
```

Another great way to automatically load NPM tasks is to use the [load-grunt-tasks](https://github.com/sindresorhus/load-grunt-tasks) plugin.

## The "output" task

### Overview
In your project's Gruntfile, add a section named `output` to the data object passed into `grunt.initConfig()`.

```js
grunt.initConfig({
  output: {
    my_style: {
      before :
      {
        mode : 'log',
        func : 'writeln',
        before : '',
        text : '************************************************************************************',
        color : 'magenta',
        styles: ['bold']
      },
      content :
      {
        mode : 'log',
        func : 'writeln',
        before : '>> ',
        after : ' <<',
        color: 'white',
        styles: ['underline', 'bold']
      },
      after :
      {
        mode : 'log',
        func : 'writeln',
        text : '************************************************************************************',
        color : 'magenta',
        styles: ['bold']
      }
    }
  }
});
```

Another great way to automatically configure tasks is to use the [load-grunt-config](https://github.com/firstandthird/load-grunt-config) plugin.

After that you can log messages calling the output task:
```js
grunt.task.run('output:my_style:This is the begining of my build script!');
```

Or include the taks among your custom task:
```js
grunt.registerTask(
  'default',
  [
    'output:my_style:This is the beginning of my build script!',
    'build:dev',
    'clean:dev',
    ...
  ]
);
```

Or even from the shell if that makes sense for you:
```shell
grunt output:my_style:This is the begining of my build script!
```

Which will log the following text when the task is executed (asterisks will be in magenta color and text in white):
```shell
************************************************************************************
>> This is the begining of my build script! <<
************************************************************************************
```

### Options
You can set three options in your targets: 'before', 'content' and 'after'. Options 'before' and 'after' define the text lines that will be logged before and after the message you want to output. The 'content' option defines the line with your message. Each of these options accept the same suboptions, represented here:

#### options.mode
Type: `String`
Default value: `'log'`

Which grunt object use to log (either 'log' or 'verbose'). More information about them can be found at the [Grunt API website](http://gruntjs.com/api/grunt.log).

#### options.func
Type: `String`
Default value: `'writeln'`

Which of the grunt object's functions use to log. More information about them can be found at the [Grunt API website](http://gruntjs.com/api/grunt.log).

#### options.before
Type: `String`
Default value: `''`

String to add before the text line.

#### options.after
Type: `String`
Default value: `''`

String to add after the text line.

#### options.text
Type: `String`
Default value: `''`

String in the middle of the line.

#### options.color
Type: `String`
Default value: `''`

Text color of the line to be printed. You can use any color available in the [npm colors package](https://www.npmjs.com/package/colors).

#### options.styles
Type: `Array`
Default value: `[]`

Array of styles to apply to the line of text. You can use any style available in the [npm colors package](https://www.npmjs.com/package/colors).

#### options.extras
Type: `Array`
Default value: `[]`

Array of extras to apply to the line of text. You can use any extra available in the [npm colors package](https://www.npmjs.com/package/colors).

### Usage Examples

#### Header 1 style
In this example, we are defining a colorful style for a heading:

```js
grunt.initConfig({
  output: {
    h1 : {
        before :
        {
            text : '************************************************************************************\n',
            after : '************************************************************************************',
            styles: ['bold'],
            extras: ['rainbow']
        },
        content :
        {
            before : '>> ',
            after : ' <<',
            color: 'cyan',
            styles: ['underline', 'bold']
        },
        after :
        {
            text : '************************************************************************************\n',
            after : '************************************************************************************',
            styles: ['bold'],
            extras: ['rainbow']
        }
    }
});
```

Which can be called with:

```js
grunt.task.run('output:h1:Not only I am colorful, but also important!');
```

#### Header 4 style
This is a not so visible header style that will be only showed in verbose mode (grunt -v ...).

```js
grunt.initConfig({
  output: {
    h4 : {
        content :
        {
			mode : 'verbose',
            before : '>> ',
            after : ' <<',
            color: 'yellow',
            styles: ['bold']
        }
    }
});
```

Which can be called with:

```js
grunt.task.run('output:h4:I am not a very important message, almost nobody will see me');
```

## Contributing
In lieu of a formal styleguide, take care to maintain the existing coding style. Add unit tests for any new or changed functionality. Lint and test your code using [Grunt](http://gruntjs.com/).

repository: lucaslopez/grunt-output

gha_language: JavaScript

repo_name: lucaslopez/grunt-output


text:
# Mastering OpenCV 4 with Python

This is the code repository for [Mastering OpenCV 4 with Python](https://www.packtpub.com/application-development/mastering-opencv-4-python?utm_source=github&utm_medium=repository&utm_campaign=9781789344912 ), published by Packt.

**A practical guide covering topics from image processing, augmented reality to deep learning with OpenCV 4 and Python 3.7**

## What is this book about?
OpenCV is considered to be one of the best Open Source Computer Vision and machine learning software libraries. It helps developers build complete projects on image processing, motion detection, and image segmentation. OpenCV for Python enables you to run computer vision algorithms smoothly in real time, combining the best of the OpenCV C++ API and the Python language.

This book covers the following exciting features:
* Handle files and images, and explore various image processing techniques
* Explore image transformations like translation, resizing, and cropping
* Gain insights into building histograms
* Brush up on contour detection, filtering, and drawing
* Work with augmented reality and 3D visualization frameworks
* Work with machine learning, deep learning, and neural network algorithms

If you feel this book is for you, get your [copy](https://www.amazon.com/dp/1789344913) today!

## Instructions and Navigations
All of the code is organized into folders. For example, Chapter02.

The code will look like the following:
```
channels = cv2.split(img)
eq_channels = []
for ch in channels:
    eq_channels.append(cv2.equalizeHist(ch))
```

## Code Testing Specifications
*Mastering OpenCV 4 with Python* requires some installed packages, which you can see next.

* Chapter01 (*Setting Up OpenCV*): ``opencv-contrib-python``
* Chapter02 (*Image Basics in OpenCV*): ``opencv-contrib-python matplotlib``
* Chapter03 (*Handling Files and Images*): ``opencv-contrib-python matplotlib``
* Chapter04 (*Constructing Basic Shapes in OpenCV*): ``opencv-contrib-python matplotlib``
* Chapter05 (*Image Processing Techniques*): ``opencv-contrib-python matplotlib``
* Chapter06 (*Constructing and Building Histograms*): ``opencv-contrib-python matplotlib``
* Chapter07 (*Thresholding Techniques*): ``opencv-contrib-python matplotlib scikit-image, scipy``
* Chapter08 (*Contours Detection, Filtering, and Drawing*): ``opencv-contrib-python matplotlib``
* Chapter09 (*Augmented Reality*): ``opencv-contrib-python matplotlib``
* Chapter10 (*Machine Learning with OpenCV*): ``opencv-contrib-python matplotlib``
* Chapter11 (*Face Detection, Tracking, and Recognition*): ``opencv-contrib-python matplotlib dlib face-recognition cvlib requests progressbar keras tensorflow``
* Chapter12 (*Introduction to Deep Learning*): ``opencv-contrib-python matplotlib tensorflow keras``
* Chapter13 (*Mobile and Web Computer Vision with Python and OpenCV*): ``opencv-contrib-python matplotlib flask tensorflow keras requests pillow``

Make sure that the version numbers of your installed packages are equal to, or greater than, versions specified below to ensure the code examples run correctly. If you want to install the exact versions this book was tested on, include the version when installing from pip.

* Install opencv-contrib-python:

```
pip install opencv-contrib-python==<IP_ADDRESS>
```
It should be noted that OpenCV requires: ``numpy``

``numpy-1.16.1`` has been installed when installing ``opencv-contrib-python==<IP_ADDRESS>``

 * Install matplotlib:

```
pip install matplotlib==3.0.2
```
It should be noted that matplotlib requires: ``kiwisolver pyparsing six cycler python-dateutil``

``cycler-0.10.0 kiwisolver-1.0.1  pyparsing-2.3.1 python-dateutil-2.8.0 six-1.12.0`` have been installed when installing ``matplotlib==3.0.2``

 * Install scikit-image:
```
pip install scikit-image==0.14.2
```
It should be noted that scikit-image requires: ``cloudpickle decorator networkx numpy toolz dask pillow PyWavelets six``

``PyWavelets-1.0.1 cloudpickle-0.8.0 dask-1.1.1 decorator-4.3.2 networkx-2.2 numpy-1.16.1 pillow-5.4.1 six-1.12.0 toolz-0.9.0`` have been installed when installing ``scikit-image==0.14.2``

 * Install scipy:
```
pip install scipy==1.2.1
```
It should be noted that scipy requires: ``numpy``

``numpy-1.16.1`` has been installed when installing ``scipy==1.2.1``

 * Install dlib:
```
pip install dlib==19.8.1
```

 * Install face-recognition:
```
pip install face-recognition==1.2.3
```
It should be noted that face-recognition requires: ``dlib Click numpy face-recognition-models pillow``

``dlib-19.8.1 Click-7.0 face-recognition-models-0.3.0 pillow-5.4.1`` have been installed when installing ``face-recognition==1.2.3``

 * Install cvlib:
```
pip install cvlib==0.1.8
```

 * Install requests:
```
pip install requests==2.21.0
```

It should be noted that requests requires: ``urllib3 chardet certifi idna``

``urllib3-1.24.1 chardet-3.0.4 certifi-2018.11.29 idna-2.8`` have been installed when installing ``requests==2.21.0``

 * Install progressbar:
```
pip install progressbar==2.5
```

 * Install keras:

```
pip install keras==2.2.4
```
It should be noted that keras requires: ``numpy six h5py keras-applications scipy keras-preprocessing pyyaml``

``h5py-2.9.0 keras-applications-1.0.7 keras-preprocessing-1.0.9 numpy-1.16.1 pyyaml-3.13 scipy-1.2.1 six-1.12.0`` have been installed when installing ``keras==2.2.4``

 * Install tensorflow:

```
pip install tensorflow==1.12.0
```
It should be noted that tensorflow requires: ``termcolor numpy wheel gast six setuptools protobuf markdown grpcio werkzeug tensorboard absl-py h5py keras-applications keras-preprocessing, astor``

``termcolor-1.1.0 numpy-1.16.1 wheel-0.33.1 gast-0.2.2 six-1.12.0 setuptools-40.8.0 protobuf-3.6.1 markdown-3.0.1 grpcio-1.18.0 werkzeug-0.14.1 tensorboard-1.12.2 absl-py-0.7.0 h5py-2.9.0 keras-applications-1.0.7 keras-preprocessing-1.0.9 astor-0.7.1`` have been installed when installing ``tensorflow==1.12.0``

 * Install flask:
```
pip install flask==1.0.2
```

It should be noted that flask requires: ``Werkzeug click itsdangerous MarkupSafe Jinja2``

``Jinja2-2.10 MarkupSafe-1.1.1 Werkzeug-0.14.1 click-7.0 itsdangerous-1.1.0`` have been installed when installing ``flask==1.0.2``

**Following is what you need for this book:**
This book is designed for computer vision developers, engineers, and researchers who want to develop modern computer vision applications. Basic experience of OpenCV and Python programming is a must.

With the following software and hardware list you can run all code files present in the book (Chapter 1-13).
### Software and Hardware List
| Chapter | Software required | Hardware required                        |
| --------| ----------------- | ------------------------------------------------------------------------------------------------------- |
| 1-13    | Python            | Either 32-bit or 64-bit architecture, 2+ GHz CPU, 4 GB RAM, At least 10 GB of hard disk space available |
| 1-13    | OpenCV            | Either 32-bit or 64-bit architecture, 2+ GHz CPU, 4 GB RAM, At least 10 GB of hard disk space available |
| 1-13    | PyCharm           | Either 32-bit or 64-bit architecture, 2+ GHz CPU, 4 GB RAM, At least 10 GB of hard disk space available |

We also provide a PDF file that has color images of the screenshots/diagrams used in this book. [Click here to download it](https://www.packtpub.com/sites/default/files/downloads/9781789344912_ColorImages.pdf).

### Referencing
If you use the code samples or book content in your study/work or want to cite the book, please use:

Fernández Villán Alberto. Mastering OpenCV 4 with Python. Packt Publishing Ltd, 2019.

```bibtex
@book{fernandez2019mastering,
 author = {Fern{\'a}ndez, Alberto},
 title = {Mastering OpenCV 4 with Python},
 year = {2019},
 isbn = {978-1789344912},
 publisher = {Packt Publishing Ltd},
}
```

### Related products
* Learn OpenCV 4 By Building Projects - Second Edition [[Packt]](https://www.packtpub.com/application-development/learn-opencv-4-building-projects-second-edition?utm_source=github&utm_medium=repository&utm_campaign=9781789341225 ) [[Amazon]](https://www.amazon.com/dp/1789341221)

* Mastering OpenCV 4 - Third Edition [[Packt]](https://www.packtpub.com/application-development/mastering-opencv-4-third-edition?utm_source=github&utm_medium=repository&utm_campaign=9781789533576 ) [[Amazon]](https://www.amazon.com/dp/1789533570)

* OpenCV Computer Vision with Python [[Packt]](https://www.packtpub.com/application-development/opencv-computer-vision-python)  [[Amazon]](https://www.amazon.com/dp/1782163921)

* OpenCV: Computer Vision Projects with Python [[Packt]](https://www.packtpub.com/application-development/opencv-computer-vision-projects-python) [[Amazon]](https://www.amazon.com/dp/1787125491)

* Augmented Reality for Developers [[Packt]](https://www.packtpub.com/web-development/augmented-reality-developers) [[Amazon]](https://www.amazon.com/dp/1787286436)

* Deep Learning with Python and OpenCV [[Packt]](https://www.packtpub.com/big-data-and-business-intelligence/deep-learning-python-and-opencv)  [[Amazon]](https://www.amazon.com/dp/1788627326)

* Deep Learning with Keras [[Packt]](https://www.packtpub.com/big-data-and-business-intelligence/deep-learning-keras)  [[Amazon]](https://www.amazon.com/dp/1787128423)

* Getting Started with TensorFlow  [[Packt]](https://www.packtpub.com/big-data-and-business-intelligence/getting-started-tensorflow)  [[Amazon]](https://www.amazon.com/dp/1786468573)

* Mastering Flask Web Development - Second Edition [[Packt]](https://www.packtpub.com/web-development/mastering-flask-web-development-second-edition)  [[Amazon]](https://www.amazon.com/dp/1788995406)

## Get to Know the Author
**Alberto Fernández Villán**
is a software engineer with more than 12 years of experience in developing innovative solutions. In the last couple of years, he has been working in various projects related to monitoring systems for industrial plants, applying both Internet of Things (IoT) and big data technologies. He has a Ph.D. in computer vision (2017), a deep learning certification (2018), and several publications in connection with computer vision and machine learning in journals such as Machine Vision and Applications, IEEE Transactions on Industrial Informatics, Sensors, IEEE Transactions on Industry Applications, IEEE Latin America Transactions, and more. As of 2013, he is a registered and active user (albertofernandez) on the Q&A OpenCV forum.

### Suggestions and Feedback
[Click here](https://docs.google.com/forms/d/e/1FAIpQLSdy7dATC6QmEL81FIUuymZ0Wy9vH1jHkvpY57OiMeKGqib_Ow/viewform) if you have any feedback or suggestions.

### Download a free PDF

 If you have already purchased a print or Kindle version of this book, you can get a DRM-free PDF version at no cost.
Simply click on the link to claim your free PDF.

 https://packt.link/free-ebook/9781789344912

gha_language: Python

gha_license_id: MIT


text:
Brunch Marcos
=============

Simple javascript backbone.js skeleton built on top of [brunch](http://brunch.io) that includes essentials and provides some guidance at creating maintainable apps.

### Included tech

* Javascript - because you should know JS before learning anything that compiles
into it, and by using brunch you can incorporate `CoffeeScript`, `LiveScript` or anything, really, when you feel comfortable.
* [Backbone](http://backbonejs.org/) - why we're all here
* [Lo-Dash](http://lodash.com) - An alternative to Underscore.js, delivering consistency, customization, performance, and extra features.
* [jQuery](http://jquery.com/) - For that low level dom manipulation
* [Handlebars](http://handlebarsjs.com) - sensible default thats faster than underscore, wildly used and understood
* [Stylus](http://learnboost.github.com/stylus/) with [nib](http://visionmedia.github.com/nib/) - dynamic stylesheet language, comparable with LESS and SASS
* [LayoutManager](https://github.com/tbranyen/backbone.layoutmanager) - because you need something to help backbone wrangle those views

## Install

- Install [node.js](http://nodejs.org)
- Install Brunch & bower: `sudo npm install -g brunch bower`
- Run `brunch new gh:smazurov/brunch-marcos <project name>` to download the skeleton into your project folder.
- `$ cd <project name>`

## Compiling

- To build your project run `brunch build`.
- To continually watch your project folder changes and auto-compile, use `brunch watch`.
- If you want to start a simple server, run `brunch watch --server`
- If you want to build a minified version (for production release), run `brunch build --production`

See more info on the [official brunch docs](http://brunch.readthedocs.org/en/latest/)

## Unit Testing

The [Mocha](http://mochajs.org/) test-suite is included by default.  Files located in `tests` that end with `_test.js` (or .coffee) are automatically packaged.

To run tests in the browser, navigate to `public/test/index.html`.  Test-related code is automatically recompiled during `brunch build` and `brunch watch`, and will automatically refresh the test page.
To set up and get started on using a specific testing interface, take a look at the following guides:

* [TDD](https://gist.github.com/4361378)
* [BDD](https://gist.github.com/4361398)
* [QUnit](https://gist.github.com/4361413)

# More Information

## Todo

- ~~Get a basic skeleton app~~
- ~~Write tests for it~~
- Write generators
- Include a more involved example application
- Screencast

## Whys

### Stylus

Stylus is used as a `CSS` preprocessor that abstracts away a lot of painful things
we deal in with in `CSS`. It is solving the same problem as `LESS` and `SASS` with following
additional benefits:

* Customizable syntax. You can write plain CSS or you may drop braces / semicolons or even `:`s etc.
* Transparent mixins. If you don't need a lot of features, you can just stick to dropping vendor prefixes. But, of course, advanced features are supported too.
* `nib.` Contains all CSS3 mixins. It's like a `compass` for `sass` but way better because of transparent mixins

As far as compiling is concerned everything gets turned into css and compiles down to a single file when you have `brunch watch` running or do `brunch build`

### Handlebars

Handlebars was chosen due to its popularity and the way it restricts what you can do in templates, while retaining the flexibility of allowing custom functions to format the data you pass in. It closely resembles `Mustache` and immediately familiar to anyone that knows `HTML` as its not inventing any new syntax.  As a result its also one of the fastest options on the market. It is also faster than underscores template implementation.

### Backbone LayoutManager (LM)

`LM` provides a logical structure for assembling layouts with Backbone Views. Designed to be adaptive and configurable for painless integration. Includes automatic cleanup as well as ability to define your own to make sure all your events are taken care of and no zombies are left hungering for brains.

## License

Brunch Marcos is distributed under the MIT license. All 3rd party libraries and components are distributed under their respective license terms.

```
Copyright (C) 2012 Stepan Mazurov

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
documentation files (the "Software"), to deal in the Software without restriction, including without limitation the
rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit
persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the
Software.

 IN NO EVENT SHALL THE AUTHORS OR
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
```

repository: smazurov/brunch-marcos

repo_name: smazurov/brunch-marcos


text:
---
id: cb-overview
title: How to write a Litmusbook
sidebar_label: Overview
---
------

The Litmus framework adopts a user-friendly structure in which addition
of new experiments (tests) can be an easy exercise without too-much logic
building. However, it does involve packaging the test in the standard
litmusbook structure and reusing/creating simple ansible task files and
playbooks. Here is a quick refresher on the moving parts & basic process
flow in a Litmus experiment:

- The deployable/executable artifact associated with the Litmus experiment
is called Litmusbook. This is a Kubernetes Job (run_litmus_test.yml)

- The Litmusbook runs & manages the “ansible-runner” container

- The ansible-runner container executes the test playbook (test.yml) as
the entrypoint function.

- The test playbook executes several locally defined tasks as well as
those in (statically imported) LitmusLib utils/taskfiles (viz. common,
funclib, chaoslib)

- The LitmusLib utils may utilize specific “third-party” tools (deployed
as Kubernetes deployments or daemonsets) to perform specific functions

- Test result is updated in a dedicated Kubernetes custom resource called
litmusresult

The subsequent sections will explain how an interested user/developer can
approach the creation of a new Litmus Experiment. An intermediate-level
knowledge of ansible (& of course, Kubernetes) is expected.

repository: avinash-rath/litmus-docs

repo_name: avinash-rath/litmus-docs


text:
# ansible playbook (local) for dockerized kolide-fleet with automated first-time setup.

To build/deploy, cd into this directory and run: sudo ansible-playbook -k fleet-run-all.yml

Fleet will be listening on https://<IP_ADDRESS>:8443 with a self-signed certificate. On a Linux host there should be a shortcut on the desktop pointing to Fleet's homepage. On a Windows PC, this playbook will surely fail...
...if you have problems, it works on my PC (CentOS 7 w/ KDE) so try that.

To update docker-compose and related configs, run the following: sudo ansible-playbook -k update-fleet-containers.yml

repository: cyberguerrilla/kolide-fleet

repo_name: cyberguerrilla/kolide-fleet


text:
# nlp-paper-reading

[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat)](http://makeapullrequest.com)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
[![Template from jarvis](https://img.shields.io/badge/Hi-Jarvis-ff69b4.svg)](https://github.com/Armour/Jarvis)

"The more papers you read, the more you will learn and the faster this process of reading becomes."

## Contributing

See [CONTRIBUTING.md](https://github.com/Armour/nlp-paper-reading/blob/master/.github/CONTRIBUTING.md)

## License

[MIT License](https://github.com/Armour/nlp-paper-reading/blob/master/LICENSE)

repository: Armour/nlp-paper-reading

repo_name: Armour/nlp-paper-reading


text:
Private Endpoints

# Orders

## Place a New Order

> **Place a post-only limit order to sell 10 ETH at 0.04.**

> Request

```json
{
  "product": "ETH-BTC",
  "side": "sell",
  "size": "10",
  "price": "0.04",
  "condition": "PO"
}
```

> Response

```json
{
  "id": "9d335cce-d581-45f6-8efc-bdab3d61c6e2",
  "product": "ETH-BTC",
  "type": "limit",
  "side": "sell",
  "size": "5.00000000",
  "price": "0.04",
  "condition": "PO",
  "filled": "0.00000000",
  "timestamp": "1511467005839.417"
}
```

### HTTP Request

**`POST /new-order`**

### Parameters

Name | Description
---------- | -------
product | A valid product
type | Can be `limit` (default), `market`, `stop`, or `stop-limit`
side | Can be `buy` or `sell`
size | \* Amount of base asset to buy or sell
funds | \* Amount of counter asset to buy or sell
price | \*\* Price of your order

\* `funds` must be specified for `buy` `market` and `buy` `stop` orders. In all other cases, `size` must be specified. Only one of `funds` or `size` can be set.

\*\* Price can be ignored for `market` orders, but is required for all other order types. For `stop` orders, this is the price at which the stop order triggers, otherwise known as the stop price.

As an alternative to market orders, you can use limit orders with price set deep in the opposite book. This allows you to set `size` (instead of `funds`) and a price, which gives you an execution price guarantee.

A list of products can be found by querying the [/products](#products) endpoint.

#### Limit order parameters

For limit orders, you can specify an execution policy condition. This parameter is optional.

Name | Description
---------- | -------
condition | Can be `GTC` (default), `IOC`, or `PO`

`GTC` Good till cancelled orders will remain on the book until they're cancelled. This is the default order placement behavior.

`IOC` Immediate-or-cancel orders will only remove liquidity from the book. When submitted, the order will fill partially or fully and any outstanding amount that is not matched is cancelled, so that no part of the order appears on the book.

`PO` Post-only orders will only add liquidity to the book. If any part of the order results in taking liquidity, the order will be rejected and no part of it will execute.

#### Stop-limit order parameters

For stop-limit orders, `limit_price` must be specified in addition to the default request parameters.

Name | Description
---------- | -------
limit_price | The limit price of a stop-limit order

### Client order ID

You can send an optional `client_order_id` in your new order request. This field's value must be a UUID (v4) generated by you. It will be included in `received`, `open`, and `set` WebSocket events and can be used to track the state of your order.

### Order lifecycle

Your account must have sufficient balance to submit an order. An order that is successfully received by the matching engine will be marked as **received**.

If any part of the order results in adding liquidity to the book, it will be marked as **open**.

Stop orders do not appear on the book. A successfully submitted stop order will be marked as **set**.

An order that is completely filled and off the book will be marked as **done**.

### Response

An order accepted by the matching engine will be assigned an order `id` which will appear in the response.

`filled` specifies the amount that was matched as a result of your order.

`timestamp` is the time your order was received by the matching engine.

repository: dounbleprince/docs

repo_name: dounbleprince/docs


text:
# postgres-import-wizard
Responsible for importing a raw data set into a table in a Postgres database

# Table of Contents

- [About](#about)
- [Installation](#installation)
- [Usage](#usage)
- [Examples](#examples)
  - [CSV, Public Schema](#csvpublic)
  - [CSV, Non-Public Schema](#csvnotpublic)
  - [Tab-Delimited](#tab)

## About

We often have data in a file on our filesystem and we want to import it into a
table in a Postgres database. Other database management systems like Microsoft
SQL Server offer a graphical import wizard, but Postgres does not. The
`postgres-import-wizard` (henceforth known as the `wizard`) fills this gap.

The `wizard` is used to copy a flat-data file such as a CSV or tab-delimited
text file into a Postgres table. Although Postgres provides a built-in
[`COPY`](https://www.postgresql.org/docs/current/sql-copy.html) command, it
requires the table to already exist; otherwise there's nothing to copy the data
into. To circumvent this issue, the `wizard` reads the header of the file you
want to load and uses those fields to define a table with `text` columns in
a database you specify.

## Installation

No installation is required. The `wizard` is meant to be run directly from
source using Python 3.

That said, it's recommended to run this in a [Python virtual
environment](https://docs.python.org/3/library/venv.html) and install what's in
`requirements.txt` (which is really just
[`psycopg`](http://initd.org/psycopg/)).

## Usage

```
usage: postgres-import-wizard [-h] [--clean] [--delimiter {,,|,	}] --file FILE
                              [--postgres_connection POSTGRES_CONNECTION]
                              [--schema SCHEMA] --table TABLE

Responsible for importing a raw data set into a table in a Postgres database

optional arguments:
  -h, --help            show this help message and exit
  --clean               Delete any table that already exists at
                        "SCHEMA"."TABLE" (default: False)
  --delimiter {,,|,	}   Delimiter separating fields in FILE (default: ,)
  --file FILE           Path to data file to import (default: None)
  --postgres_connection POSTGRES_CONNECTION
                        Postgres connection string for raw database (default:
                        postgresql://postgres@localhost:5433/raw)
  --schema SCHEMA       Name of schema in which to create table (default:
                        public)
  --table TABLE         Name of table to create (default: None)
```

Executing the `wizard` will print log messages on standard out within your
shell. It will also persist messages from your most recent run in a file under
a `logs/` folder in your working directory. Log settings can be configured in
`log.py`, and the `logs/` folder can safely be removed whenever.

## Examples

In this section we provide a few examples illustrating usage of the `wizard`.
All examples are run using the default Postgres connection string provided by
the utility, so be sure that if you run them verbatim you have a local Postgres
instance running on your machine, available at port 5433, with a database named
`raw`. Also be sure to use a [Postgres password
file](https://www.postgresql.org/docs/current/libpq-pgpass.html) to specify
credentials for your database user.

The example data sets provided under the `examples/` folder are subsets of the
[Private School Universe Survey](https://nces.ed.gov/surveys/pss/) (PSS) for
the 2017-18 school year provided by the National Center for Education
Statistics. The PSS is one of [Niche's data
sources](https://www.niche.com/about/data/) and provides a biennial census of
private schools in the United States of America.

### CSV, Public Schema

Importing the example CSV data set provided under `examples/` can be done by
executing

```sh
$ python main.py --file examples/data.csv --table "example1"
```

Issuing that will load the comma-separated data set into a table named
`example1` within the default `public` schema of the database.

We can reimport the data:

```sh
$ python main.py --clean --file examples/data.csv --table "example1"
```

If we had omitted the `--clean` flag the `wizard` would have thrown an error
like

```sh
CRITICAL [2019-10-04 08:56:20] Exception occurred while loading raw data: relation "example1" already exists
```

### CSV, Non-Public Schema

This example functions much the same as [CSV, Public Schema](#csvpublic) above
except we try to load the data into a non-public schema:

```sh
$ python main.py --file examples/data.csv --schema "sonic" --table "example1"
INFO [2019-10-04 09:12:27] Starting the wizard
...
CRITICAL [2019-10-04 09:12:27] Exception occurred while loading raw data: schema "sonic" does not exist
LINE 2:         CREATE TABLE "sonic"."example1" (
                             ^

INFO [2019-10-04 09:12:27] Exiting the wizard. Goodbye.
```

Oh no! We need to create the schema before we can create tables in it. After
creating a schema called `sonic` we can rerun the command above and everything
should complete fine.

### Tab-Delimited

The only tricky thing with importing a tab-delimited data file is that we need
to specify a literal `\t` as the delimiter. This can be accomplished by
escaping it in our argument list with `$'\t'`, like the following

```sh
$ python main.py --delimiter $'\t' --file examples/data.txt --table "example2"
```

repository: nicheinc/postgres-import-wizard

repo_name: nicheinc/postgres-import-wizard


text:
# @tinyhttp/session

[![npm (scoped)][npm-badge]](https://npmjs.com/package/@tinyhttp/session) [![npm][dl-badge]](https://npmjs.com/package/@tinyhttp/session) [![][web-badge]](https://tinyhttp.v1rtl.site/mw/session)

Session middleware for tinyhttp. A rewrite of [micro-session](https://github.com/meyer9/micro-session).

## Install

```sh
pnpm i @tinyhttp/session
```

## API

```js
import { SessionManager, MemoryStore } from '@tinyhttp/session'
```

### `SessionManager`

Creates a `getSession` async function to get session data from `req` and `res` and save them to `store`. Only `secret` option is required, all of the rest are optional. Inherits all options from `SessionOptions`.

### Options

See [express-session `session(options)`](https://github.com/expressjs/session#sessionoptions).

### `MemoryStore`

Simple in-memory store that implements `ExpressStore` (from `express-session` types). Not intended to use in production.

```js
const store = new MemoryStore()
```

#### `store.clear(cb)`

Clear all sessions.

#### `store.destroy(id, cb)`

Destroy a session by the given ID.

#### `store.get(id, cb)`

Fetch session by the given session ID.

#### `store.set(id, session, cb)`

Commit the given session associated with the given sessionId to the store.

#### `length(cb)`

Get number of active sessions.

#### `touch(id, session, cb)`

Touch the given session object associated with the given session ID.

## Example

```js
import { App } from '@tinyhttp/app'
import { SessionManager, MemoryStore } from '@tinyhttp/session'

const app = new App()

const store = new MemoryStore()
const getSession = SessionManager({
  store,
  secret: 'test',
})

app.use(async (req, res) => {
  // Get session data from req and res
  const session = await getSession(req, res)

  // Increment on every request and keep in memory
  if (!session.test) {
    session.test = 1
  } else {
    session.test += 1
  }

  res.json({ t: session.test })
})

server.listen(3000)
```

## Alternatives

- [express-session](https://github.com/expressjs/session) - Simple session middleware for Express
- [micro-session](https://github.com/meyer9/micro-session) - Session middleware for micro
- [next-session](https://github.com/hoangvvo/next-session) - Simple promise-based session middleware for Next.js, micro, Express, and more
- [node-client-sessions](https://github.com/mozilla/node-client-sessions) - secure sessions stored in cookies

## License

MIT © [v1rtl](https://v1rtl.site)

[npm-badge]: https://img.shields.io/npm/v/@tinyhttp/session?style=flat-square
[dl-badge]: https://img.shields.io/npm/dt/@tinyhttp/session?style=flat-square
[web-badge]: https://img.shields.io/badge/website-visit-hotpink?style=flat-square


text:
# gruf-prometheus - Prometheus support for gruf

[![CircleCI](https://circleci.com/gh/bigcommerce/gruf-prometheus/tree/main.svg?style=svg)](https://circleci.com/gh/bigcommerce/gruf-prometheus/tree/main)  [![Gem Version](https://badge.fury.io/rb/gruf-prometheus.svg)](https://badge.fury.io/rb/gruf-prometheus) [![Documentation](https://inch-ci.org/github/bigcommerce/gruf-prometheus.svg?branch=main)](https://inch-ci.org/github/bigcommerce/gruf-prometheus?branch=main)

Adds Prometheus support for [gruf](https://github.com/bigcommerce/gruf) 2.7.0+. Supports Ruby 2.7-3.1.

## Installation

```ruby
gem 'gruf-prometheus'
```

In your gruf initializer:

```ruby
require 'gruf/prometheus'

Gruf.configure do |c|
  c.hooks.use(Gruf::Prometheus::Hook)
end
```

Then `bundle exec gruf` and you'll automatically have prometheus metrics for your gruf server.

The gruf server will by default run on port 9394, and can be scraped at `/metrics`.

## Integrations

### System Metrics

The gem comes with general system metrics out of the box that illustrate server health/statistics:

|Name|Type|Description|
|---|---|---|
|ruby_grpc_pool_jobs_waiting_total|gauge|Number of jobs in the gRPC thread pool that are actively waiting|
|ruby_grpc_pool_ready_workers_total|gauge|The amount of non-busy workers in the thread pool|
|ruby_grpc_pool_workers_total|gauge|Number of workers in the gRPC thread pool|
|ruby_grpc_pool_initial_size|gauge|Initial size of the gRPC thread pool|
|ruby_grpc_poll_period|gauge|Polling period for the gRPC thread pool|

### Server Metrics

Furthermore, the server interceptor measures general counts (and optionally, latencies), and can be setup via:

```ruby
::Gruf.interceptors.use(::Gruf::Prometheus::Server::Interceptor)
```

This will output the following metrics:

|Name|Type|Description|
|---|---|---|
|ruby_grpc_server_started_total|counter|Total number of RPCs started on the server|
|ruby_grpc_server_failed_total|counter|Total number of RPCs that throw an unknown, internal, data loss, failed precondition, unavailable, deadline exceeded, or cancelled exception on the server|
|ruby_grpc_server_handled_total|counter|Total number of RPCs completed on the server, regardless of success or failure|
|ruby_grpc_server_handled_latency_seconds|histogram|Histogram of response latency of RPCs handled by the server, in seconds|

Note that the histogram is disabled by default - you'll have to turn it on either through the `server_measure_latency`
configuration yielded in `Gruf::Prometheus.configure`, or through the `PROMETHEUS_SERVER_MEASURE_LATENCY` environment
variable. Also, the `measure_latency: true` option can be passed as a second argument to `Gruf.interceptors.use` to
configure this directly in the interceptor.

The precedence order for this is, from first to last, with last taking precedence:
1) `measure_latency: true` passed into the interceptor
2) `Gruf::Prometheus.configure` explicit setting globally
3) `PROMETHEUS_SERVER_MEASURE_LATENCY` ENV var globally. This is the only value set by default - to `false` - and will
   be the default unless other methods are invoked.

### Client Metrics

gruf-prometheus can also measure gruf client timings, via the interceptor:

```ruby
Gruf::Client.new(
  service: MyService,
  client_options: {
    interceptors: [Gruf::Prometheus::Client::Interceptor.new]
  }
)
```

|Name|Type|Description|
|---|---|---|
|ruby_grpc_client_started_total|counter|Total number of RPCs started by the client|
|ruby_grpc_client_failed_total|counter|Total number of RPCs that throw an unknown, internal, data loss, failed precondition, unavailable, deadline exceeded, or cancelled exception by the client|
|ruby_grpc_client_completed|counter|Total number of RPCs completed by the client, regardless of success or failure|
|ruby_grpc_client_completed_latency_seconds|histogram|Histogram of response latency of RPCs completed by the client, in seconds|

Note that the histogram is disabled by default - you'll have to turn it on either through the `client_measure_latency`
configuration yielded in `Gruf::Prometheus.configure`, or through the `PROMETHEUS_CLIENT_MEASURE_LATENCY` environment
variable. Optionally, you can pass in `measure_latency: true` into the Interceptor directly as an option argument in the
initializer.

The precedence order for this is, from first to last, with last taking precedence:
1) `measure_latency: true` passed into the interceptor
2) `Gruf::Prometheus.configure` explicit setting globally
3) `PROMETHEUS_CLIENT_MEASURE_LATENCY` ENV var globally. This is the only value set by default - to `false` - and will
   be the default unless other methods are invoked.

### Running the Client Interceptor in Non-gRPC Processes

One caveat is that you _must_ have the appropriate Type Collector setup in whatever process you are running in. If
you are already doing this in a gruf gRPC service that is using the hook provided by this gem above, no further
configuration is needed. Otherwise, in whatever bc-prometheus-ruby configuration you have setup, you'll need to ensure
the type collector is loaded:

```ruby
# prometheus_server is whatever `::Bigcommerce::Prometheus::Server` instance you are using in the current process
# Often hooks into these are exposed as configuration options, e.g. `web_collectors`, `resque_collectors`, etc
prometheus_server.add_type_collector(::Gruf::Prometheus::Client::TypeCollector.new)
```

Note that you don't need to do this for the `Gruf::Prometheus::Client::Collector`, as it is an on-demand collector
that does not run in a threaded loop.

See [bc-prometheus-ruby](https://github.com/bigcommerce/bc-prometheus-ruby#custom-server-integrations)'s documentation
on custom server integrations for more information.

## Configuration

You can further configure `Gruf::Prometheus` globally using the block syntax:

```ruby
Gruf::Prometheus.configure do |config|
  # config here
end
```

where the options available are:

| Option | Description | Default | ENV Name |
| ------ | ----------- | ------- | -------- |
| process_label | The label to use for metric prefixing | grpc | PROMETHEUS_PROCESS_LABEL |
| process_name | Label to use for process name in logging | grpc | PROMETHEUS_PROCESS_NAME |
| collection_frequency | The period in seconds in which to collect metrics | 30 | PROMETHEUS_COLLECTION_FREQUENCY |
| collectors | Any collectors you would like to start with the server. Passed as a hash of collector class => options | {} | |
| type_collectors | Any type collectors you would like to start with the server. Passed as an array of collector objects | [] | |
| server_measure_latency| Whether or not to measure latency as a histogram for servers | 0 | PROMETHEUS_SERVER_MEASURE_LATENCY |
| client_measure_latency| Whether or not to measure latency as a histogram for clients | 0 | PROMETHEUS_CLIENT_MEASURE_LATENCY |

## License

Copyright (c) 2019-present, BigCommerce Pty. Ltd. All rights reserved

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
documentation files (the "Software"), to deal in the Software without restriction, including without limitation the
rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit
persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the
Software.

 IN NO EVENT SHALL THE AUTHORS OR
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

repository: bigcommerce/gruf-prometheus

gha_language: Ruby

repo_name: bigcommerce/gruf-prometheus


text:
<!---
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

# Overview of column binary

Spread of Yosegi's components is represented by a collection of columns.
Column is an object that implements the IColumn interface on memory.
In order to write to a file, it is necessary to mutually convert each column to binary.

There is IColumnBinaryMaker as an interface for converting.
You can add your own implementation by implementing interconversion interface.

![Convert columns](column_binary.png)

# Class with column binary
ColumnBinary is a class that expresses the data structure in memory as binary.
This class has arrays of not only byte arrays but also statistical information, class information used for conversion, compression, and child ColumnBinary class.

```
  public final String makerClassName;

  public final String compressorClassName;

  public final String columnName;
  public final ColumnType columnType;
  public final int rowCount;
  public final int rawDataSize;
  public final int logicalDataSize;
  public final int cardinality;

  public int binaryStart;
  public int binaryLength;
  public byte[] binary;

  public List<ColumnBinary> columnBinaryList;
```

| variable | summary |
|:-----------|:------------|
| makerClassName  | Class name of IColumnBinaryMaker used for conversion  |
| compressorClassName | Class name used for compression  |
| columnName | Column name |
| columnType | Column type |
| rowCount | Number of elements in this column. |
| rawDataSize | Binary byte size. |
| logicalDataSize | Logical data size of this column.<br>[What is logical data size?](../statistics/data_size.md) |
| cardinality | Cardinality. Not required. If it can not be calculated, -1. |
| binaryStart | Starting position of byte array |
| binaryLength | The length of the byte array |
| binary | Byte array converted column to binary |
| columnBinaryList | Child's ColumnBinary. It is used for Struct and Array columns. If there is no child, it is an empty List or Null. |

# Interface of IColumnBinaryMaker

```
public interface IColumnBinaryMaker{

  ColumnBinary toBinary( final ColumnBinaryMakerConfig commonConfig , final ColumnBinaryMakerCustomConfigNode currentConfigNode , final IColumn column ) throws IOException;

  IColumn toColumn( final ColumnBinary columnBinary ) throws IOException;

  int calcBinarySize( final IColumnAnalizeResult analizeResult );

  void loadInMemoryStorage( final ColumnBinary columnBinary , final IMemoryAllocator allocator ) throws IOException;

  void setBlockIndexNode( final BlockIndexNode parentNode , final ColumnBinary columnBinary , final int spreadIndex ) throws IOException;

}
```

## toBinary()

This method converts columns that are data structures on memory to binary.

## toColumn()

This method converts from binary to column.

* columnBinary is a binary that should have been converted in this class.

## calcBinarySize()
This method calculates binary uncompressed size from column statistics.
This data size is used to determine the encoding rule.

## loadInMemoryStorage()
This method loads directly into the data structure of another column type.
Assigning data is done through the interface of IMemoryAllocator.

## setBlockIndexNode
This function sets the index of Predicate pushdown.
Since it is not required, this function does nothing if it does not have an index.

# Example

## A simple example of column encoding

```
  ColumnBinary toBinary( final ColumnBinaryMakerConfig commonConfig , final ColumnBinaryMakerCustomConfigNode currentConfigNode , final IColumn column ) throws IOException;
```

This section describes the implementation of toBinary().

In this example, it holds the information of the Int type column containing Null.
Therefore, the binary layout is a byte array representing the column length and NULL and a byte array of Int.
Conversion of Int to binary is done by using ByteBuffer.

```
    // Use common settings. If there is a setting unique to this column, use that setting.
    ColumnBinaryMakerConfig currentConfig = commonConfig;
    if( currentConfigNode != null ){
      currentConfig = currentConfigNode.getCurrentConfig();
    }
```

This code has selected the setting to adapt. If there is a setting unique to this column, this class adopts that setting.If there is not, this class adopts common setting.

This setting will be passed from Writer when converting Spread.

```
    // Calculate buffer size from column size.
    // In this example, 0 and 1 are represented by bytes in the expression of NULL.
    // Therefore, the buffer size is the sum of the byte size used to represent NULL and the Int byte size represe
nting Int and the number of columns.
    int nullByteLength = column.size() * Byte.BYTES;
    int intBinaryLength = column.size() * Integer.BYTES;
    byte[] buffer = new byte[ Integer.BYTES + nullByteLength + intBinaryLength ];

    // Substitute numeric and NULL information into buffer
    ByteBuffer.wrap( buffer , 0 , Integer.BYTES ).putInt( column.size() );
    ByteBuffer nullBuffer = ByteBuffer.wrap( buffer , Integer.BYTES , column.size() );
    ByteBuffer intBuffer = ByteBuffer.wrap( buffer , Integer.BYTES + nullByteLength , intBinaryLength );
    int rows = 0;
    for( int i = 0 ; i < column.size() ; i++ ){
      ICell cell = column.get(i);
      if( cell.getType() == ColumnType.NULL ){
        nullBuffer.put( (byte)1 );
        intBuffer.putInt( 0 );
      }
      else{
        nullBuffer.put( (byte)0 );
        int num = ( (PrimitiveCell)cell ).getRow().getInt();
        intBuffer.putInt( num );
        rows++;
      }
    }
```
Calculate the buffer size from the column size beforehand and create a byte array.
It then loops by the size of the column and retrieves the data from the column.

Get Cell from the column and insert 1 into the null buffer offset if the column type is NULL.
Otherwise, we will get the data in the Int type and insert it into the offset of the Int type buffer.

In this example we implement an implementation that does not contain unintended columns, but you may want to check if it is Int column type strictly.

```
    // Compress buffer
    byte[] compressBinary = currentConfig.compressorClass.compress( buffer , 0 , buffer.length );
```

About compression can be obtained from setting.
It is possible to ignore this setting, but you should not.

```
    // Create a new ColumnBinary
    return new ColumnBinary(
      this.getClass().getName(),
      currentConfig.compressorClass.getClass().getName(),
      column.getColumnName(),
      ColumnType.INTEGER,
      rows,
      compressBinary.length,
      intBinaryLength,
      -1,
      compressBinary,
      0,
      compressBinary.length,
      null
    );
```
Create a ColumnBinary.
The returned object is included in toBinary () as an argument.

## A simple example of column decoding

```
IColumn toColumn( final ColumnBinary columnBinary ) throws IOException;
```

This section describes the implementation of toClumn().

Convert binary to Yosegi column objects.
In this example, PrimitiveColumn is used, but in case of seeking performance etc., delay processing and light column mounting etc are required.
Please refer to the implementation included below common if performance is required.

```
    // Decompress binary
    ICompressor compressor = FindCompressor.get( columnBinary.compressorClassName );
    byte[] binary = compressor.decompress( columnBinary.binary , columnBinary.binaryStart , columnBinary.binaryLength );
```

The class for decompressing has a class name in ColumnBinary.
I will get a class to do decompress with Util.
Use this class to decompress binary byte arrays from ColumnBinary.

```
    int columnSize = ByteBuffer.wrap( binary ).getInt();
    int nullByteLength = Byte.BYTES * columnSize;
    int intBinaryLength = Integer.BYTES * columnSize;
    ByteBuffer nullBuffer = ByteBuffer.wrap( binary , Integer.BYTES , nullByteLength );
    ByteBuffer intBuffer = ByteBuffer.wrap( binary , Integer.BYTES + nullByteLength , intBinaryLength );

    // Column creation and data set.
    IColumn column = new PrimitiveColumn( ColumnType.INTEGER , columnBinary.columnName );
    for( int i = 0 ; i < columnSize ; i++ ){
      byte isNull = nullBuffer.get();
      int number = intBuffer.getInt();
      if( isNull == (byte)0 ){
        column.add( ColumnType.INTEGER , new IntegerObj( number ) , i );
      }
    }
    return column;
```

It converts from binary to Int and adds it to Column only when the byte which holds null is 0.

## A simple example of in-memory loading

```
  void loadInMemoryStorage( final ColumnBinary columnBinary , final IMemoryAllocator allocator ) throws IOException;
```

This section describes the implementation of loadInMemoryStorage().
Skip the same process as toColumn.

```
    // Load data structure using IMemoryAllocator.
    for( int i = 0 ; i < columnSize ; i++ ){
      byte isNull = nullBuffer.get();
      int number = intBuffer.getInt();
      if( isNull == (byte)0 ){
        allocator.setInteger( i , number );
      }
      else{
        allocator.setNull( i );
      }
    }
```

The point different from toColumn is to set data using API of allocator.
Since this column is Int type, we use setInteger ().
Because the allocator is an interface, if you guarantee to call the same set as the type of the column, you do not have to guarantee whether it is added to the data structure held by the allocator.

```
    // Finally, set the size of the column. Load processing needs to fill in NULL value.
    allocator.setValueCount( columnSize );
```
The allocator must set the column size at the end.
The allocator does not know the column size.
For example, allocator substitutes Null using column size.

repository: yohto/yosegi

repo_name: yohto/yosegi


text:
---
name: New language/framework support
about: Suggest a support for a new language and/or test framework
title: ''
labels: language and framework support
assignees: ''

---

**Which language do you want to see supported?**

**Which test framework ?**
A link to the framework would be super useful :)

**Additional context**
Add any other context or screenshots about the feature request here.

repository: hiptest/hiptest-publisher

repo_name: hiptest/hiptest-publisher


text:
---
title: "[LeetCode] Trapping Rain Water"
categories:
- LeetCode
tags:
- 배열
toc: true   #Table Of Contents 목차
use_math: true
toc_sticky: true
---

## 문제

Given `n` non-negative integers representing an elevation map where the width of each bar is `1`, compute how much water it can trap after raining.

## 예제

**Example 1:**

![img](https://assets.leetcode.com/uploads/2018/10/22/rainwatertrap.png)

```
Input: height = [0,1,0,2,1,0,1,3,2,1,2,1]
Output: 6
Explanation: The above elevation map (black section) is represented by array [0,1,0,2,1,0,1,3,2,1,2,1]. In this case, 6 units of rain water (blue section) are being trapped.
```

**Example 2:**

```
Input: height = [4,2,0,3,2,5]
Output: 9
```

## 조건

> - `n == height.length`
> - `0 <= n <= 3 * 104`
> - `0 <= height[i] <= 105`

## 답

### 풀이 1

투 포인터를 활용한 풀이.

```python
class Solution:
    def trap(self, height: List[int]) -> int:
        n = len(height)

        left = 0
        right = n-1
        left_max = 0
        right_max = 0
        ans = 0

        if not height:
            return ans
        while(left != right):
            left_max = max(left_max, height[left])
            right_max = max(right_max, height[right])

            if left_max <= right_max:
                ans += left_max - height[left]
                left += 1
            else:
                ans += right_max - height[right]
                right -= 1

        return ans
```

### 풀이 2

스택을 활용한 풀이. (아직 이해하기 힘들다.)

```python
class Solution:
    def trap(self, height: List[int]) -> int:
        stack = []
        ans = 0

        for i in range(len(height)):
            while stack and height[i] > height[stack[-1]]:
                top = stack.pop()

                if not stack:
                    break

                distance = i - stack[-1] - 1
                waters = min(height[i], height[stack[-1]]) - height[top]

                ans += distance * waters

            stack.append(i)

        return ans
```

path: /_posts/2021-03-23-[LeetCode] 42번 Trapping Rain Water.md


text:
# Date and Time Data Plugin

[Sketch](https://sketch.com) data plugin that generates random dates and
times.

<img src="https://user-images.githubusercontent.com/1472553/53435993-fe6d7a80-39f1-11e9-8e89-0407d5a5b391.gif" alt="Screencast">

## Installation

### From a release (simplest)

- [Download](https://github.com/skpm/data-date-time/releases/latest) the latest
release of the plugin
- Double-click the .zip archive file to extract the plugin
- Double-click the `data-date-time.sketchplugin` to install the plugin

### From source

- Clone the repo
- Install the dependencies with `npm install`

## Usage

- Select one or more text layers.
- Click on the Data button in the top left of the document's toolbar.
- Under _Text_, select _Date & Time_ and then select a date or time format to
  generate.
- Press _SHIFT + COMMAND + D_ to get another random date or time.

The date or time is localized to your current language and region system
settings e.g. in the US the date will be in the format _month/day/year_ while in Europe it will be in the format _day/month/year_.

repository: iamdanduke/date-time-sketchplugin

repo_name: iamdanduke/date-time-sketchplugin


text:
---
id: 798
title: Hello Transportation Techies!
date: 2017-10-30T16:58:42+00:00
author: Tyler Green
layout: post
guid: http://www.tyleragreen.com/blog/?p=798
permalink: /blog/2017/10/hello-transportation-techies/
categories:
  - Presentations
tags:
  - dc
  - graph
  - transportation techies
---
Last Wednesday, I rode the Amtrak Northeast Regional to our nation's capital to attend my first Transportation Techies meetup! Michael had been urging me to attend since we first met at TransportationCamp Colorado 2016. Now that I'm on the east coast, it was an easy trip!

Michael always puts on an incredible event and Metro Hack Night VII was no exception! We had a great line-up and turnout. I presented on how we can use graph theory to study transit networks, specifically WMATA Metrorail with and without the Purple line.

All the presenters blew me away with their excellent, excellent projects. The DC transit developer community is special and I loved getting to experience it for one night. You can read about the all projects and presenters in this write-up from Mobility Lab.

If you live in the DC region (or anywhere along the Northeast Corridor), I encourage you to drop in for the next gathering of the Transportation Techies!

You can download the PDF deck of my presentation.

gha_language: CSS


text:
# PyWatch

This repo contains a Python implementation mimicking the classic Unix [watch](https://linux.die.net/man/1/watch) command.
It provides additional functionality to allow for scrolling, paging and cycling between multiple commands.
It should also be able to handle dynamic terminal resizing, including height, width & in/out zooming.

## Requirements
- Python >= 3.x
- Tested on Ubuntu 16.04
- Tested on the default terminal, [Yakuake](https://kde.org/applications/system/org.kde.yakuake) and [Terminator](https://gnometerminator.blogspot.com/p/introduction.html)

Update: Removed all Python >= 3.6 requirements. It should now run with any version of Python 3.

## Usage
- Remember to make the file executable: `chmod +x path/to/scripts/pywatch`
- (Optional) Add to system path, e.g. ~/.bashrc: `export PATH=/path/to/scripts/:$PATH`

#### Commands
Assuming the script has been added to the system path, it can be used as a simple replacement for "watch".
Otherwise, replace `pywatch` with `path/to/stripts/pywatch` in the examples below.

```
# Default 1 sec interval
pywatch echo Hello World

# Set interval to half a second
pywatch -n .5 "cat long_logfile.log"

# Two commands we can switch between (default 5 secs)
pywatch "echo Hello World && echo Foo Bar"

# Set cycle time and begin cycling
pywatch -c 10 "cat long_logfile.log && echo Hello World" --cycle
```

#### Controls
+ **Scrolling**: UP/DOWN keys or mouse wheel
+ **Paging**: LEFT/RIGHT keys or PageUp/PageDown keys
+ **Top/bottom of command**: Home/End keys
+ **Cycle between commands**: Tab/Ctrl+Tab keys
+ **Toggle cycling mode**: C key
+ **Change refresh interval**: I key + enter the new interval time

##### Additional Notes:
+ As an indicator, the header is displayed in CYAN when "Cycling" mode is active.
+ The TextBox used to change interval time uses Emacs bindings (which I don't think can be changed), e.g. to delete use Ctrl+H

### Licence
[MIT Licence](LICENCE)

repository: jspenmar/PyWatch

repo_name: jspenmar/PyWatch


text:
# InventoryM
**Inventory Management web application using MEAN stack (MongoDB, Express.js, Angular.js, Node.js)**

This is a personal project that I do just for fun on my free time. Will be updated with more features and tools such as firebase if I have the chance to do it.

**Features Implemented:**
- Created general layout displaying data in a view
- Designed and Implemented UI and added data to the main page
- Implemented authentication
- Passwords security implementation
- Enabled authorization capability for admin
- Implemented Signup via server-side controller and model
- Established the main layout for the hardware list that can be sorted and filtered

repository: cojocariuovidiu/InventoryM

repo_name: cojocariuovidiu/InventoryM


text:
# On the Robustness of Graph Neural Diffusion

```
@incollection{robustgnd_neurips22,
author = {Song, Yang and Kang, Qiyu and Wang, Sijie and Kai, Zhao and Peng Tay, Wee},
title = {On the Robustness of Graph Neural Diffusion to Topology Perturbations},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS) 35},
pages = {6384--6396},
year = {2022},
publisher = {Curran Associates, Inc.}
}
```

links
- [neurips](https://nips.cc/Conferences/2022/Schedule?showEvent=53396)
- [openreview](https://openreview.net/forum?id=-8tU21J6BcB)
- [arxiv](https://arxiv.org/abs/2209.07754)
- [proceedings](https://papers.nips.cc//paper_files/paper/2022/hash/29a0ea49a103a233b17c0705cdeccb66-Abstract-Conference.html)

gha_language: Jupyter Notebook


text:
Raster
======

A simple python convereter to convert sentinel2 jp2 bands to tiff.
Use it to create a dataframe table of jp2 bands and sentinel2 images within the folder.
And count the S2 images and jp2 bands within the folder.

The code is Python 3.

### Installation
```
Fast install:

pip install -i https://test.pypi.org/simple/ Ras-NEW==0.0.1

For a manual install get this package:

wget https://github.com/falahfakhri-Iraq/Ras-NEW
unzip Raster.zip
rm Raster.zip
cd Raster

Install the package:

python setup.py install
```
### Features

- Find out and present the number of S2*.SAFE within the folders as dataframe file.
- Find out and present the number of B*.jp files within the S2.*SAFE folder as dataframe file.
- convert the *.jp bands to *.tif

### Usage Example

```

Import the required library

from sentinel2 import S2_JP2tiff_Converter as S

Get the number & Dataframe table of Sentinel 2 images within the giving path or directory

Senti_number = S.Counter('D:\TESTS\TEST_NEW')

Print out the number of Sentinel 2 within the giving path

print(Senti_number.count)

Print out the Dataframe table of Sentinel2 images

print(Senti_number.Senti2_Tabel)

 Get the total number of the *.jp2 bands

jp2_bands = S.BandsJP2('D:\TESTS\TEST_NEW')

Print out the Dataframe table of the *.jp2 bands

print(jp2_bands.JP2_Tabel)

 Convert all the *.jp2 bands within the giving path or directory to *.tiff format

jp2_bands.S2Converter('D:\TESTS\TEST_NEW', 'D:\TESTS\TEST_NEW\outputFolder')

```


text:
# User Modifiable Files

This folder includes all the files that can be modified by users, the file organization is described as below:

```
clusters
  - A cluster
    - namespaces
      - a namespace
        - CRD / Standard K8s Objects
          - d CRD
          - e CRD
          ...
      - b namespace
      ...
  - B cluster
  ...
```

For each namespace fold, we have an `installs` sub-folder as a reference for users, which includes specific installation manifests.

TODO(PatrickXYS):
- Add guidance to replicate same test-infra as optional-test-infra

## Add Prow jobs in Optional-Test-Infra
Add specific prows jobs in [config.yaml](https://github.com/kubeflow/testing/tree/master/aws/User/clusters/kubeflow-shared-test-infra-poc/namespaces/prow/configmap/config.yaml) file

## Add Argo Resources in Argo cluster
Add expected yaml files in [argo folder](https://github.com/kubeflow/testing/tree/master/aws/User/clusters/kubeflow-shared-test-infra-poc-argo/namespaces/argo/)

## Add Tekton Resources in Tekton cluster
Add expected yaml files in [tekton folder](https://github.com/kubeflow/testing/tree/master/aws/User/clusters/kubeflow-shared-test-infra-poc-tekton/namespaces/tekton-templates/)

Note: any change you make in this folder, please run `make optional-generate` to generate files in GitOps folder before sending PR

repository: kubeflow/testing

repo_name: kubeflow/testing


text:
# react-parcel-boilerplate

A boilerplate project using React & Parcel

Referenced in my [Medium post](https://medium.com/javascript-in-plain-english/how-to-get-your-app-ideas-built-quickly-using-a-boilerplate-of-react-parcel-5e91c4db3059) for usage instructions.

repository: lindsayjopson/react-parcel-boilerplate

gha_language: HTML

repo_name: lindsayjopson/react-parcel-boilerplate


text:
# Docker Registry replication

## How it works

On every push of new Docker image, GitLab creates a special Geo event that is
propagated to secondary nodes. On the secondary node, there is a specialized
worker called `Geo::ContainerRepositoryServiceWorker` that will fetch the
image from primary node.

## How to set up

Docker Registry replication can be configured in GDK.

Follow steps below to enable it on your local machine. These instructions assume
you have two Geo nodes (a primary and a secondary) on your local machine. If not, follow the
[GitLab Geo](geo.md) instructions to set them up.

### Enable Docker Registry on both nodes

To enable Docker Registry on both nodes, follow [Docker Registry](registry.md).

### Enable notification on primary's Registry

Add the following lines to `registry/config.yml` of your primary node:

```yaml
notifications:
  endpoints:
    - name: geo_event
      url: http://docker.for.mac.localhost:3001/api/v4/container_registry_event/events
      headers:
        Authorization: [Bearer <secret>]
      timeout: 500ms
      threshold: 5
      backoff: 1s
```

In this example:

- `secret` is a secret word that will be used for communication between Registry and the primary
  node.
- The primary node is running on port `3001` of your localhost.
- The host name is for macOS. If you use Linux, use `host.docker.internal` instead of
  `docker.for.mac.localhost`,

### Configure the primary node

Add the secret from the Registry configuration above to `config/gitlab.yml`:

```yaml
registry:
  notification_secret: <secret>
```

### Install certificates on the secondary node

Copy `localhost.crt` and `localhost.key` from your primary node to
the secondary node.

### Restart

Please restart both nodes.


text:
![Decorative banner image](https://bennettfeely.com/usa-css/img/banner.png)

# USA.css

**Live**: https://bennettfeely.com/usacss

The most American CSS stylesheet. Style your webpage with the stars and stripes as CSS backgrounds. Units are set in inches.

God Bless America!

[![Netlify Status](https://api.netlify.com/api/v1/badges/982ee79f-6625-4916-b432-1e31bfb8b843/deploy-status)](https://app.netlify.com/sites/bennettfeely/deploys)

repository: bennettfeely/usa-css

repo_name: bennettfeely/usa-css