text:
# DCV Color Primitives - dcp

[![Build Status](https://github.com/aws/dcv-color-primitives/actions/workflows/ci.yml/badge.svg)](https://github.com/aws/dcv-color-primitives/actions/workflows/ci.yml)
[![MIT licensed](https://img.shields.io/badge/license-MIT-blue.svg)](./LICENSE)
[![crates.io](https://img.shields.io/crates/v/dcv-color-primitives.svg)](https://crates.io/crates/dcv-color-primitives)
[![documentation](https://docs.rs/dcv-color-primitives/badge.svg)](https://docs.rs/dcv-color-primitives)
[![Coverage Status](https://codecov.io/gh/aws/dcv-color-primitives/branch/master/graph/badge.svg)](https://codecov.io/gh/aws/dcv-color-primitives)

DCV Color Primitives is a library to perform image color model conversion.

## Design guidelines

* Aware of the underlying hardware and supplemental cpu extension sets (up to avx2)
* Support data coming from a single buffer or coming from multiple image planes
* Support non-tightly packed data
* Support images greater than 4GB (64 bit)
* Support ARM (aarch64)[*]
* Support WebAssembly[*]

[*]: Supplemental cpu extension sets not yet supported.

## Image format conversion

The library is currenty able to convert the following pixel formats:

| Source pixel format  | Destination pixel formats  |
| -------------------- | -------------------------- |
| ARGB                 | I420, I444, NV12           |
| BGR                  | I420, I444, NV12, RGB      |
| BGRA                 | I420, I444, NV12, RGB      |
| I420                 | BGRA, RGBA                 |
| I444                 | BGRA, RGBA                 |
| NV12                 | BGRA, RGB, RGBA            |
| RGB                  | BGRA                       |

### Color models

The supported color models are:

* YCbCr, ITU-R Recommendation BT.601 (standard video system)
* YCbCr, ITU-R Recommendation BT.709 (CSC systems)

Both standard range (0-235) and full range (0-255) are supported.

## Requirements

* Rust 1.70 and newer

### Windows

* Install rustup: https://www.rust-lang.org/tools/install

### Linux

* Install rustup (see https://forge.rust-lang.org/infra/other-installation-methods.html)
    ```
    curl https://sh.rustup.rs -sSf | sh
    ```

You may require administrative privileges.

## Building

Open a terminal inside the library root directory.

To build for debug experience:
```
cargo build
```

To build an optimized library:
```
cargo build --release
```

Run unit tests:
```
cargo test
```

Run benchmark:
```
cargo bench
```

Advanced benchmark mode.
There are two benchmark scripts:
* `run-bench.ps1` for Windows
* `run-bench.sh` for Linux and MacOS

They allow to obtain more stable results than `cargo bench`, by reducing variance due to:
* CPU migration
* File system caching
* Process priority

Moreover, the Linux script support hardware performance counters, e.g. it is possible to output
consumed CPU cycles instead of elapsed time.

Linux examples:
```
./run-bench -c 1 # runs cargo bench and outputs CPU cycles
./run.bench -c 1 -p "/i420" # runs cargo bench, output CPU cycles, filtering tests that contains '/i420'
```

## WebAssembly

Install the needed dependencies:
```
rustup target add wasm32-unknown-unknown
```

To build for debug experience:
```
cargo build --target wasm32-unknown-unknown
```

To test, ensure you have installed [wasm-pack](https://rustwasm.github.io/wasm-pack/installer/). Then:
```
wasm-pack test --node
```

## Usage

### Image conversion

Convert an image from bgra to nv12 (single plane) format containing yuv in BT601:

```rust
use dcv_color_primitives as dcp;
use dcp::{convert_image, ColorSpace, ImageFormat, PixelFormat};

fn main() {
    const WIDTH: u32 = 640;
    const HEIGHT: u32 = 480;

    let src_data = Box::new([0u8; 4 * (WIDTH as usize) * (HEIGHT as usize)]);
    let mut dst_data = Box::new([0u8; 3 * (WIDTH as usize) * (HEIGHT as usize) / 2]);

    let src_format = ImageFormat {
        pixel_format: PixelFormat::Bgra,
        color_space: ColorSpace::Rgb,
        num_planes: 1,
    };

    let dst_format = ImageFormat {
        pixel_format: PixelFormat::Nv12,
        color_space: ColorSpace::Bt601,
        num_planes: 1,
    };

    convert_image(
        WIDTH,
        HEIGHT,
        &src_format,
        None,
        &[&*src_data],
        &dst_format,
        None,
        &mut [&mut *dst_data],
    );
}
```

### Error Handling

The library functions return a `Result` describing the operation outcome:

| Result                             | Description                                                           |
| ---------------------------------- | --------------------------------------------------------------------- |
| `Ok(())`                           | The operation succeeded                                               |
| `Err(ErrorKind::InvalidValue)`     | One or more parameters have invalid values for the called function    |
| `Err(ErrorKind::InvalidOperation)` | The combination of parameters is unsupported for the called function  |
| `Err(ErrorKind::NotEnoughData)`    | One or more buffers are not correctly sized                           |

In the following example, `result` will match `Err(ErrorKind::InvalidValue)`, because `ColorSpace::Bt709`
color space is not compatible with `PixelFormat::Bgra`:

```rust
use dcv_color_primitives as dcp;
use dcp::{convert_image, ColorSpace, ErrorKind, ImageFormat, PixelFormat};

fn main() {
    const WIDTH: u32 = 640;
    const HEIGHT: u32 = 480;

    let src_data = Box::new([0u8; 4 * (WIDTH as usize) * (HEIGHT as usize)]);
    let mut dst_data = Box::new([0u8; 3 * (WIDTH as usize) * (HEIGHT as usize) / 2]);

    let src_format = ImageFormat {
        pixel_format: PixelFormat::Bgra,
        color_space: ColorSpace::Bt709,
        num_planes: 1,
    };

    let dst_format = ImageFormat {
        pixel_format: PixelFormat::Nv12,
        color_space: ColorSpace::Bt601,
        num_planes: 1,
    };

    let status = convert_image(
        WIDTH,
        HEIGHT,
        &src_format,
        None,
        &[&*src_data],
        &dst_format,
        None,
        &mut [&mut *dst_data],
    );

    match status {
        Err(ErrorKind::InvalidValue) => (),
        _ => panic!("Expected ErrorKind::InvalidValue"),
    }
}
```

Even better, you might want to propagate errors to the caller function or mix with some other error types:
```rust
use dcv_color_primitives as dcp;
use dcp::{convert_image, ColorSpace, ImageFormat, PixelFormat};
use std::error;

fn main() -> Result<(), Box<dyn error::Error>> {
    const WIDTH: u32 = 640;
    const HEIGHT: u32 = 480;

    let src_data = Box::new([0u8; 4 * (WIDTH as usize) * (HEIGHT as usize)]);
    let mut dst_data = Box::new([0u8; 3 * (WIDTH as usize) * (HEIGHT as usize) / 2]);

    let src_format = ImageFormat {
        pixel_format: PixelFormat::Bgra,
        color_space: ColorSpace::Bt709,
        num_planes: 1,
    };

    let dst_format = ImageFormat {
        pixel_format: PixelFormat::Nv12,
        color_space: ColorSpace::Bt601,
        num_planes: 1,
    };

    convert_image(
        WIDTH,
        HEIGHT,
        &src_format,
        None,
        &[&*src_data],
        &dst_format,
        None,
        &mut [&mut *dst_data],
    )?;

    Ok(())
}
```

### Buffer size computation

So far, buffers were sized taking into account the image pixel format and dimensions; However,
you can use a function to compute how many bytes are needed to store an image of a given format
and size:

```rust
use dcv_color_primitives as dcp;
use dcp::{get_buffers_size, ColorSpace, ImageFormat, PixelFormat};
use std::error;

fn main() -> Result<(), Box<dyn error::Error>> {
    const WIDTH: u32 = 640;
    const HEIGHT: u32 = 480;
    const NUM_PLANES: u32 = 1;

    let format = ImageFormat {
        pixel_format: PixelFormat::Bgra,
        color_space: ColorSpace::Rgb,
        num_planes: NUM_PLANES,
    };

    let sizes: &mut [usize] = &mut [0usize; NUM_PLANES as usize];
    get_buffers_size(WIDTH, HEIGHT, &format, None, sizes)?;

    let buffer: Vec<_> = vec![0u8; sizes[0]];

    // Do something with buffer
    // --snip--

    Ok(())
}
```

### Image planes

If your data is scattered in multiple buffers that are not necessarily contiguous, you can provide image planes:

```rust
use dcv_color_primitives as dcp;
use dcp::{convert_image, get_buffers_size, ColorSpace, ImageFormat, PixelFormat};
use std::error;

fn main() -> Result<(), Box<dyn error::Error>> {
    const WIDTH: u32 = 640;
    const HEIGHT: u32 = 480;
    const NUM_SRC_PLANES: u32 = 2;
    const NUM_DST_PLANES: u32 = 1;

    let src_format = ImageFormat {
        pixel_format: PixelFormat::Nv12,
        color_space: ColorSpace::Bt709,
        num_planes: NUM_SRC_PLANES,
    };

    let src_sizes: &mut [usize] = &mut [0usize; NUM_SRC_PLANES as usize];
    get_buffers_size(WIDTH, HEIGHT, &src_format, None, src_sizes)?;

    let src_y: Vec<_> = vec![0u8; src_sizes[0]];
    let src_uv: Vec<_> = vec![0u8; src_sizes[1]];

    let dst_format = ImageFormat {
        pixel_format: PixelFormat::Bgra,
        color_space: ColorSpace::Rgb,
        num_planes: NUM_DST_PLANES,
    };

    let dst_sizes: &mut [usize] = &mut [0usize; NUM_DST_PLANES as usize];
    get_buffers_size(WIDTH, HEIGHT, &dst_format, None, dst_sizes)?;

    let mut dst_rgba: Vec<_> = vec![0u8; dst_sizes[0]];

    convert_image(
        WIDTH,
        HEIGHT,
        &src_format,
        None,
        &[&src_y[..], &src_uv[..]],
        &dst_format,
        None,
        &mut [&mut dst_rgba[..]],
    )?;

    Ok(())
}
```

### Stride support

To take into account data which is not tightly packed, you can provide image strides:

```rust
use dcv_color_primitives as dcp;
use dcp::{convert_image, get_buffers_size, ColorSpace, ImageFormat, PixelFormat};
use std::error;

fn main() -> Result<(), Box<dyn error::Error>> {
    const WIDTH: u32 = 640;
    const HEIGHT: u32 = 480;
    const NUM_SRC_PLANES: u32 = 1;
    const NUM_DST_PLANES: u32 = 2;
    const RGB_STRIDE: usize = 4 * (((3 * (WIDTH as usize)) + 3) / 4);

    let src_format = ImageFormat {
        pixel_format: PixelFormat::Bgr,
        color_space: ColorSpace::Rgb,
        num_planes: NUM_SRC_PLANES,
    };

    let src_strides: &[usize] = &[RGB_STRIDE];

    let src_sizes: &mut [usize] = &mut [0usize; NUM_SRC_PLANES as usize];
    get_buffers_size(WIDTH, HEIGHT, &src_format, Some(src_strides), src_sizes)?;

    let src_rgba: Vec<_> = vec![0u8; src_sizes[0]];

    let dst_format = ImageFormat {
        pixel_format: PixelFormat::Nv12,
        color_space: ColorSpace::Bt709,
        num_planes: NUM_DST_PLANES,
    };

    let dst_sizes: &mut [usize] = &mut [0usize; NUM_DST_PLANES as usize];
    get_buffers_size(WIDTH, HEIGHT, &dst_format, None, dst_sizes)?;

    let mut dst_y: Vec<_> = vec![0u8; dst_sizes[0]];
    let mut dst_uv: Vec<_> = vec![0u8; dst_sizes[1]];

    convert_image(
        WIDTH,
        HEIGHT,
        &src_format,
        Some(src_strides),
        &[&src_rgba[..]],
        &dst_format,
        None,
        &mut [&mut dst_y[..], &mut dst_uv[..]],
    )?;

    Ok(())
}
```

See documentation for further information.

## C bindings

DCV Color Primitives provides C bindings. A static library will be automatically generated for the
default build.

In order to include DCV Color Primitives inside your application library, you need to:
* Statically link to dcv_color_primitives
* Link to ws2_32.lib, userenv.lib, bcrypt.lib and ntdll.lib, for Windows
* Link to libdl and libm, for Linux

The API is slightly different than the rust one. Check dcv_color_primitives.h for examples and further information.

A meson build system is provided in order to build the static library and install it together
with include file and a pkgconfig file. There are also some unit tests written in C,
to add some coverage also for the bindings. Minimal instructions are provided below, refer to meson's
help for further instructions:

* **Windows**
  Visual Studio is required. At least the following packages are required:
  * MSBuild
  * MSVC - C++ build tools
  * Windows 10 SDK

  Install meson, you can choose one of the following methods:
  1. Using meson msi installer
   * Download from https://github.com/mesonbuild/meson/releases
   * Install both Meson and Ninja
  2. Install meson through pip
   * Download and install python3: https://www.python.org/downloads/
   * Install meson and ninja:
     ```
     pip install meson ninja
     ```

  Note: Minimum required meson version is 1.0.0.

  All build commands have to be issued from Native Tools Command Prompt for VS (x86 or x64 depending on what platform you want to build)

* **Linux**
  The following example is for Ubuntu:

  ```
  #install python3
  apt install python3

  #install meson. See https://mesonbuild.com/Getting-meson.html for details or if you want to install through pip.
  apt install meson

  #install ninja
  apt install ninja-build
  ```

  You may require administrative privileges.

* **Build**
  Move inside the library root directory:
  ```
  cd `dcv_color_primitives_root_dir`
  ```

  Then:
  ```
  meson setup --buildtype release builddir
  ninja -C builddir
  ```

* **Run the tests**
  ```
  cd builddir
  meson test -t 10
  ```

  A timeout scale factor of 10 is required because some tests take longer than default
  30 seconds to complete.

* **Install**
  ```
  ninja -C builddir install
  ```

## License

This library is licensed under the MIT-0 License. See the LICENSE file.

repository: aws/dcv-color-primitives

gha_language: Rust

repo_name: aws/dcv-color-primitives


text:
#Circulo

###Contribute
Contributors welcome!! If you would like to contribute to the research page, please issue a pull request.

###About
Circulo is an exploratory effort focusing on "Community Detection" in Graphs. Though there are many areas on the web that provide useful information on the subject matter, we found that there still lacked a single place where someone new to the field could learn.  The goal of this project, therefore, is to be a central place for advancing the field. We have divided the project into the following three areas:

- Knowledge:  Build a comprehensive knowledge base that includes details about the field of Community Detection, including but not limited to summaries of algorithms, references to academic literature, and general information about the field. To view the research page for Circulo please visit [http://lab41.github.io/Circulo](http://lab41.github.io/Circulo/)
- Algorithms: Implement algorithms using existing Graph frameworks such as iGraph. Many Graph frameworks today still do not implement Community Detection algorithms, or if they do, they only implement a narrow set of algorithms. The implementations can include weighted/unweighted, directed/undirected, and nonoverlapping/overlapping graphs to name a few.
- Metrics: Implement a set of evaulation metrics that can quantitativley assess the accuracy of a partitions of a Graph into communities. The metrics assess various attributes of Graph partitions including network-wide metrics and internal/external community metrics.

###Base Frameworks
Since we did not want to reimplement the notion of a graph, we decided to pick an existing framework as a backdrop for our work.  Though any of the popular graph frameworks could have been used, iGraph was chosen as our primary graph framework. iGraph offeres number of advantages:

- First and foremost, iGraph implements a number of community detection algorithms out of the box. It also provides two data structures for community detection: VertexClustering (non-overlapping communities) and VertexCover (overlapping communities)
- iGraph is written in C at its core making it fast
- iGraph has wrappers for Python and R
- iGraph is a mature framework

Other frameworks which could be used include GraphLab, SNAP, NetworkX.

###Setup
#####Requirements

-  python 3
-  igraph
-  matplotlib
-  cairo
-  scipy
-  scikit.learn

#####Circulo
Below are instructions for using Circulo

	#clone Circulo repository
	git clone https://github.com/Lab41/circulo.git
	#set PYTHONPATH env variable
	export PYTHONPATH=/path/to/Circulo

#####iGraph
For Ubuntu

```
 sudo apt-get install igraph
 sudo apt-get install libxml2-dev libz-dev python-dev
```

For OS X, do the following. (NOTE: iGraph homepage is a good resource)

	#using brew install igraph
	brew install igraph

	#install Cairo
	#installs the core libraries for cairo
	brew install cairo

	#installs the python site-packages. NOTE: pip does not work for pycairo.
	#If you want to use pip, create sym links to the site packages in brew
	brew install py3cairo

branch_name: refs/heads/master

extension: md

license_type: permissive


text:
# TwitterIDs

The program `TwitterConsole.exe` downloads the IDs of the followers of a certain Twitter user.

Thanks to **Bovey Qin**, who wrote the entire thing!

Here is his README:

    Unzip the zip file to an folder, which we refer as unzipped folder.

    The application is TwitterConsole.exe, a windows console application to get followers ids for a given screen name.

    Usage:
    TwitterConsole.exe [screenName]
    For example:

    TwitterConsole.exe billclinton

    What does the above command do :

    1. find billclinton's followerID, and save to a ID txt file in format of billclinton-followers-ids-635763954267209817.txt
    the number part is system clock ticks and can be different in each run.
    2. create a cursor file in name of billclinton_cursor, no file extension and the content is the last processed cursor.

    Whenever the application runs, it will try to read saved cursor, and starts from this one.  That is a feature to
    make the application  resumable. It  will never start from the firt one, but always pick up the last operation and
    contiue. It makes possible to get all followers  in an increamental maner.

    When this applicaiton runs for the same screen name multi-times, you need to combine all  different ID file.

    For example, we have run 3 time for billclinton and we got 3 ID file like below:
    billclinton-followers-ids-635763954267209817.txt
    billclinton-followers-ids-635763955428943858.txt
    billclinton-followers-ids-635763957891564183.txt

    Combine all above 3 file to get all follower's ID

    Configuration.

    There is a file called TwitterConsole.exe.config
    The only thing interesting is the following section
    <:appSettings>:
        <:add key="key1" value="o2tUMOkODE9hA5kvjjANZfUMq|NQAJiCJKtIAHHh1sB6SfoLnmXCnU6y0aBkxf9ShRyGz3J9os3e|3421384840-nMXEvElKMkJzdRMz6wc6WlkkaApmxGXRyy5VdVs|cMV6Ep7YyvX8sRJErJRECXluWvUBnd5MPrLJun4OiFP1a" />:
        <:add key="key2" value="RW1cM5XZxcYOxX6jfwvyiaQ5f|tMPSwTm8Nt2xORlJMhWpHOS6kuwykBAscJjuaPrMfOyuAhJox5|3421384840-6iMukx2bVGLrSbI2I5Z6pq6Yl6I8CUBXyugx4fo|Ssob8xneLwimSlRnNwSxHXAessidZtGr1qAYWJ67bhbNj" />:
        <:add key="key3" value="RW1cM5XZxcYOxX6jfwvyiaQ5f|tMPSwTm8Nt2xORlJMhWpHOS6kuwykBAscJjuaPrMfOyuAhJox5|3421384840-6iMukx2bVGLrSbI2I5Z6pq6Yl6I8CUBXyugx4fo|Ssob8xneLwimSlRnNwSxHXAessidZtGr1qAYWJ67bhbNj" />:
        <:add key="key4" value="vispVHANmFSw1yBjE4X87tZJg|QhTXU496wbn340FGWAnHQOdmChjAlqSnICmItBnhoyoebGyc1A|3421384840-ea70MplzaSuYX56fKLtmHdWJkaC1wVpFM6yvtRA|9uRU4TfgpwkR82fhQs7M5XX9p1Lwwd70ujUR8aGcFAJ5y" />:
      <:/appSettings>:

    Each key is an Twitter API App. It is extendable.
    If you need add another key,
     add key5, key6,key7 etc.

    Most powerful feature:

    You can run the application to different screen name in the same time.

    Open command line windows and navigate to the unzipped folder and run
    TwitterConsole.exe hillaryclinton

    then open another command line windows and navigate to the unzipped folder and run it again to target another people:
    TwitterConsole.exe joebidden

    you can repeat above step to start working on mulit people.

    My test is to work on 6 people for about 2 or 3 hours and get more than 2 million ids.

    all joebidden's followers is done. Don't run him again.
    When one's followers is fetched completely, there will some showing like
    All followers fetched. No more followers
    then you can close this window.

    In the zip file there are 6 cursor files, that meand I have started to work on 6 persons, but only joebidden is done.

    Other 5 persons is not complete. if you start run this applicaiton on any one of the remaining 5, you will
    continue from where I left.

    You need to divid persons among you 2. You can run the same person in different time at the same machine, but not
    the same time at different machine. One person has to stick to one machine. If you need to change person please share
    cursor file.

license: MIT


text:
---
title: Content buckets
lang: en
description: Which pages each content bucket of Ethereum.org contains
---

# Content buckets {#content-buckets}

As mentioned in our [Translation Program overview](/contributing/translation-program/), we use 'content buckets' within Crowdin to get the highest priority content released first. When you check out a language to translate, for example, [German](https://crowdin.com/project/ethereum-org/de) you'll see folders for each content bucket.

Below is a breakdown of the website pages each content bucket contains.

## 1) Homepage {#homepage}

- [Ethereum.org homepage](/)
- Main navbar
- Footer links

## 2) Essential pages {#essential-pages}

- [What is Ethereum?](/what-is-ethereum/)
- [What is ether (ETH)?](/eth/)
- [Get ETH](/get-eth/)
- [Wallets](/wallets/)
- [Find wallets](/wallets/find-wallet/)

## 3) Exploring {#exploring}

- [Non-fungible tokens (NFT)](/nft/)
- [Dapps](/dapps/)
- [Stablecoins](/stablecoins/)
- Template usecase

## 4) Use Ethereum pages {#use-ethereum-pages}

- [Decentralized autonomous organizations (DAOs)](/dao/)
- [Layer 2](/layer-2/)
- [Run a node](/run-a-node/)
- [Developers' Home](/developers/)
- [Developer learning tools](/developers/learning-tools/)
- [Developer local environment setup](/developers/local-environment/)
- [Language support](/languages/)

## 5) Use case pages {#use-case-pages}

- [Decentralized finance (DeFi)](/defi/)
- [Introduction to smart contracts](/smart-contracts/)
- [Decentralized identity)](/decentralized-identity/)
- [Decentralized social networks)](/social-networks/)
- [Decentralized science (DeSci))](/desci/)

## 6) Staking pages {#staking-pages}

- [Staking](/staking/)
- [Solo staking](/staking/solo/)
- [Pooled staking](/staking/pools/)
- [Staking as a service](/staking/saas/)
- [Staking deposit contract](/staking/deposit-contract/)
- [Staking withdrawals](/staking/withdrawals/)

## 7) Learn pages {#learn-pages}

- [Energy consumption](/energy-consumption/)
- [Governance](/governance/)
- [Ethereum security and scam prevention](/security/)
- [Blockchain bridges](/bridges/)
- [Web3](/web3/)
- [Zero-knowledge proofs](/zero-knowledge-proofs/)

## 8) Learn hub & guides {#learn-hub}

- [Learn hub](/learn/)
- [Ethereum guides](/guides/)
- [How to "create" an Ethereum account](/guides/how-to-create-an-ethereum-account/)
- [How to use a wallet](/guides/how-to-use-a-wallet/)
- [How to revoke smart contract access to your crypto funds](/guides/how-to-revoke-token-access/)
- [How to bridge tokens to layer 2](/guides/how-to-use-a-bridge/)
- [How to swap tokens](/guides/how-to-swap-tokens/)
- [Learning quizzes](/quizzes/)

## 9) Upgrades {#upgrades}

- [Ethereum roadmap](/roadmap/)
- [Ethereum vision](/roadmap/vision/)
- [The Beacon Chain](/roadmap/beacon-chain/)
- [The Merge](/roadmap/merge/)
- [How The Merge impacted ETH supply](/roadmap/merge/issuance/)
- [Ethereum Improvement Proposals (EIPs)](/eips/)
- [Scaling Ethereum](/roadmap/scaling/)
- [A more secure Ethereum](/roadmap/security/)
- [Improving user experience](/roadmap/user-experience/)
- [Future-proofing Ethereum](/roadmap/future-proofing/)
- [Danksharding](/roadmap/danksharding/)
- [Single slot finality](roadmap/single-slot-finality/)
- [Proposer-builder separation](/roadmap/pbs/)
- [Secret leader election](/roadmap/secret-leader-election/)
- [Account abstraction](/roadmap/account-abstraction/)
- [Verkle trees](/roadmap/verkle-trees/)
- [Statelessness, state expiry and history expiry](/roadmap/statelessness/)

## 10) Community pages {#community-pages}

- [Ethereum events](/community/events/)
- [How can I get involved?](/community/get-involved/)
- [Ethereum grants](/community/grants/)
- [Language resources](/community/language-resources/)
- [Online communities](/community/online/)
- [Ethereum support](/community/support/)
- [Community hub](/community/)

## 11) Foundational developer docs {#foundational-docs}

- [Overview](/developers/docs/)
- [Intro to Ethereum](/developers/docs/intro-to-ethereum/)
- [Intro to ether](/developers/docs/intro-to-ether/)
- [Intro to dapps](/developers/docs/dapps)
- [Web2 vs Web3](/developers/docs/web2-vs-web3/)
- [Accounts](/developers/docs/accounts/)
- [Transactions](/developers/docs/transactions/)
- [Blocks](/developers/docs/blocks/)
- [Ethereum virtual machine (EVM)](/developers/docs/evm/)
- [Opcodes for the EVM](/developers/docs/evm/opcodes/)
- [Gas](/developers/docs/gas/)
- [Networks](/developers/docs/networks/)
- Developer docs sidebar

## 12) Foundational docs - Nodes and clients {#nodes-and-clients}

- [Nodes and clients](/developers/docs/nodes-and-clients/)
- [Nodes as a service](/developers/docs/nodes-and-clients/nodes-as-a-service/)
- [Ethereum archive node](/developers/docs/nodes-and-clients/archive-nodes/)
- [Introduction to Ethereum bootnodes](/developers/docs/nodes-and-clients/bootnodes/)
- [Light clients](/developers/docs/nodes-and-clients/light-clients/)
- [Node architecture](/developers/docs/nodes-and-clients/node-architecture/)
- [Client diversity](/developers/docs/nodes-and-clients/client-diversity/)
- [Spin up your own Ethereum node](/developers/docs/nodes-and-clients/run-a-node/)

## 13) Foundational docs - Proof-of-Stake {#PoS}

- [Consensus mechanisms](/developers/docs/consensus-mechanisms/)
- [Proof-of-stake](/developers/docs/consensus-mechanisms/pos/)
- [Ethereum proof-of-stake attack and defense](/developers/docs/consensus-mechanisms/pos/attack-and-defense/)
- [Attestations](/developers/docs/consensus-mechanisms/pos/attestations/)
- [Block proposal](/developers/docs/consensus-mechanisms/pos/block-proposal/)
- [Proof-of-stake FAQs](/developers/docs/consensus-mechanisms/pos/faqs/)
- [Keys in proof-of-stake Ethereum](/developers/docs/consensus-mechanisms/pos/keys/)
- [Proof-of-stake rewards and penalties](/developers/docs/consensus-mechanisms/pos/rewards-and-penalties/)
- [Gasper](/developers/docs/consensus-mechanisms/pos/gasper/)
- [Weak subjectivity](/developers/docs/consensus-mechanisms/pos/weak-subjectivity/)

## 14) Foundational docs - Proof-of-Work {#PoW}

- [Proof-of-work](/developers/docs/consensus-mechanisms/pow/)
- [Mining](/developers/docs/consensus-mechanisms/pow/mining/)
- [Mining algorithms](/developers/docs/consensus-mechanisms/pow/mining-algorithms/)
- [Dagger-Hashimoto](/developers/docs/consensus-mechanisms/pow/mining-algorithms/dagger-hashimoto/)
- [Ethash](/developers/docs/consensus-mechanisms/pow/mining-algorithms/ethash/)

## 15) Ethereum stack developer docs {#ethereum-stack-docs}

- [Introduction to the Ethereum stack](/developers/docs/ethereum-stack/)
- [Deployment networks](/developers/docs/development-networks/)
- [Development frameworks](/developers/docs/frameworks/)
- [JavaScript APIs](/developers/docs/apis/javascript/)
- [Backend APIs](/developers/docs/apis/backend/)
- [JSON-RPC](/developers/docs/apis/json-rpc/)
- [Data and analytics](/developers/docs/data-and-analytics/)
- [Block explorers](/developers/docs/data-and-analytics/block-explorers/)
- [Storage](/developers/docs/storage/)
- [Integrated Development Environments (IDEs)](/developers/docs/ides/)
- [Programming languages](/developers/docs/programming-languages/)
- [Delphi](/developers/docs/programming-languages/delphi/)
- [.NET](/developers/docs/programming-languages/dot-net/)
- [Golang](/developers/docs/programming-languages/golang/)
- [Java](/developers/docs/programming-languages/java/)
- [JavaScript](/developers/docs/programming-languages/javascript/)
- [Python](/developers/docs/programming-languages/python/)
- [Rust](/developers/docs/programming-languages/rust/)
- [Ruby](/developers/docs/programming-languages/ruby/)
- [Dart](/developers/docs/programming-languages/dart/)

## 16) Smart contracts - Basics {#smart-contracts-basics}

- [Smart contracts](/developers/docs/smart-contracts/)
- [Smart contract languages](/developers/docs/smart-contracts/languages/)
- [Smart contract anatomy](/developers/docs/smart-contracts/anatomy/)
- [Smart contract libraries](/developers/docs/smart-contracts/libraries)
- [Compiling smart contracts](/developers/docs/smart-contracts/compiling)
- [Deploying smart contracts](/developers/docs/smart-contracts/deploying)
- [Smart contract security](/developers/docs/smart-contracts/security/)

## 17) Smart contracts - Advanced {#smart-contracts-advanced}

- [Testing smart contracts](/developers/docs/smart-contracts/testing)
- [Composability](/developers/docs/smart-contracts/composability/)
- [Formal verification of smart contracts](/developers/docs/smart-contracts/formal-verification/)
- [Verifying smart contracts](/developers/docs/smart-contracts/verifying/)
- [Upgrading smart contracts](/developers/docs/smart-contracts/upgrading/)

## 18) Whitepaper {#whitepaper}

- [Whitepaper](/whitepaper/)

## 19) Additional Learn pages {#learn-pages2}

- [History](/history/)
- [Glossary](/glossary/)
- [Zero-knowledge proofs](/zero-knowledge-proofs/)

## 20) Advanced developer docs {#advanced-docs}

- [Standards](/developers/docs/standards/)
- [Token standards](/developers/docs/standards/tokens/)
- [ERC-20](/developers/docs/standards/tokens/erc-20/)
- [ERC-721](/developers/docs/standards/tokens/erc-721/)
- [ERC-777](/developers/docs/standards/tokens/erc-777/)
- [ERC-1155](/developers/docs/standards/tokens/erc-1155/)
- [ERC-4626](/developers/docs/standards/tokens/erc-4626/)
- [Maximal extractable value (MEV)](/developers/docs/mev/)
- [Oracles](/developers/docs/oracles/)
- [Bridges](/developers/docs/bridges/)
- [Data availability](/developers/docs/data-availability/)

## 21) Advanced developer docs - Scaling {#scaling-docs}

- [Scaling](/developers/docs/scaling/)
- [Optimistic rollups](/developers/docs/scaling/optimistic-rollups/)
- [Zero-knowledge rollups](/developers/docs/scaling/zk-rollups/)
- [State channels](/developers/docs/scaling/state-channels)
- [Sidechains](/developers/docs/scaling/sidechains)
- [Plasma](/developers/docs/scaling/plasma)
- [Validium](/developers/docs/scaling/validium/)

## 22) Research documentation {#research-documentation}

- [Networking layer](/developers/docs/networking-layer/)
- [Patricia Merkle Trees](/developers/docs/data-structures-and-encoding/patricia-merkle-trie/)
- [Data structures and encoding](/developers/docs/data-structures-and-encoding/)
- [Recursive-length prefix (RLP) serialization](/developers/docs/data-structures-and-encoding/rlp/)
- [Network addresses](/developers/docs/networking-layer/network-addresses/)
- [Simple serialize](/developers/docs/data-structures-and-encoding/ssz/)
- [Mining algorithms](/developers/docs/consensus-mechanisms/pow/mining-algorithms/)
- [Dagger-Hashimoto](/developers/docs/consensus-mechanisms/pow/mining-algorithms/dagger-hashimoto/)
- [Ethash](/developers/docs/consensus-mechanisms/pow/mining-algorithms/ethash/)
- [Gasper](/developers/docs/consensus-mechanisms/pos/gasper/)
- [Weak subjectivity](/developers/docs/consensus-mechanisms/pos/weak-subjectivity/)
- [Web3 secret storage definition](/developers/docs/data-structures-and-encoding/web3-secret-storage/)

## 23) Miscellaneous {#miscellaneous}

- [About ethereum.org](/about/)
- [Enterprise Ethereum](/enterprise/)
- [Private Ethereum](/enterprise/private-ethereum/)
- [Brand assets](/assets/)
- [About the Ethereum Foundation](/foundation/)

## 24) Contributing {#contributing}

- [Contributing to ethereum.org](/contributing/)
- [Adding developer tools](/contributing/adding-developer-tools/)
- [Adding exchanges](/contributing/adding-exchanges/)
- [Adding glossary terms](/contributing/adding-glossary-terms/)
- [Adding layer 2s](/contributing/adding-layer-2s/)
- [Adding products](/contributing/adding-products/)
- [Adding staking products](/contributing/adding-staking-products/)
- [Adding content resources](/contributing/content-resources/)
- [Adding DeSci projects](/contributing/adding-desci-projects/)
- [Adding wallets](/contributing/adding-wallets/)
- [Design principles](/contributing/design-principles/)
- [Translation Program](/contributing/translation-program/)
- [Translation guide](/contributing/translation-program/translation-guide/)
- [Translator acknowledgements](/contributing/translation-program/)
- [Our translators](/contributing/translation-program/contributors/)
- [Translation FAQ](/contributing/translation-program/faq/)
- [How to translate](/contributing/translation-program/how-to-translate/)
- [Translation Program mission and vision](/contributing/translation-program/mission-and-vision/)
- [Translator resources](/contributing/translation-program/resources/)

## 25) Developer tutorials 1 {#tutorials-1}

- [Calling a smart contract from JavaScript](/developers/tutorials/calling-a-smart-contract-from-javascript/)
- [How to write & deploy an NFT (Part 1/3 of NFT tutorial series)](/developers/tutorials/how-to-write-and-deploy-an-nft/)
- [How to mint an NFT (Part 2/3 of NFT tutorial series)](/developers/tutorials/how-to-mint-an-nft/)
- [How to view your NFT in your wallet (Part 3/3 of NFT tutorial series)](/developers/tutorials/how-to-view-nft-in-metamask/)
- [Transfers and approval of ERC-20 tokens from a solidity smart contract](/developers/tutorials/transfers-and-approval-of-erc-20-tokens-from-a-solidity-smart-contract/)
- [Understand the ERC-20 token smart contract](/developers/tutorials/understand-the-erc-20-token-smart-contract/)
- [Uniswap-v2 contract walkthrough](/developers/tutorials/uniswap-v2-annotated-code/)
- Submit a tutorial

## 26) Developer tutorials 2 {#tutorials-2}

- [A Python developer's introduction to Ethereum](/developers/tutorials/a-developers-guide-to-ethereum-part-one/)
- [Downsizing contracts to fight the contract size limit](/developers/tutorials/downsizing-contracts-to-fight-the-contract-size-limit/)
- [Hello world smart contract for beginners](/developers/tutorials/hello-world-smart-contract/)
- [How to turn your Raspberry Pi 4 into a node just by flashing the MicroSD card](/developers/tutorials/run-node-raspberry-pi/)
- [Interact with other contracts from Solidity](/developers/tutorials/interact-with-other-contracts-from-solidity/)
- [NFT Minter tutorial](/developers/tutorials/nft-minter/)
- [Reverse engineering a contract](/developers/tutorials/reverse-engineering-a-contract/)
- [Sending tokens using ethers.js](/developers/tutorials/send-token-etherjs/)
- [The Graph: Fixing Web3 data querying](/developers/tutorials/the-graph-fixing-web3-data-querying/)
- [Transfers and approval of ERC-20 tokens from a Solidity smart contract](/developers/tutorials/transfers-and-approval-of-erc-20-tokens-from-a-solidity-smart-contract/)

## 27) Developer tutorials 3 {#tutorials-3}

- [A guide to smart contract security tools](/developers/tutorials/guide-to-smart-contract-security-tools/)
- [All you can cache](/developers/tutorials/all-you-can-cache/)
- [ERC-20 contract walkthrough](/developers/tutorials/erc20-annotated-code/)
- [ERC-20 with safety rails](/developers/tutorials/erc20-with-safety-rails/)
- [Getting Started with Ethereum Development](/developers/tutorials/getting-started-with-ethereum-development-using-alchemy/)
- [How to mock Solidity smart contracts for testing](/developers/tutorials/how-to-mock-solidity-contracts-for-testing/)
- [Kickstart your dapp frontend development with create-eth-app](/developers/tutorials/kickstart-your-dapp-frontend-development-wth-create-eth-app/)
- [Logging data from smart contracts with events](/developers/tutorials/logging-events-smart-contracts/)
- [Merkle proofs for offline data integrity](/developers/tutorials/merkle-proofs-for-offline-data-integrity/)
- [Sending transactions using Web3](/developers/tutorials/sending-transactions-using-web3-and-alchemy/)
- [Smart contract security checklist](/developers/tutorials/secure-development-workflow/)
- [Testing simple smart contract with Waffle library](/developers/tutorials/waffle-test-simple-smart-contract/)
- [Vyper ERC-721 contract walkthrough](/developers/tutorials/erc-721-vyper-annotated-code/)

## 28) Developer tutorials 4 {#tutorials-4}

- [Create and deploy a DeFi app](/developers/tutorials/create-and-deploy-a-defi-app/)
- [Deploying your first smart contract](/developers/tutorials/deploying-your-first-smart-contract/)
- [How to implement an ERC-721 market](/developers/tutorials/how-to-implement-an-erc721-market/)
- [How to set up Tellor as your oracle](/developers/tutorials/how-to-use-tellor-as-your-oracle/)
- [How to use Echidna to test smart contracts](/developers/tutorials/how-to-use-echidna-to-test-smart-contracts/)
- [How to use Manticore to find bugs in smart contracts](/developers/tutorials/how-to-use-manticore-to-find-smart-contract-bugs/)
- [How to use Slither to find smart contract bugs](/developers/tutorials/how-to-use-slither-to-find-smart-contract-bugs/)
- [Learn foundational Ethereum topics with SQL](/developers/tutorials/learn-foundational-ethereum-topics-with-sql/)
- [Monitoring Geth with InfluxDB and Grafana](/developers/tutorials/monitoring-geth-with-influxdb-and-grafana/)
- [Optimism standard bridge contract walkthrough](/developers/tutorials/optimism-std-bridge-annotated-code/)
- [Set up web3.js to use the Ethereum blockchain in JavaScript](/developers/tutorials/set-up-web3js-to-use-ethereum-in-javascript/)
- [Short ABIs for calldata optimization](/developers/tutorials/short-abi/)
- [Smart contract security guidelines](/developers/tutorials/smart-contract-security-guidelines/)
- [Solidity and Truffle continuous integration setup](/developers/tutorials/solidity-and-truffle-continuous-integration-setup/)
- [Testing ERC-20 tokens with Waffle](/developers/tutorials/testing-erc-20-tokens-with-waffle/)
- [Token integration checklist](/developers/tutorials/token-integration-checklist/)
- [Using WebSockets](/developers/tutorials/using-websockets/)
- [Waffle: Dynamic mocking and testing contract calls](/developers/tutorials/waffle-dynamic-mocking-and-testing-calls/)
- [Waffle say hello world tutorial with Hardhat and ethers](/developers/tutorials/waffle-say-hello-world-with-hardhat-and-ethers/)

repository: ethereum/ethereum-org-website

branch_name: refs/heads/dev

gha_language: Markdown

gha_license_id: MIT

repo_name: ethereum/ethereum-org-website


text:
# Image Captioning

![readme](./images/readme.png)

This project uses CNNs and LSTMs to automatically generate captions from images.

### Data

The Microsoft Common Objects in Context (MS COCO) dataset is a large-scale dataset for scene understanding. The dataset is commonly used to train and benchmark object detection, segmentation, and captioning algorithms.

![coco-examples](./images/coco-examples.jpg)

You can read more about the dataset on the [website](http://cocodataset.org/#home) or in the [research paper](https://arxiv.org/pdf/1405.0312.pdf).

To obtain and explore the dataset, you can use the [COCO API](https://github.com/cocodataset/cocoapi).

### Model Architecture

The core architecture is an encoder-decoder, where the encoder is a pretrained ResNet CNN on ImageNet, and the decoder is a basic LSTM.

![encoder-decoder](./images/encoder-decoder.png)

# Instructions

1. Clone this repo: https://github.com/cocodataset/cocoapi
```
git clone https://github.com/cocodataset/cocoapi.git
```

2. Setup the coco API (also described in the readme [here](https://github.com/cocodataset/cocoapi))
```
cd cocoapi/PythonAPI
make
cd ..
```

3. Download some specific data from here: http://cocodataset.org/#download (described below)

* Under **Annotations**, download:
  * **2014 Train/Val annotations [241MB]** (extract captions_train2014.json and captions_val2014.json, and place at locations cocoapi/annotations/captions_train2014.json and cocoapi/annotations/captions_val2014.json, respectively)
  * **2014 Testing Image info [1MB]** (extract image_info_test2014.json and place at location cocoapi/annotations/image_info_test2014.json)

* Under **Images**, download:
  * **2014 Train images [83K/13GB]** (extract the train2014 folder and place at location cocoapi/images/train2014/)
  * **2014 Val images [41K/6GB]** (extract the val2014 folder and place at location cocoapi/images/val2014/)
  * **2014 Test images [41K/6GB]** (extract the test2014 folder and place at location cocoapi/images/test2014/)

4. The project is structured as a series of Jupyter notebooks that are designed to be completed in sequential order (`0_Dataset.ipynb, 1_Preliminaries.ipynb, 2_Training.ipynb, 3_Inference.ipynb`).

____________________________________________________________________

This is a project from Udacity's computer vision nanodegree

LICENSE: This project is licensed under the terms of the MIT license.


text:
# FiveM - heli by mraes
Version 1.3 - 2017/06/09

Instructions:
 * Download resource
 * Add heli entry to your citmp-server.yml AutoStartResources
 * Press the the E key to switch to the HeliCam
 * Whilst in Helicam: move mouse to rotate cam, scroll to zoom and use the RIGHT MOUSE BUTTON to switch between normal, nightvision and thermal vision
 * Whilst in Helicam: when hovering over a vehicle and close enough to it, you can see vehicle info. If you can see this, press SPACE to lock camera onto the vehicle. SPACE again to unlock camera.
 * Press the X key to rappel from the helicopter
 * Press the G key as helicopter pilot to switch on the spotlight (this is synced across the network to all players)

 Changelog:
 * 2017/06/08: Added smoothing to camera movement.
 * 2017/06/08: Added Nightvision and Thermal Vision.
 * 2017/06/09: Made camera movement smoother when zoomed in.
 * 2017/06/12: Added rappel, spotlight and showing of vehicle information + lock onto vehicle. Changed some keys. Added screen effect overlay to make it look like cam feed.

branch_name: refs/heads/main


text:
# Modules

Tasks and plans are packaged in Puppet modules, so you can install them as you would any module and manage them with a Puppetfile.

## Packaged modules

Bolt is packaged with a collection of useful modules to support common workflows.

This list of packaged modules is available in a [Puppetfile](https://github.com/puppetlabs/bolt/blob/master/Puppetfile) in the Bolt repository. The modules and supporting documentation are publicly available on the [Puppet Forge](https://forge.puppet.com/).

**Note:** If you installed Bolt as a Ruby Gem, make sure you have installed these core modules.

### Modules with useful task and plan content

-   [package](https://forge.puppet.com/puppetlabs/package): Install, uninstall, update, and check the status of packages.
-   [service](https://forge.puppet.com/puppetlabs/service): Manage and inspect the state of services.
-   [puppet_conf](https://forge.puppet.com/puppetlabs/puppet_conf): Inspect and change the configuration options in the `puppet.conf` file.
-   [facts](https://forge.puppet.com/puppetlabs/facts): Retrieve facts from specified targets.
-   [puppet_agent](https://forge.puppet.com/puppetlabs/puppet_agent): Install Puppet Agent package.
-   [reboot](https://forge.puppet.com/puppetlabs/reboot): Manage system reboots.

### Core Puppet providers

-   [augeas_core](https://forge.puppet.com/puppetlabs/augeas_core): Manage configuration files using Augeas.
-   [host_core](https://forge.puppet.com/puppetlabs/host_core): Manage host entries in a hosts file.
-   [scheduled_task](https://forge.puppet.com/puppetlabs/scheduled_task): Provider capable of using the Version 2 Windows API for task management.
-   [sshkeys_core](https://forge.puppet.com/puppetlabs/sshkeys_core): Manage `SSH`, `authorized_keys`, and `ssh_known_hosts` files.
-   [zfs_core](https://forge.puppet.com/puppetlabs/zfs_core): Manage `zfs` and `zpool` resources.
-   [cron_core](https://forge.puppet.com/puppetlabs/cron_core): Install and manage `cron` resources.
-   [mount_core](https://forge.puppet.com/puppetlabs/mount_core): Manage mounted filesystems and mount tables.
-   [selinux_core](https://forge.puppet.com/puppetlabs/selinux_core): Manage Security-Enhanced Linux.
-   [yumrepo_core](https://forge.puppet.com/puppetlabs/yumrepo_core): Manage client yum repo configurations by parsing INI configuration files.
-   [zone_core](https://forge.puppet.com/puppetlabs/zone_core): Manage Solaris zone resources.

### Bolt-specific modules that are not available on the Forge

-   [aggregate](https://github.com/puppetlabs/bolt/tree/master/modules/aggregate): Aggregate task, script or command results.
-   [canary](https://github.com/puppetlabs/bolt/tree/master/modules/canary): Run action against a small number of targets and only if it succeeds will it run on the rest.
-   [puppetdb_fact](https://github.com/puppetlabs/bolt/tree/master/modules/puppetdb_fact): Collect facts for the specified targets from the configured PuppetDB connection and stores the collected facts on the targets.

**Tip:** To override a packaged module with another version, download the version you want and configure your modulepath to point to it.

### Modules that contain helper code for writing your own tasks

-   [ruby_task_helper](https://forge.puppet.com/puppetlabs/ruby_task_helper): A helper for writing tasks in Ruby.
-   [python_task_helper](https://forge.puppet.com/puppetlabs/python_task_helper): A helper for writing tasks in Python.

**Tip:** To override a packaged module with another version, download the version you want and configure your modulepath to point to it.

**Related information**

[Puppetfile example](https://github.com/puppetlabs/bolt/blob/master/Puppetfile)

[Puppet Forge](https://forge.puppet.com/)

[Bolt configuration options](bolt_configuration_reference.md)

## Configure Bolt to download and install modules

Before you can use Bolt to install modules, you must first create a Puppetfile. A Puppetfile is a formatted text file that contains a list of modules and their versions. It can include modules from the Forge or a Git repository.

For modules that require Ruby Gems, see [Install Gems with Bolt packages](bolt_installing.md#)

For more details about specifying modules in a Puppetfile, see the [Puppetfile documentation](https://puppet.com/docs/pe/latest/puppetfile.html).

1.  Create a file named Puppetfile and store it in the [Bolt project directory](bolt_project_directories.md#).
1.  Open the Puppetfile in a text editor and add the modules and versions that you want to install. If the modules have dependencies, list those as well.
    ```
    # Modules from the Puppet Forge.
    mod 'puppetlabs-apache', '4.1.0'
    mod 'puppetlabs-postgresql', '5.12.0'
    mod 'puppetlabs-puppet_conf', '0.3.0'
    mod 'puppetlabs-facts', '0.3.1'

    # Modules from a Git repository.
    mod 'puppetlabs-haproxy', git: 'https://github.com/puppetlabs/puppetlabs-haproxy.git', ref: 'master'
    ```
1.  Add any task or plan modules stored locally in `modules/` to the list. If these modules are not listed in the Puppetfile, they are deleted.
    ```
    mod 'my_awesome_module', local: true
    ```
    Install any modules that you don't want to manage with the Puppetfile to a different subdirectory. Conventionally, `site-modules` is used for this purpose and is included on the default `modulepath`.
1.  From a terminal, install the modules listed in the Puppetfile: `bolt puppetfile install`. Bolt installs modules to the first directory in the modulepath setting. By default, this is the `modules/` subdirectory inside the Bolt project directory. To override this location, update the modulepath setting in the Bolt config file.

repository: ochuerta/bolt

repo_name: ochuerta/bolt


text:
# portworx-install
This document provides instructions to install portworx on VPC Gen2 managed OpenShift Cluster on IBM Cloud (Similar steps can be followed for VPC IKS cluster).

The steps below are created following the ibm cloud docs below. Refer to these for more info and detailed steps.
- [Portworx SDS with RH Openshift on IBM Cloud](https://cloud.ibm.com/docs/openshift?topic=openshift-portworx)
- [Portworx SDS with IKS on IBM Cloud](https://cloud.ibm.com/docs/containers?topic=containers-portworx)

 > NOTE: The below steps are only created for reference. The portworx setup for production usecases might involve additional steps and configuration.

## Prerequisites
- A [Managed openshift cluster (ROKS) on IBM Cloud](https://cloud.ibm.com/docs/openshift?topic=openshift-clusters) to install portworx (atleast 3 workers). Refer
    > [About Portworx](https://cloud.ibm.com/docs/openshift?topic=openshift-portworx#about-portworx) for the preferred worker node flavor and type.

    > NOTE: ROKS cluster that has both public and private endpoints is used in this setup. Some steps might vary if private-only cluster is used. Refer [Managed OpenShift on IBM Cloud VPC cluster service architecture](https://cloud.ibm.com/docs/openshift?topic=openshift-service-arch#service-architecture_vpc) for more details on the architecture of VPC Clusters

    As of now, on VPC as baremetal servers are not supported, SDS worker node options are not available. For Non-SDS worker nodes on Virtual machines, `b2.16x64` or better is preferred for portworx.
- [ibmcloud cli](https://cloud.ibm.com/docs/cli?topic=cli-install-ibmcloud-cli#shell_install)

## Create the volumes
- Refer to Portworx min reqs - https://docs.portworx.com/start-here-installation/
- Provision block storage volumes for VPC. If the cluster is multi-zone and if portworx storage layer should be spanned across all zones, create block volumes in all zones.
    > Refer [Create a stand-alone block storage volume](https://cloud.ibm.com/docs/vpc?topic=vpc-creating-block-storage#create-standalone-vol)
- Choose a desired volume profile. Choose IOPS tier and enter size. Eg: 128 GB, 10 IOPS/GB. A custom IOPS profile can also be used.
    > Refer [Custom IOPS profile for VPC Block Volumes](https://cloud.ibm.com/docs/vpc?topic=vpc-block-storage-profiles#custom)

> NOTE: A volume can only be attached to one worker node. Create mutiple volumes for all/some worker nodes.

## Attach the volumes to workers
> Refer: [VPC: Adding raw Block Storage to VPC worker nodes](https://cloud.ibm.com/docs/openshift?topic=openshift-utilities#vpc_api_attach)

- Login into IBM Cloud account from the cli
    ```
    ▶ ibmcloud login
    ```
    Use `ibmcloud login --sso` to login with federated ID.
    Other ways - [ibmcloud login](https://cloud.ibm.com/docs/cli?topic=cli-ibmcloud_cli#ibmcloud_login)

- Get cluster ID
    ```
    ▶ ibmcloud ks cluster get --cluster <cluster-name>
    ```
- Get worker node IDs
    ```
    ▶ ibmcloud oc worker ls --cluster <cluster-name>
    ```
- Get volume IDs
    ```
    ▶ ibmcloud is volumes | grep <volume-name>
    ```
- Get IAM token
    ```
    ▶ ibmcloud iam oauth-tokens
    ```
- Make an curl `POST` request to attach the created volume to a worker node
Repeat this step for all volumes. Make sure to attach the volume and worker form same zone or datacenter

    ```
    ▶ curl -X POST -H "Authorization: Bearer <token>" "https://us-south.containers.cloud.ibm.com/v2/storage/vpc/createAttachment?cluster=<cluster-id>&worker=<worker-id>&volumeID=<volume-id>"
    ```
    An output like below will be observed
    ```
    {"id":"0727-b5965eb8-f7ac-4845-9ac9-5413a33aebc2","volume":{"name":"pwx-dal2-vol","id":"r006-394d474d-d5ec-47c2-8955-494c58ca880c"},"device":{"id":""},"name":"volume-attachment","status":"attaching","type":"data"}
    ```

## Setup external etcd database
> NOTE: This setup is using an external etcd db. A portworx KVDB can be used as well. But, if an automatically setup KVDB from portworx install is used, the metadata is stored inside the cluster. Refer [Setting up a key-value store for Portworx metadata](https://cloud.ibm.com/docs/openshift?topic=openshift-portworx#portworx_database)
- Provision IBM Cloud Databases for etcd from the IBM Cloud catalog
- Refer [Setting up a Databases for etcd service instance](https://cloud.ibm.com/docs/openshift?topic=openshift-portworx#databases-for-etcd) for the configuration
- Create etcd secret that stores username, pwd, and cert info
    ```yaml
    apiVersion: v1
    kind: Secret
    metadata:
    name: pwx-etcd-certs
    namespace: kube-system
    type: Opaque
    data:
     ca.pem:
     username:
     password:
    ```
    ```
    ▶ oc create -f pwx-etcd-secret.yaml
    ```

## Encryption with KeyProtect
<details>
<summary> (Click to Expand) Setup portworx to use IBM Key Protect for encryption </summary>

Install can be configured to enable encryption of volumes with IBM Key Protect (KP) during provisioning or it can be enabled later.
In either case, a KeyProtect service instance and a root key should be created.

Follow the steps 1 through 11 [Setup KeyProtect Instance for volume encryption with Portworx](https://cloud.ibm.com/docs/openshift?topic=openshift-portworx#setup_encryption) and proceed to [Portworx install](#install-portworx) to enable encryption using IBM KP during provisioning

### Enable encryption on existing portworx setup
If encryption with IBM KP has not been enabled during provisioning, it can be setup with an existing Portworx setup on OpenShift cluster by editing the `portworx` daemonset. Step 12 from [Setup KeyProtect Instance for volume encryption with Portworx](https://cloud.ibm.com/docs/openshift?topic=openshift-portworx#setup_encryption)

```
▶ oc edit daemonset portworx -n kube-system
```
Replace the `secret_type` argument value to be `ibm-kp` from the `containers` spec as below:

```
containers:
- args:
  - -A
  - -f
  - -k
  - etcd:<REDACTED>
  - -c
  - px-storage-cluster
  - -secret_type
  - ibm-kp
  - -userpwd
  - $(ETCD_USERNAME):$(ETCD_PASSWORD)
  - -ca
  - /etc/pwx/etcdcerts/ca.pem
  - -r
  - "17001"
  - -x
  - kubernetes
```
Portworx pods will be restarted automatically after the daemonset is edited

View `config.json` from one of the portworx pods to verify if the change is made. The `"secret_type": "ibm-kp"` should be added in the secret section

```
▶ oc exec -it portworx-r4bdw  -n kube-system bash
[root@kube-bs6tcs3d0v6dus9f0n2g-<cluster>-default-00000307 /]# cd etc/pwx
[root@kube-bs6tcs3d0v6dus9f0n2g-<cluster>-default-00000307 pwx]# cat config.json
{
    "alertingurl": "",
    "cafile": "/etc/pwx/etcdcerts/ca.pem",
    "clusterid": "px-storage-cluster",
    "dataiface": "",
    "kvdb": [
        "etcd:<REDACTED>"
    ],
    "mgtiface": "",
    "password": "<REDACTED>",
    "scheduler": "kubernetes",
    "secret": {
        "cluster_secret_key": "",
        "secret_type": "ibm-kp"
    },
    "storage": {
        "cache": [],
        "devices": [
            "/dev/vdd"
        ],
        "journal_dev": "",
        "kvdb_dev": "",
        "max_storage_nodes_per_zone": 0,
        "rt_opts": {},
        "system_metadata_dev": ""
    },
    "username": "<REDACTED>",
    "version": "1.0"
}
```
</details>

After portworx is setup for encryption using KP, check the section [Enable per volume encryption](#enable-per-volume-encryption)

## Install portworx
### Provisioning
> Refer [Installing Portworx in your cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-portworx#install_portworx)
- Provision portworx service from IBM Cloud Catalog
- Choose the desired plan (DR Plan provides some additional capabilities and also allows backup to IBM COS - See details [here](https://cloud.ibm.com/docs/openshift?topic=openshift-portworx#px-dr)
- Fill in all the details. For guidance, [Install Portworx](https://cloud.ibm.com/docs/openshift?topic=openshift-portworx#install_portworx)
  For quick dev setup, you can proceed without encryption and leave the volume encryption key to default value
    > For volume encryption, refer - [Setting up volume encryption with IBM Key Protect](https://cloud.ibm.com/docs/openshift?topic=openshift-portworx#encrypt_volumes).

    To choose **Portworx secret store type**, see below info from ibm cloud docs

    >   Kubernetes Secret: Choose this option if you want to store your own custom key to encrypt your volumes in a Kubernetes Secret in your cluster. The secret must not be present before you install Portworx. You can create the secret after you install Portworx. For more information, see the Portworx documentation.

    > IBM Key Protect: Choose this option if you want to use root keys in IBM Key Protect to encrypt your volumes. Make sure that you follow the instructions to create your IBM Key Protect service instance, and to store the credentials for how to access your service instance in a Kubernetes secret in the portworx project before you install Portworx.

- Enter [iam user apikey](https://cloud.ibm.com/docs/account?topic=account-userapikey#create_user_key); select the resource group, list of clusters that the user has access to are shown. Choose the cluster to install portworx on

### Verify Portworx Install
- Connect to the cluster
    ```
    ▶ ibmcloud oc cluster config -c <cluster-name> --admin
    ```
> Refer [Connecting to the cluster from the CLI](https://cloud.ibm.com/docs/openshift?topic=openshift-access_cluster#access_oc_cli)

- Verify install
    ```
    ▶ helm ls -A
    NAME    	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART          	APP VERSION
    portworx	default  	1       	2020-07-17 14:38:16.543872499 +0000 UTC	deployed	portworx-1.0.18	2.5.1
    ```

    ```
    ▶ oc get pods -n kube-system | grep 'portworx\|stork'

    portworx-api-d94zv                         1/1       Running     0          56m
    portworx-api-nbg88                         1/1       Running     0          56m
    portworx-api-s7g2t                         1/1       Running     0          56m
    portworx-pvc-controller-64d8578f8b-7fkv5   1/1       Running     0          56m
    portworx-pvc-controller-64d8578f8b-jxcr6   1/1       Running     0          56m
    portworx-pvc-controller-64d8578f8b-q8cbb   1/1       Running     0          56m
    portworx-r6j54                             1/1       Running     0          56m
    portworx-tj6vt                             1/1       Running     0          56m
    portworx-vdvjd                             1/1       Running     0          56m
    stork-6dd4b6cd97-27jdr                     1/1       Running     1          56m
    stork-6dd4b6cd97-56gsp                     1/1       Running     0          56m
    stork-6dd4b6cd97-nzdlv                     1/1       Running     0          56m
    stork-scheduler-7b444f46dc-htwkw           1/1       Running     0          56m
    stork-scheduler-7b444f46dc-sqhfs           1/1       Running     0          56m
    stork-scheduler-7b444f46dc-x6c5m           1/1       Running     0          56m
    ```
    > NOTE: It may take some time for the pods to reach to a `Running` state if the other portworx related pods are not Ready yet. If there are errors in other pods, you might have to check logs or use the below command to check status

- Check portworx storage cluster status.
  Exec into the portworx pods using the command below
  ```
  ▶ kubectl exec portworx-r6j54 -it -n kube-system -- /opt/pwx/bin/pxctl status

	Status: PX is operational
	License: PX-Enterprise IBM Cloud DR (expires in 1356 days)
	Node ID: <id>
		IP: <ip1>
		Local Storage Pool: 1 pool
		POOL	IO_PRIORITY	RAID_LEVEL	USABLE	USED	STATUS	ZONE		REGION
		0	LOW		raid0		128 GiB	8.4 GiB	Online	us-south-1	us-south
		Local Storage Devices: 1 device
		Device	Path		Media Type		Size		Last-Scan
		0:1	/dev/vdd	STORAGE_MEDIUM_MAGNETIC	128 GiB		17 Jul 20 14:40 UTC
		total			-			128 GiB
		Cache Devices:
		No cache devices
	Cluster Summary
		Cluster ID: px-storage-cluster
		Cluster UUID: <pwx-cluster-id>
		Scheduler: kubernetes
		Nodes: 3 node(s) with storage (3 online)
		IP	ID	SchedulerNodeName StorageNode Used Capacity  Status StorageStatus	Version		Kernel		OS
		<ip1>	<node>	<ip1>		Yes		8.4 GiB	128 GiB		Online	Up		<IP_ADDRESS>-176ddb7	3.10.0-1127.13.1.el7.x86_64	Red Hat
		<ip2>	<node>	<ip2>		Yes		8.4 GiB	128 GiB		Online	Up		<IP_ADDRESS>-176ddb7	3.10.0-1127.13.1.el7.x86_64	Red Hat
		<ip3>	<node>	<ip3>		Yes		8.4 GiB	128 GiB		Online	Up (This node)	<IP_ADDRESS>-176ddb7	3.10.0-1127.13.1.el7.x86_64	Red Hat
		Warnings:
			 WARNING: Persistent journald logging is not enabled on this node.
	Global Storage Pool
		Total Used    	:  25 GiB
		Total Capacity	:  384 GiB
  ```

- Check portworx I/O classification that was assigned to the disks that are part of the Portworx cluster
    ```
    ▶ oc exec -it portworx-r6j54 -n kube-system -- /opt/pwx/bin/pxctl cluster provision-status
    NODE	NODE STATUS	POOL POOL STATUS	IO_PRIORITY	SIZE	AVAILABLE	USED	PROVISIONED	ZONE		REGIORACK
    <node>	Up		0  ( pool )	Online		LOW		128 GiB	 118 GiB		10 GiB	0 GiB		us-south-1	us-south	default
    <node>	Up		0  ( pool )	Online		LOW		128 GiB	 118 GiB		10 GiB	0 GiB		us-south-3	us-south	default
    <node>	Up		0  ( pool )	Online		LOW		128 GiB	 118 GiB		10 GiB	0 GiB		us-south-2	us-south	default
    ```
- List storageclasses setup by default from portworx install
    ```
    ▶ oc get storageclasses | grep portworx
    portworx-db-sc                          kubernetes.io/portworx-volume   30m
    portworx-db2-sc                         kubernetes.io/portworx-volume   30m
    portworx-null-sc                        kubernetes.io/portworx-volume   30m
    portworx-shared-sc                      kubernetes.io/portworx-volume   30m
    ```
> Once portworx is installed and the storage cluster is formed successfully, available storage classes can be used or custom SCs can be created. Peristence can be obtained by use of PVCs similar to how its done on k8s or openshift in general

## Validate using a sample pod

> Refer [Creating Portworx Volumes](https://cloud.ibm.com/docs/openshift?topic=openshift-portworx#add_portworx_storage)

Use dynamic provisioning to automatically create Portworx volume by creating a PVC and, optionally a StorageClass.
```
▶ mkdir -p ./portworx
```
> NOTE: One of the available storage classes can also be used in the PVC without the need of creating a custom storage class
```yaml
▶ cat <<EOF > ./portworx/custom-sc.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: pwx-custom-random-sc
provisioner: kubernetes.io/portworx-volume
parameters:
  repl: "1"
  io_profile: "random"
EOF
````

```yaml
▶ cat <<EOF > ./portworx/pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pwx-test-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: pwx-custom-random-sc
EOF
````

```yaml
▶ cat <<EOF > ./portworx/pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: test
  labels:
    name: test
spec:
  containers:
  - name: test
    image: alpine:3.8
    command: ["/bin/sh"]
    args: ["-c", "mkdir -p /usr/share/test; while true; do date >> /usr/share/test/date.log; sleep 10; done"]
    volumeMounts:
      - name: datelog
        mountPath: /usr/share/test
  volumes:
    - name: datelog
      persistentVolumeClaim:
        claimName: pwx-test-pvc

EOF
```
The test pod above, writes current time and date to `/usr/share/test/date.log` and the directory is mounted as a persistent volume

```sh
▶ oc create -f ./portworx/custom-sc.yaml
▶ oc create -f ./portworx/pvc.yaml
▶ oc create -f ./portworx/pod.yaml
```

Shell into the pod and check the contents being written to the file
```
▶ oc exec -it test sh

/ # tail -f /usr/share/test/date.log
Mon Jul 20 05:01:19 UTC 2020
Mon Jul 20 05:01:29 UTC 2020
Mon Jul 20 05:01:39 UTC 2020
Mon Jul 20 05:01:49 UTC 2020
Mon Jul 20 05:01:59 UTC 2020
Mon Jul 20 05:02:09 UTC 2020
```

To check persistence of the log file, delete the pod and create a new pod that uses the same pvc

```
▶ oc delete -f ./portworx/pod.yaml
```

```yaml
▶ cat <<EOF > ./portworx/test-persistence-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: test-persistence
  labels:
    name: test
spec:
  containers:
  - name: test-persistence
    image: alpine:3.8
    command: ["/bin/sh"]
    args: ["-c", "cat /data/date.log"]
    volumeMounts:
      - name: datelog
        mountPath: /data
  volumes:
    - name: datelog
      persistentVolumeClaim:
        claimName: pwx-test-pvc
EOF
```
This pod uses the same pvc and prints the contents(generated from previous pod) of the log file saved in the volume.

```
▶ oc create -f ./portworx/test-persistence-pod.yaml
```
```
▶ oc logs test-persistence
```
The pod logs will display the contents of file with all the log entries saved until previous pod has been deleted. The `date.log` file is persisted in the volume and hence the contents can be retrieved from this new pod by pointing it to the pvc the earlier pod is backed by.

Delete the resources created for validation

```
▶ oc delete -f ./portworx/test-persistence-pod.yaml
▶ oc delete -f ./portworx/custom-sc.yaml
▶ oc delete -f ./portworx/pvc.yaml
```

## Enable per volume encryption

There are two ways to enable per volume encryption
<details>
<summary>With Storage Class </summary>

- Create secure storage class with `secure` parameter set to `true`

  ```yaml
  ▶ cat <<EOF > ./portworx/custom-secure-sc.yaml
  kind: StorageClass
  apiVersion: storage.k8s.io/v1
  metadata:
    name: pwx-custom-random-secure-sc
  provisioner: kubernetes.io/portworx-volume
  parameters:
    repl: "1"
    io_profile: "random"
    secure: "true"
  EOF
  ```

  ```
  ▶ oc create -f ./portworx/custom-secure-sc.yaml
  ```

- Create a PVC to use the secure storage class. The volumes created by PVCs that use the secure storage class will be encrypted

  ```yaml
  ▶ cat <<EOF > ./portworx/pvc-secured-through-sc.yaml
  kind: PersistentVolumeClaim
  apiVersion: v1
  metadata:
    name: pwx-test-pvc-secured-through-sc
  spec:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 1Gi
    storageClassName: pwx-custom-random-secure-sc
  EOF
  ````

  ```
  ▶ oc create -f ./portworx/pvc-secured-through-sc.yaml
  ```
</details>

<details>
<summary>With PVC </summary>

### Using a PVC and specifying the annonation to create a secure volume

If enabling encryption with the storageclass is not preferred, you can also enable it during PVC creation using
```
annotations:
 px/secure: "true"
```
- Create PVC with secure annotation
  ```yaml
  ▶ cat <<EOF > ./portworx/pvc-secure.yaml
  kind: PersistentVolumeClaim
  apiVersion: v1
  metadata:
    name: pwx-test-pvc-secure
    annotations:
     px/secure: "true"
  spec:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 1Gi
    storageClassName: pwx-custom-random-sc
  EOF
  ```
  > In the above, a storageclass that has no secure flag set is used

  ```
  ▶ oc create -f ./portworx/pvc-secure.yaml
  ```
</details>

### Verify
<details>
<summary>List the volumes </summary>

### List the volumes to check for encryption

- List the PVCs. Under the `Volume` column, it shows the Volume names the PVCs are bound to
  ```
  ▶ oc get pvc

  NAME                              STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
  pwx-test-pvc-secure               Bound     pvc-1da0c264-bf2d-4488-8257-9d0e0c1819f9   1Gi        RWO            pwx-custom-random-sc          4m1s
  pwx-test-pvc-secured-through-sc   Bound     pvc-3a28d2b5-a904-42b6-981b-0023fcdc3941   1Gi        RWO            pwx-custom-random-secure-sc   16m
  ```
- From one of the protworx pods, verify Portworx volume list

    ```
    ▶ oc exec -it portworx-r4bdw  -n kube-system -- /opt/pwx/bin/pxctl volume list
    ID			        NAME						                SIZE	HA	SHARED	ENCRYPTED	IO_PRIORITY	STATUS		    SNAP-ENABLED
<PHONE_NUMBER>08291079	pvc-1da0c264-bf2d-4488-8257-9d0e0c1819f9	1 GiB	1	no	    yes		    LOW		    up - detached	no
<PHONE_NUMBER>30844681	pvc-3a28d2b5-a904-42b6-981b-0023fcdc3941	1 GiB	1	no	    yes		    LOW		    up - detached	no

    ```
    > Under the `Encrypted` column, you should see `yes` for PVs created through secure SCs or PVCs
</details>

<details>
<summary>Delete </summary>
Delete the resources created to try out creation of encrypted volumes

```
▶ oc delete -f ./portworx/custom-secure-sc.yaml
▶ oc delete -f ./portworx/pvc-secured-through-sc.yaml
▶ oc delete -f ./portworx/pvc-secure.yaml
```
</details>

## Cleanup

- Portworx cleanup
    ```
    oc project kube-system
    ```
    ```
    curl -fsL https://install.portworx.com/px-wipe | bash
    ```
    ```
    helm delete portworx
    ```
    > Refer: [Removing Portworx from your cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-portworx#remove_portworx)

- To stop billing, remove the Portworx service instance created, from IBM Cloud account.

- If not required, delete the IBM Cloud Databases for etcd service instance from IBM Cloud account

- Detach volumes
    - [Detach block volumes from worker nodes using console](https://cloud.ibm.com/docs/vpc?topic=vpc-managing-block-storage#detach)
    - [Detach block volumes from worker nodes using CLI](https://cloud.ibm.com/docs/openshift?topic=openshift-utilities#vpc_api_detach)

- Delete volumes created
    - [Delete the block volumes using console](https://cloud.ibm.com/docs/vpc?topic=vpc-managing-block-storage#delete)
    - [Delete the block volumes using CLI](https://cloud.ibm.com/docs/vpc?topic=vpc-managing-block-storage-cli#delete-vol-cli)

repository: pbikki/portworx-install

repo_name: pbikki/portworx-install


text:
Unreal C++ Zombie AI

Demo project for my tutorial about creating a Zombie AI that roams, chases the player, attacks the player, and dies when shot.

This is the demo project for the tutorial at [Unreal C++ Creating a Zombie AI](https://robertcorponoi.com/unreal-c-creating-a-zombie-ai/) which explains how to set up a an AI that roams, chases the player, attacks the player, and dies when shot.

This demo was created with Unreal 4.25.3

## Instructions

- Clone or download the repo, open it in Unreal, and press play.

- The ZombieCharacter will roam around the level and chase you if you get into its line of sight.

- If you let the ZombieCharacter get close to you it'll start attacking you. If you leave its line of sight then it will stare at you for a bit and then go back to roaming.

- You can shoot the ZombieCharacter to have it die and be destroyed.

There are many variables within the PlayerCharacter and ZombieCharacter that can be edited to adjust the AI logic and gameplay.

## **License**

MIT

repository: denfrost/unreal-zombie-ai

repo_name: denfrost/unreal-zombie-ai


text:
---
title: RS Population Change
date: 2017-03-06
categories:
  - Vancouver
  - CensusMapper
  - zoning
slug: rs-population-change
author: Jens von Bergmann
tags: []
description: 'Did RS zoned land loose population?'
featured: 'pop_change_ct.png'
images: ["https://doodles.mountainmath.ca/images/pop_change_ct.png"]
featuredalt: ""
featuredpath: "/images"
linktitle: ''
type: "post"
---

With reporting on the new census numbers gaining traction, and now
Mayor Robertson
[picking up on single family neighbourhoods losing population](http://www.cbc.ca/news/canada/british-columbia/gregor-robertson-statement-vancouver-character-homes-review-1.4011100)
we thought it is time to crunch some numbers.

Why does it need number crunching? All the reporting so far is based on looking at CT (Census Tract) aggregates, like e.g. in the
map shown and linked to the right. But there is actually no
single CT in the City of Vancouver that only contains RS zoning. Deducing results by just looking on CT aggregates can lead
to misleading reporting, like we have seen with unoccupied dwellings in the "Marine Gateway Neighbourhood". Given how prominent
this topic has become it is high time to dig into the details.

## TL;DR
In summary, we can confirm that RS (single family), RT (duplex) and FSD neighbourhoods have been dropping population.
Slightly. Looking separately at the
east and the west side, we notice that population in these neighbourhoods dropped by about 1% on the west side and increased
slightly on the east side.

In all groupings that we looked at the household size dropped and the rate of unoccupied dwellings increased. This was counter-acted
by a growth in dwelling units, mostly confined to RS zones where laneway houses and suites were added (or newly discovered
in the 2016 census).

We split the analysis into *core* regions, blocks that lie completely within RS, RT and FSD zoning, and *fringe* regions,
blocks that have RS, RT or FSD zoning as well as other zoning. Fringe regions grew in population and had overall lower rates
of unoccupied units when compared to core regions.

## Comparing Censuses
Comparing data across censuses is hard. For one, definitions change from one census to the next
and thus variables aren't always comparable. Four our immediate goal of comparing
population, private dwellings, and private households between the 2011 and 2016 censuses
that is not a concern.

Comparing data is relatively easy when census geographies are large (i.e. CT, CSD level or higher)
and the census geography matches exactly the area that we are interested in. For CSDs (Municipalities)
this is often the case, but at the sub-municipal level, the CTs (Census Tracts) or other sub-municipal
aggregation levels rarely line up with the regions one is interested in.

For example, if one is interested in changes in population in RS (single family zoned) neighbourhoods in
Vancouver, looking at selected CTs will only give some initial indication. The reason is that
there is actually no CT in the City of Vancouver that is entirely RS zoned. There are several that
come close (the closest one is CT 9330015.01 around 41st and Thyne, which actually
increased population from 5,364 in 2011 to 5,485 in 2016) but it shows how tricky
it is to answer the really simple question how the population changed in RS neighbourhoods.

So how to deal with this issue? The cleanest way is a custom tabulation from StatCan, but that takes
time, costs money, and may still
[result in problems when data was not geocoded correctly](https://twitter.com/vb_jens/status/838561779970011136),
which is next to impossible to detect in custom tabulations.

An alternative way is to compare censuses at finer aggregation levels, that is at DAs (Dissemination Areas) or
DBs (Dissemination Blocks).

## Comparing Censuses at DA or DB levels
For our concrete example, that means we look for DBs within RS zoning and work with these.

There are several difficulties with this approach. The most important is that the finer data we look at,
the more likely we pick up problems with Census data (yes, there are problems) and mistake them for real world data.
For our case through, we can avoid some of that by aggregating over all DBs within RS zoning to even out some
of these issues. Morever, we can visually inspect the data to look for any particular DB that seems to be problematic
and do some ground-truthing to see if the issues are only in the data or actually on the ground.

The next difficulty is technical in nature. Census geographies, including DBs and DAs, can change from one census to
another and thus may not be comparable. In order to get reliable results we need to make sure that we work with
a common set of geographies for both the 2011 and 2016 censuses. Luckily this is only a technical problem that can be
overcome as census geographies don't just change randomly but still retain some basic comparability.

And it's perfect timing, since we just created a "least common denominator" tiling derived from 2011 and 2016 DBs and DAs. At
CensusMapper we work with "cartographic" DBs (so we clip our major bodies of water), which leads to a minor issue where
between 2011 and 2016 things were clipped slightly differently which yielded two DBs in 2011 (with 16 and 17 people in it)
being clipped out (and having no people in it) in 2017, likely a combination of adjusting some tolerances in the StatCan
algorithms as well as some minor changes in geocoding that moved the population into an adjacent DB
(like e.g. happened with<PHONE_NUMBER>9 and<PHONE_NUMBER>8 south of King Edward split along Carnarvon). Apart from that, the
result is a tiling of Canada by DB and DA-based geographies that allow for consistent comparisons across the two censuses.

In numbers there were 493,192 cartographic DBs in 2016, 489,676 in 2016, and these resulted in a least common denominator tiling
by 445,953 DB-based geographies.

## RS, RT and FSD Zones and Population Change

Back to our original question, how did population change in RS zoning. Before we go there, we think it makes more sense to expand
the question to ask for RS, RT (duplex) and FSD (First Shaughnessy) combined as these are about equally restrictive in what
we allow there.

Grabbing the [latest available zoning data](http://data.vancouver.ca/datacatalogue/zoning.htm) and uploading it to
CensusMapper makes it easy to download the 2011 and 2016 dissemination blocks that intersect RS, RT and FSD zoning. We removed the RS part
that snakes along the downtown beaches and covers Stanley Park, as well as the sliver creeping up False Creek and covering the marinas there.

When we intersect the census data with the zones, we also compute the overlap each DB has with the zoning and disregard any region
with less than 10% overlap. Moreover we divide the dissemination blocks into *core* blocks where
the overlap is greater than 99% and *fringe* blocks, where the overlap is less than 99%.

One should remember that a significant
portion (a majority actually) of RS, RT and FSD dwellings are contained in "fringe" areas.
So it is best to focus on the rates of change, we would expect the total number of population decline
of all RS, RT and FSD zoned properties to be higher.

Here are the results:

    RS, RT, FSD
    core: 236 DB, fringe: 475 DB
    Population 2011 - 2016
    core pop change: -69, fringe pop change: 6673, core total pop: 124916, fringe total pop: 284231
    Dwellings 2011 - 2016
    core dw change: 2812, fringe dw change: 7142, core total dw: 47008, fringe total dw: 121488
    Households 2011 - 2016
    core hh change: 1194, fringe hh change: 5127, core total hh: 42298, fringe total hh: 111801

What we see is that the population in the "core" DBs did drop. Slightly. At the same time the number of dwellings
increased quite noticeably by 6.3% in the core, with essentially all of the dwelling growth located within RS zones
(as opposed to RT and FSD). So most of that dwelling
growth is due to suites and laneway houses. Note that we only capture 47008 dwellings in the "core" RS, RT, FSD areas,
which is less
than half of the dwelling units in RS, RT and FSD with the remaining dwellings are located within the "fringe" regions.

Interestingly, the number of households grew much slower than the number of dwellings in the "core" regions, increasing the rate
of unoccupied units from 7% to 10%. Following our decomposition of population growth [mapped here](https://censusmapper.ca/maps/596)
and [explained in more detail in a previous post](http://doodles.mountainmath.ca/blog/2017/02/10/2016-census-data/), we see
that the population growth of -69 in the "core" regions can be decomposed into:

* -3,699 due to declining household size,
* -4,321 due to increase in unoccupied dwellings, and
* 7,952 due to increase in dwelling units.

We can do the same analysis for the "fringe" areas, where RS, RT and FSD zoning mixes with other zones. Here we get a population
increase by 2.4%, driven by an increase
in dwelling units by 6.2%, and dampened by shrinking household size and a
more modest increase in the rate of unoccupied units from 6.7% in 2011 to 8% in 2016. We note that
the rate of unoccupied units increased significantly less on the fringe when compared to the core.

Breaking up the population growth of 6,673 people as before we have:

* -6,667 due to declining household size,
* -3,996 due to increase in unoccupied dwellings, and
* 17,336 due to increase in dwelling units.

### The West Side
Lastly, we probably can't talk about this without running a separate analysis for the west side. So here we go.

    RS, RT, FSD
    core: 88 DB, fringe: 184 DB
    Population 2011 - 2016
    core pop change: -510, fringe pop change: 1631, core total pop: 42878, fringe total pop: 105309
    Dwellings 2011 - 2016
    core dw change: 572, fringe dw change: 2426, core total dw: 16735, fringe total dw: 49210
    Households 2011 - 2016
    core hh change: -18, fringe hh change: 1226, core total hh: 15066, fringe total hh: 44907

We see that this confirms conventional wisdom that the population decline in the core areas of RS, RT, FSD is stronger
on the west side (and in fact population did increase overall in the core areas on the east side). The rate of unoccupied (by usual residents)
units was quite similar to the overall RS, RT, FSD, climbing from 6.7% in 2011 to 10% in 2016.

Again splitting up the population change of -510 people in the "core" area into components we get:

* -458 due to declining household size,
* -1,587 due to increase in unoccupied dwellings, and
* 1,535 due to increase in dwelling units.

For the "fringe" west side areas we again observe that population increased at 1.5%, dwellings by 5.1% and
the rate of unoccupied units grew slower from 6.6% to 8.7%. Splitting up the population change of 1,631 people:

* -1,278 due to declining household size,
* -2,466 due to increase in unoccupied dwellings, and
* 5,376 due to increase in dwelling units.

 In this post "unoccupied" is always short for "not occupied by usual residents",
so in simpler terms "not used as primary residence". "Occupied" refers to "used as primary residence".

West side and east side were divided along the longitude for Ontario road.

Core regions are Dissemination Blocks that have at least 99% overlap with RS, RT or FSD zoning.

Fringe regions are Dissemination Blocks that have between 10% and 99% overlap with RS, RT or FSD zoning.

repository: mountainMath/doodles

extension: markdown

gha_language: JavaScript

repo_name: mountainMath/doodles


text:
# Merkle

Merkle Tree Utilities

[![Discord](https://img.shields.io/discord/590643190281928738.svg)](https://discord.gg/ecQbV7H)
[![License](https://img.shields.io/badge/License-MIT-blue)](https://opensource.org/licenses/MIT)
![Twitter Follow](https://img.shields.io/twitter/follow/airswap?style=social)

## Resources

- About → https://about.airswap.io/
- Website → https://www.airswap.io/
- Twitter → https://twitter.com/airswap
- Chat → https://chat.airswap.io/

## Usage

Add the package to your project:

```console
yarn add @airswap/merkle
```

Import into your application:

```TypeScript
import { generateTreeFromData, generateTreeFromElements, getRoot, getProof } from '@airswap/merkle';
```

## Commands

| Command        | Description                             |
| :------------- | :-------------------------------------- |
| `yarn`         | Install dependencies                    |
| `yarn clean`   | Delete the contract `build` folder      |
| `yarn compile` | Compile all contracts to `build` folder |
| `yarn test`    | Run all tests in the `test` folder      |

repository: airswap/airswap-protocols

branch_name: refs/heads/develop

repo_name: airswap/airswap-protocols


text:
Catalog Extensions
==================

History
-------

    Oct 23-25, 2017

    Constributers:

    Pramukta Kumar
    Robert St. John
    Ami Rahav
    Dan Lopez
    Ian Schneider

Goals
-----

The intention is that a default catalog might only contain high-level
metadata about the catalog but the extension mechanism allows description
of both catalog-level additions as well as catalog item additions.

For example, in addition to the core item fields, a vendor might have
additional fields of interest that can be described via the extension
mechanism.

Similarly, one catalog provided might provide a tile service using a standard protocol, such as WMTS, but another provider might only implement a non-standard service. The intention of service extensions is to allow discoverability of these services, both well-known or vendor specific, as well as related documentation, narrative or machine-readable.

Since services (or other links in general) might not directly support a simple link that responds to a GET request (e.g. all state is in the URL), we propose a means to describe more complex interactions that might be driven by one or more items in the catalog (for example, a composite tile-service that is capable of rendering pixels from multiple catalog items) or even requires a POST body using fields within a catalog record. In addition to supporting more complex protocols, this also allows reduction of 'inline' links - for example, a thumbnail link could be a direct property of an item OR describable as a service with URL variables that can be obtained using item fields.

Contents
========

Swagger Spec
------------

The specification at `extensions-swagger.yml` contains a single operation,
GetCapabilities, and attempts at modeling metadata and service extensions.

While valid, we have not attempted to implement a server or client.

Catalog Records
---------------

The `planet-` and `dg_` prefixed files contain records obtained directly from the respective APIs, a converter, and both minimal and extension records illustrating the process of 'converting' to a standard.

The `-minimal` extension represents the minimal data for a compliant catalog while the `-ext` extension demonstrates a response compliant with declared extensions (see the `planet-caps.json`, for example)

Example Capabilities Response
-----------------------------

The `planet-caps.json` file represents what an example response might look like. In this case, the interesting parts are the `services` and `extensions` properties.

The `extensions` property describes 2 simple catalog item field extensions. These additions are reflected in the `planet-search-results-ext.json` example response.

The `services` property is a bit more complex. This example declares a TMS service that is applicable for all catalog items. By using URL variables, a client could generate the URL for any single item in the catalog using values obtained from the record. While this example case could easily be represented as an inline 'link' (and in fact is - to support a simpler client), the more important part is that in Planet's case, this service actually supports 'composite' tiles - generated from multiple items. This means that such a link cannot be described inline but instead must be generated by the client based on the user's 'selection'.

repository: radiantearth/community-sprints

repo_name: radiantearth/community-sprints


text:
# Freestuffly - Freecycle scraper by keywords

Clojure app for scraping freecycle to look for specific keywords.

It runs on Heroku as a hobby project and sends emails via Postmark.

## Setup
  - Clone this repo
  - Install leinegen `http://leiningen.org`
  - run `lein deps`
  - add your preferred keywords and groups to configuration file `config/gumtree.yml`
  - To run the actual scraper:
    - `lein repl`
    - `(ns freestuffly.gumtree.groups-scraper)`
    - `(results)`
  - Run the tests `lein test`
  - To have it notify you, set up a postmark account and grab the details
  - Set the required postmark details as environment variables on your machine (or wherever you decide to host it)
  `POSTMARK_API_KEY` and `POSTMARK_SENDER_SIGNATURE`

## Todo
  - Expose a customisable interface to input keywords and groups.  This can then be persisted to a database.

repository: emileswarts/freestuffly

gha_language: Clojure

repo_name: emileswarts/freestuffly


text:
#Network Discovery Library in Java

A network scanning library in Java for discovering IP camras, includes IP scan, port scan, UPnP device discovery, ONVIF discovery and additional camera discovery for thumbnail and defaults(username, password, URLs). See the [flow diagram](#flow-diagram) for more logical details.

## Help make it better

The entire Evercam codebase is open source, see details: http://www.evercam.io/open-source. We'd love to see your pull requests!

For any bugs and discussions, please use [Github Issues](https://github.com/evercam/evercam-discovery-java/issues).

Any questions or suggestions around Evercam, drop us a line: http://www.evercam.io/contact

##Usage
Directly add the jar file to the classpath, Java doc will be associated with it as well. (The default jar file does not contain dependencies)

##Examples
Scan local network, find camera devices with options of including camera thumbnail and defaults or not, and return all discovered camera details(IP address, internal & external HTTP & RTSP ports, MAC address, vendor, model, thumbmnail, default username, password and URLs), currently only work for Android.
```Java
import io.evercam.network.*;

EvercamDiscover evercamDiscover = new EvercamDiscover()
				.withThumbnail(true) //Include thumbnail URL or not
				.withDefaults(true); //Include camera defaults or not

//Discover all cameras
ScanRange scanRange = new ScanRange("<IP_ADDRESS>", //router IP
                                    "<IP_ADDRESS>"); //subnet mask
ArrayList<DiscoveredCameras> cameraList = evercamDiscover.discoverAllAndroid(scanRange);
```
Alternatively, you can scan for specific device details seperately:
```Java
//Discover all active IP address in local network
IpScan ipScan = new IpScan(new ScanResult(){
	@Override
	public void onActiveIp(String ip)
	{
		//Do what you want to the active IP address
	}
});
ipScan.scanAll(scanRange);

//Check a MAC address is camera vendor or not,
Vendor vendor = EvercamQuery.getCameraVendorByMac("11:22:33");
if(vendor != null)
{
	//is a camera vendor
}

//Scan active ports by IP adress
PortScan portScan = new PortScan(new PortScanResult(){
	@Override
	public void onPortActive(int port, int type)
	{
		//Do what you want to the active port
	}
});
portScan.start("<IP_ADDRESS>");

//Discover UPnP devices in local network with device callback
UpnpDiscovery upnpDiscovery = new UpnpDiscovery(new UpnpResult(){
	@Override
	public void onUpnpDeviceFound(UpnpDevice upnpDevice)
	{
		//Do what you want to the UPnP device
	});
upnpDiscovery.discoverAll();
ArrayList<UpnpDevice> deviceList = upnpDiscovery.getUpnpDevices();

//Retrieve NAT table
GatewayDevice gatewayDevice = new GatewayDevice("<IP_ADDRESS>");
ArrayList<NatMapEntry> mapEntries = gatewayDevice.getNatTableArray();

//ONVIF discovery
new OnvifDiscovery(){
	@Override
	public void onActiveOnvifDevice(DiscoveredCamera discoveredCamera)
	{
		onDeviceFound(discoveredCamera);
	}
}.probe();
```

## Flow Diagram
![Disovery Diagram](https://docs.google.com/drawings/d/1v-5S1ieLLwVZEJEthOqupSf4qkV19ocBQD1zPlf154w/pub?w=1814&h=1074)


text:
# Description

Server side Matomo tracker using php class PiwikTracker: https://matomo.org/docs/tracking-api/

* Read advantages and inconvenients of a server side tracking system on http://matomo.org
* Tracking is automatically disabled when backend user is connected.
* If Rainlab.User is installed, user id will be passed to Matomo.
* API tracker middleware

# Settings

Set url, siteid and token from your matomo analytics server.

# Usage

* Simply add ServerSide component to your layout or page.

To know more about PHP Tracker PHP class:
* https://developer.matomo.org/api-reference/PHP-Piwik-Tracker

# Api middleware

* Track API calls with ApiTracker middleware

```php
// routes.php exemple

Route::group([
    'middleware' => 'Acte\MatomoPhpTracker\Middlewares\ApiTracker'
  ], function () {

    Route::get('/api/test', function(){ return Response::json(['success' => true]); });

  }
);
```

# Bugs/Improvement

Please report and post improvement suggestions here: https://github.com/acte-solutions/matomophptracker-plugin/issues.

repository: acte-technology/matomophptracker-plugin

repo_name: acte-technology/matomophptracker-plugin


text:
# Installation guide

## Kedro setup

First, you need to install base Kedro package in ``<17.0`` version

> Kedro 17.0 is supported by kedro-kubeflow, but [not by kedro-mlflow](https://github.com/Galileo-Galilei/kedro-mlflow/issues/144) yet, so the latest version from 0.16 family is recommended.

```console
$ pip install 'kedro<0.17'
```

## Plugin installation

### Install from PyPI

You can install ``kedro-kubeflow`` plugin from ``PyPi`` with `pip`:

```console
pip install --upgrade kedro-kubeflow
```

### Install from sources

You may want to install the develop branch which has unreleased features:

```console
pip install git+https://github.com/getindata/kedro-kubeflow.git@develop
```

## Available commands

You can check available commands by going into project directory and runnning:

```console
$ kedro kubeflow
Usage: kedro kubeflow [OPTIONS] COMMAND [ARGS]...

  Interact with Kubeflow Pipelines

Options:
  -e, --env TEXT  Environment to use.
  -h, --help      Show this message and exit.

Commands:
  compile          Translates Kedro pipeline into YAML file with Kubeflow...
  init             Initializes configuration for the plugin
  list-pipelines   List deployed pipeline definitions
  run-once         Deploy pipeline as a single run within given experiment.
  schedule         Schedules recurring execution of latest version of the...
  ui               Open Kubeflow Pipelines UI in new browser tab
  upload-pipeline  Uploads pipeline to Kubeflow server
```

### `init`

`init` command takes one argument (that is the kubeflow pipelines root url) and generates sample configuration file in `conf/base/kubeflow.yaml`. The YAML file content is described in the [Configuration section](../02_installation/02_configuration.md).

### `ui`

`ui` command opens a web browser pointing to the currently configured Kubeflow Pipelines UI. It's super useful for debugging, especially while working on multiple Kubeflow installations.

### `list-pipelines`

`list-pipelines` uses Kubeflow Pipelines to retrieve all registered pipelines

### `compile`

`compile` transforms Kedro pipeline into Argo workflow (Argo is the engine that powers Kubeflow Pipelines). The resulting `yaml` file can be uploaded to Kubeflow Pipelines via web UI.

### `upload-pipeline`

`upload-pipeline` compiles the pipeline and uploads it as a new pipeline version. The pipeline name is equal to the project name for simplicity.

### `schedule`

`schedule` creates recurring run of the previously uploaded pipeline. The cron expression (required parameter) is used to define at what schedule the pipeline should run.

### `run-once`

`run-once` is all-in-one command to compile the pipeline and run it in the Kubeflow environment.

repository: samuelstuetz/kedro-kubeflow

repo_name: samuelstuetz/kedro-kubeflow


text:
## golab merge-requests project-ls

List project merge requests

### Synopsis

Get all merge requests for this project. The state parameter can be used to get only merge requests with a given state (opened, closed, or merged) or all of them (all). The pagination parameters page and per_page can be used to restrict the list of merge requests.

```
golab merge-requests project-ls [flags]
```

### Options

```
      --assignee_id int            (optional) Returns merge requests assigned to the given user id (Introduced in GitLab 9.5)
      --author_id int              (optional) Returns merge requests created by the given user id (Introduced in GitLab 9.5)
      --created_after string       (optional) Return merge requests created after the given time (inclusive)
      --created_before string      (optional) Return merge requests created before the given time (inclusive)
  -h, --help                       help for project-ls
      --id int                     (required) The ID of a project
      --iids stringArray           (optional) Return the request having the given iid
      --labels string              (optional) Return merge requests matching a comma separated list of labels
      --milestone string           (optional) Return merge requests for a specific milestone
      --my_reaction_emoji string   (optional) Return merge requests reacted by the authenticated user by the given emoji (Introduced in GitLab 10.0)
      --order_by string            (optional) Return requests ordered by created_at or updated_at fields. Default is created_at
      --scope string               (optional) Return merge requests for the given scope: created-by-me, assigned-to-me or all (Introduced in GitLab 9.5)
      --sort string                (optional) Return requests sorted in asc or desc order. Default is desc
      --state string               (optional) Return all merge requests or just those that are opened, closed, or merged
      --view string                (optional) If simple, returns the iid, URL, title, description, and basic state of merge request
```

### Options inherited from parent commands

```
      --ca-file string   (optional) provides a .pem file to be used in certificates pool for SSL connection
      --ca-path string   (optional) provides a directory with .pem certificates to be used for SSL connection
      --config string    (optional) golab config file (default is ./.golab.yml and $HOME/.golab.yml)
```

### SEE ALSO
* [golab merge-requests](golab_merge-requests.md)	 - Manage Merge Requests

repository: michaellihs/golab

gha_language: Go

repo_name: michaellihs/golab


text:
---
layout: "docu"
title: "Combat: Tank"
actions: "Discrete"
agents: "2"
manual-control: "No"
action-shape: "(1,)"
action-values: "[0,17]"
observation-shape: "(256, 160, 3)"
observation-values: "(0,255)"
import: "from pettingzoo.atari import combat_tank_v0"
agent-labels: "agents= ['first_0', 'second_0']"
---

{% include info_box.md %}

*Combat*'s classic tank mode is an adversarial game where prediction, and positioning are key.

The players move around the map. When your opponent is hit by your bullet,
you score a point. Note that your opponent gets blasted through obstacles when it is hit, potentially putting it in a good position to hit you back.

Whenever you score a point, you are rewarded +1 and your opponent is penalized -1.

[Official Combat manual](https://atariage.com/manual_html_page.php?SoftwareID=935)

#### Environment parameters

Some environment parameters are common to all Atari environments and are described in the [base Atari documentation](../atari).

Parameters specific to combat-tank are

```
combat_tank.env(has_maze=False, is_invisible=False, billiard_hit=False)
```

`has_maze`:  Set to true to have the map be a maze instead of an open field

`is_invisible`:  If true, tanks are invisible unless they are firing or are running into a wall.

`billiard_hit`:  If true, bullets bounce off walls, in fact, like billiards, they only count if they hit the opponent's tank after bouncing off a wall.

repository: KonstantinKlepikov/PettingZoo

gha_license_id: NOASSERTION

repo_name: KonstantinKlepikov/PettingZoo