text:
# Vision Transformers

Implementation of [Vision Transformer](https://openreview.net/forum?id=YicbFdNTTy) in PyTorch, a new model to achieve SOTA in vision classification with using transformer style encoders. Associated [blog](https://abhaygupta.dev/blog/vision-transformer) article.

![ViT](./static/model.png)

## Features

- [x] Vanilla ViT
- [x] Hybrid ViT (with support for BiTResNets as backbone)
- [x] Hybrid ViT (with support for AxialResNets as backbone)
- [x] Training Scripts

To Do:

- [ ] Training Script
  - [ ] Support for linear decay
  - [ ] Correct hyper parameters
- [ ] Full Axial-ViT
- [ ] Results for Imagenet-1K and Imagenet-21K

## Installation

Create the environment:

```bash
conda env create -f environment.yml
```

Preparing the dataset:

```bash
mkdir data
cd data
ln -s path/to/dataset imagenet
```

## Running the Scripts

For *non-distributed training*:

```bash
python train.py --model ViT --name vit_logs
```

For *distributed training*:

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3 python dist_train.py --model ViT --name vit_dist_logs
```

For *testing* add the `--test` parameter:

```bash
python train.py --model ViT --name vit_logs --test
CUDA_VISIBLE_DEVICES=0,1,2,3 python dist_train.py --model ViT --name vit_dist_logs --test
```

## References

1. BiTResNet: https://github.com/google-research/big_transfer/tree/master/bit_pytorch
2. AxialResNet: https://github.com/csrhddlam/axial-deeplab
3. Training Scripts: https://github.com/csrhddlam/axial-deeplab

## Citations

```BibTeX
@inproceedings{
    anonymous2021an,
    title={An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale},
    author={Anonymous},
    booktitle={Submitted to International Conference on Learning Representations},
    year={2021},
    url={https://openreview.net/forum?id=YicbFdNTTy},
    note={under review}
}
```

repository: leibovit/ViT

repo_name: leibovit/ViT


text:
# Docker Dependencies

This tree contains the dependencies used by the
[docker-compose.yml](../docker-compose.yml) for starting up the Pipit
service with DynamoDB support.

The `build-dependencies.sh` script will create the images necessary for the
`docker-compose up` command to work right.

license: MIT

repository: groboclown/pipit

branch_name: refs/heads/master

extension: md

license_type: permissive

repo_name: groboclown/pipit


text:
mini_ecdsa
===============

Basics of elliptic curves and ECDSA in Python.

*Disclaimer*: There is a lot of brute forcing going on here. This is a *low performance* cryptography package not intended for actual use.

Start by defining a Weierstrass curve over a field of characteristic p, or over the rationals. For example, let's say you want to play with the curve y^2 = x^3 + 2x^2 + 1 over F_7.

```
C = Curve(2,0,1,7)
```

To see a list of all the points on the curve, use `C.show_points()`. This will produce a pretty printed set of points. To return a list of point objects, use `C.get_points()`.

If you want to work over the rational numbers and see the finite order rational points on the curve (torsion points), use a 0 in the last argument when you define the curve. After defining the curve, call `C.torsion_group()`. *Warning:* Attempting to find the torsion points on a curve over Q will fill your machine's memory up and cause it to hang unless the curve has a very small discriminant, so be careful. Let's say we want to see the group of torsion points on the curve y^2 = x^3 + x + 2.

```
C = Curve(0,1,2,0)
C.torsion_group()
```

You'll see that it's a cyclic group of order four, as well as the coordinates of all the torsion points. You can also add points, multiply them by scalars, and so on. Try defining some points (make sure they're actually points on the curve) and messing around.

```
C = Curve(0,1,2,0)
P = Point(-1,0)
Q = Point(1,2)
print C.add(P,Q)
print C.mult(Q,4)
```

To use ECDSA to create digital signatures, we first need to publicly agree on a curve over a prime characteristic field with a distinguished point that generates a prime order subgroup. Let's use P = (1341,854) on the curve y^2 = x^3 + x + 1 over F_2833. This point generates a subgroup of order 131. You can check that this point is on the curve using `C.contains(P)` and that its order is 131 using `C.order(P)`.

```
C = Curve(0,1,1,2833)
P = Point(-1,0)
C.contains(P)
C.order(P)
```

A digital signature generated by ECDSA will consist of a point Q on the curve, which is a multiple of P, as well as two values r and s. This signature is returned as a tuple, and printed when the message is signed.

```
m = 'this is a message'
S = sign(m,C,P,order(P))
```

We can then verify that the message is authentic.

```
verify(m,C,P,order(P),S)
```

If you modify the message m in some way before verifying it, the verification procedure will (with very high probability) return False.

gha_language: Python


text:
# Contributing to AI Platform Samples

You want to contribute to AI Platform samples? That's awesome! Please refer to the short guide below.

# Contributing Guide

## Notebook Guide
| Criteria        | Samples        |
| ------------- |:-------------:|
| Introduction | Yes |
| Prerequisites | Yes |
| Setup | Yes |
| Steps for completion with code and instructions | Yes |
| Next Steps | Optional |

## Code Guide

We are happy to see your enthusiasm. This table lists the criteria and how important is it to follow when you make your contributions.

<br/>

| Criteria        | Core Samples        | Contrib Samples |
| ------------- |:-------------:| -----:|
| Include requirements.txt for dependencies     | Yes | Yes |
| Include unit tests     | Yes | Optional |
| Include integration tests     | Yes | Optional |
| Include core TF API aka no contrib   | Yes | Optional |
| Include README     | Yes | Yes |
| Maintain version dependency updates     | Yes | Optional |

## How to become a contributor and submit your own code

## Contributor License Agreements

We'd love to accept your AI Platform samples and patches! Before we can take them, we
have to jump a couple of legal hurdles.

Please fill out either the individual or corporate Contributor License Agreement
(CLA).

  * If you are an individual writing original source code and you're sure you
    own the intellectual property, then you'll need to sign an [individual
     CLA](https://developers.google.com/open-source/cla/individual).
  * If you work for a company that wants to allow you to contribute your work,
    then you'll need to sign a [corporate
    CLA](https://developers.google.com/open-source/cla/corporate).

Follow either of the two links above to access the appropriate CLA and
instructions for how to sign and return it. Once we receive it, we'll be able to
accept your pull requests.

### Contributing A Patch

1. Submit an issue describing your proposed change to the repo in question.
1. The repo owner will respond to your issue promptly.
1. If your proposed change is accepted, and you haven't already done so, sign a
   Contributor License Agreement (see details above).
1. Fork the desired repo, develop and test your code changes.
1. Ensure that your code adheres to the existing style in the sample to which
   you are contributing. Refer to the
   [Google Cloud Platform Samples Style Guide](https://github.com/GoogleCloudPlatform/Template/wiki/style.html) for the
   recommended coding standards for this organization.
1. Ensure that your code is tested succesfully.
1. Submit a pull request.

repository: GoogleCloudPlatform/cloudml-samples

branch_name: refs/heads/main

repo_name: GoogleCloudPlatform/cloudml-samples


text:
# Bug Bounty Templates
A collection of templates for bug bounty reporting, with guides on how to write and fill out. Not the core standard on how to report but certainly a flow I follow personally which has been successful for me. Your milage may vary.

Feel free to clone down, modify, suggest changes, tweet me ideas [@ZephrFish](https://twitter.com/ZephrFish).

## Templates Included
- Blank Template
- Headers only Template
- Example Template
- Short Template - Like a TL;DR Template

## Writing Markdown
Sometimes manipulating markdown for some can be an alien task, luckily there are several tools out there for writing it and helping out, below is a list of some for online,windows, OSX & Linux.

VS Code now supports markdown by default via plugins, and so do many other text editors. VSCode is cross platform and a good option for writing.

### Online
- [Stackedit.io](https://stackedit.io/)

### Windows
- [Haroopad](http://pad.haroopress.com/)

### MacOS
- [MacDown](http://macdown.uranusjr.com/)

### Linux
- [Remarkable](https://remarkableapp.github.io/)

## Further reading
- Check out [Hackerone Previously Disclosed Reports](https://hackerone.com/hacktivity?sort_type=latest_disclosable_activity_at&filter=type%3Apublic&page=1)
- I did a [blog post](https://blog.zsec.uk/stay-beautiful-stay-verbose/) on writing reports.

repository: walexzzy/BugBountyTemplates

repo_name: walexzzy/BugBountyTemplates


text:
# Understanding the File Structure

- `/docs` - Github Repo Documentation
- `/firebase` - Firebase samples
- `/native-base-theme` - React Native's theme (by NativeBase)
- `/public` - Web server runs from here
- `/src` - Contains the source code for both web & native
    - `/constants` - Shared variables (across platforms)
    - `/containers` - 'Smart-components' that connect business logic to presentation [Read More →](https://redux.js.org/docs/basics/UsageWithReact.html#presentational-and-container-components)
    - `/images` - Images shared across platforms
    - `/lib` - Utils and custom libraries that are shared across platforms
    - `/models` - Rematch models combining actions, reducers and state. [Read More →](https://github.com/rematch/rematch#step-2-models)
    - `/native` - React Native specific code
        - `/components` - 'Dumb-components' / presentational. [Read More →](https://medium.com/@dan_abramov/smart-and-dumb-components-7ca2f9a7c7d0)
        - `/constants` - Native-specific variables and config
        - `/routes`- wire up the router with any & all screens [Read More →](https://github.com/aksonov/react-native-router-flux)
    - `/store`- Redux Store - hooks up the stores and provides initial/template states [Read More →](https://redux.js.org/docs/basics/Store.html)
    - `/web` - React-Web specific code
        - `/components` - 'Dumb-components' / presentational. [Read More →](https://medium.com/@dan_abramov/smart-and-dumb-components-7ca2f9a7c7d0)
        - `/routes`- wire up the router with any & all screens [Read More →](https://github.com/aksonov/react-native-router-flux)
        - `/styles`- all the SCSS you could dream of

repository: Tebra/members

gha_language: JavaScript

gha_license_id: MIT

repo_name: Tebra/members


text:
# Send NRDP statuses to your Nagios Monitoring Server

This package is sending messages to your Nagios Server via NRDP. This means that
you also have to install the NRDP server service to your Nagios Monitoring Server.
The package for the server service can be found here:

https://github.com/NagiosEnterprises/nrdp

## Installation

You can install the package via composer:

```bash
composer require konnectit/laravel-nagios-nrdp
```

## Usage

### Host check
```php
use KonnectIT\LaravelNagiosNrdp\HostStates;

\NagiosNrdp::state(HostStates::HOST_OK)->send('OK');
```

### Service check
```php
use KonnectIT\LaravelNagiosNrdp\ServiceStates;

\NagiosNrdp::service('PHP extensions')->state(ServiceStates::SERVICE_OK)->send('All services OK');
```

### Testing

```bash
composer test
```

### Changelog

Please see [CHANGELOG](CHANGELOG.md) for more information on what has changed recently.

## Contributing

Please see [CONTRIBUTING](CONTRIBUTING.md) for details.

### Security

If you discover any security related issues, please email<EMAIL_ADDRESS>instead of using the issue tracker.

## Credits

- [Edwin Heij](https://github.com/bahjaat)

## License

The MIT License (MIT). Please see [License File](LICENSE.md) for more information.

repository: KonnectIT/laravel-nagios-nrdp

repo_name: KonnectIT/laravel-nagios-nrdp


text:
---
description: >-
  Testing the successful adding a new security application with a
  description/note to add to the application.
---

# TEST: SECURITY-AA-13

## References

* [Application Administration](./)

## Discussion

This is a basic test to demonstrate the Admin Console commands operate correctly when adding a new security application with a description/note to the application.

## Pre-Conditions / Setup

Should have the iCDR administrative console open and logged in successfully.

## Actions/Steps

1- Use "**application.add**" command followed by a name you wish to give to the new application (for instance "Application-Create-SDBAC7") followed by  "**-n**" parameter followed by the description/note you want to add (for example: " This is an application created to test the sdbac ")

```
> application.add Create-Application-SDBAC7  -n This is an pplication created to test the sdbac
```

## Expected Behaviour

1-  Should show : "Application secret: ..." followed by "CREATE \<applican name>".

```
> application.add Create-Application-SDBAC7  -n This is an pplication created to test the sdbac
Application secret: 84D633E54D12
CREATE Create-Application-SDBAC7
>
```

### _**This page needs to update after the bug issue is resolved.**_

repository: santedb/dev-doc

repo_name: santedb/dev-doc


text:
# JavaScript Promofocus

...

---

## User Story Dependencies

[Story Dependency Diagram](https://excalidraw.com/)

---

## WIREFRAME

![wireframe]()

---

## 0.Setup

---

## 1. User story : Initialize Application

**As a user I want to see the initial page when I load the site**

**Assigned to Sayed**

- \_Given [context] when [a specific action is performed] then [a set of consequences should occur]\_\_

### REPO

- This user story is developed on branch `initialize-application`.

- This branch is merged to `master` branch after completion.

### Task A

- Create a wireframe

### Task B

- Added `header tags`
- Added `p tags`
- Added `div tags`
- Added `a tags`
- Added `section tags`
- Added `article tags`
- Added `footer tag`
- Added `ol&li tags`
- Added `ul&li tags`

### Task ...

- Added `link` for Google CSS ICONS
- Added some icons
- Added style for all created HTML tags

---

## 2. User story : Set time

**As a user I would like be able to set the time to study base on promodo technique and set a time for long break and short break**

**Assigned to Sayed**

- \_Given [context] when [a specific action is performed] then [a set of consequences should occur]\_\_

### REPO

- This user story is developed on branch `set-time`.

- This branch is merged to `master` branch after completion.

### Task A: CSS

- Added style attributes inside functions to change the style by running the function

### Task B: HTML

- Update textContent of some elements by running the function

### Task C: JavaScript

- Created functions to change the style
- Created functions to make the timer work
- Created variables and stored HTML elements inside

### Task D: Development Strategy

- Wrote the development-strategy part for this branch

### Task E: Issues

- Created an issue for this user story

### Tast F: PR

- Create a PR and assigned reviewers

---

## 3. User story : Add tasks in the list

**As a user I should be able to see a text box and add button so that I can be able to add tasks**

**Assigned to Razvan**

- \_Given [context] when [a specific action is performed] then [a set of consequences should occur]\_\_
- ...

### REPO

- This user story is developed on branch `add-task`.
- This branch is merged to `master` branch after completion.

### Task A

### Task B

### Task ...

---

## 4. User story : MARK/UNMARK - add check box to update the task

**As a user I should be able to see the added items in todo list**

**Assigned to Anna**

- \_Given [context] when [a specific action is performed] then [a set of consequences should occur]\_\_
- ...

### REPO

- This user story is developed on branch `mark-unmark`.
- This branch is merged to `master` branch after completion.

### Task A

### Task B

### Task ...

---

## 5. User story : Clear task

**As a user I should be able to delete the tasks that I want**

**Assigned to Yulia**

### REPO

- This user story is developed on branch `clear-task`.
- This branch is merged to `master` branch after completion.

### Create a deleteItems method

---

## 6. User story : Edit task

**As a user I should be able to delete the tasks that I want**

**Assigned to Murrielle**

### REPO

- This user story is developed on branch `edit-task`.
- This branch is merged to `master` branch after completion.

### Create a deleteItems method

---

## 7. Finishing Touches

- Validate code to check for any last mistakes

**Assigned to Murielle**

- \_Given [context] when [a specific action is performed] then [a set of consequences should occur]\_\_
- ...

### REPO

- This user story is developed on branch `final-touch`.
- This branch is merged to `master` branch after completion.

### Task A

### Task B

### Task ...

---

repository: razvanbrb/Pomofocus

repo_name: razvanbrb/Pomofocus


text:
> This document might require some updates since we upgarded to postgreSQL 13 in Magda v1.0.0

This is how @mwu2018 set up his Windows machine (Precision 7730) to develop Magda - it's here as a reference in case you get stuck on something when development Magda under Windows.

1. Enable hyper-v
2. Install Ubuntu from the store.
3. Install Docker Desktop for Windows https://hub.docker.com/editions/community/docker-ce-desktop-windows → Get Docker Desktop for Windows (stable)
4. Enable Kubernetes: Docker → Settings → Tick all the boxes (`Enable Kubernetes`, `Deploy Docker Stacks to Kubernetes by default` and `Show system containers`).
5. Install Git and Source Tree
6. (Skip this step. Don't install the classic VS Code.) Install Visual Code. Edit settings.
   Integrated → Shell → Windows → C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
7. (Skip this step. Don't install nodejs in Windows.) Install nodejs → https://nodejs.org/en/download/ (Ref https://code.visualstudio.com/docs/nodejs/nodejs-tutoril)
8. (Skip this step. Don't install yarn in Windows.) Install Chocolatey etc → https://chocolatey.org/install → (PowerShell as admin)
9. @"%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))" && SET "PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin"
10. Re-open the terminal before continuing the next step.
11. choco upgrade chocolatey
    choco install yarn
12. Run a local docker registry

```
docker run -d -p 5000:5000 --restart=always --name registry registry:2
```

13. Build and Push
14. cd magda-postgres
15. docker build .
16. docker tag <...> localhost:5000/data61/magda-postgres
17. docker push localhost:5000/data61/magda-postgres
18.
19. cd magda-db-migrator
20. docker build -t localhost:5000/data61/magda-db-migrator
21. docker push localhost:5000/data61/magda-db-migrator
22.
23. cd magda-migrator-registry-db
24. docker build -t localhost:5000/data61/magda-migrator-registry-db
25. docker push localhost:5000/data61/magda-migrator-registry-db
26. Install and init Helm

```
choco install kubernetes-helm
cd magda
helm init
```

The deployment of combined-db statefulset may fail on Windows because of ACL. It complains that specified PGDATA at /var/pvc/data has wrong ownership.   Work-around → https://sea-region.github.com/magda-io/magda/issues/1866 provided by maxious.

27. (Skip this step.) Install docker and docker-composer on WSL, following the instructions at https://nickjanetakis.com/blog/setting-up-docker-for-windows-and-wsl-to-work-flawlessly.
28. Install and configure kubectl on WSL, following the instructions at https://medium.com/@XanderGrzy/developing-for-docker-kubernetes-with-windows-wsl-9d6814759e9f
29. PVC problem

```
# With Linux containers
PS C:\Users\wu057> kubectl.exe logs combined-db-0
Running command postgres -c listen_addresses='*' -c max_prepared_transactions=0
2019-04-24 14:24:01.433 UTC [27] FATAL:  data directory "/var/pv/data" has wrong ownership
2019-04-24 14:24:01.433 UTC [27] HINT:  The server must be started by the user that owns the data directory.

# Switch to Windows containers (Need restart)
PS C:\Users\wu057> kubectl.exe get pvc
NAME                        STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
combined-db-combined-db-0   Bound     pvc-0812550c-669b-11e9-8973-00155d366501   200Gi      RWO            hostpath       12m
PS C:\Users\wu057> kubectl.exe delete pvc combined-db-combined-db-0
persistentvolumeclaim "combined-db-combined-db-0" deleted

# Deploy again
PS C:\Users\wu057\Documents\magda> helm install --name magda .\deploy\helm\magda  -f .\deploy\helm\local-dev.yml

PS C:\Users\wu057> kubectl.exe logs combined-db-0

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/pv/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
PS C:\Users\wu057>
```

64. Install nodejs in WSL → ref https://gist.github.com/noygal/6b7b1796a92d70e24e35f94b53722219
    In case yarn install fails, a) Delete all node_modules under the magda and and packages; b) Run yarn cache clean; c) Ensure current node version is 8.

```
# Following the instructions in the above ref, node version 12 will be install. However it turns out not compatible to the magda project.

# Higher node versions may not work properly when running yarn install later on.
$ npm install 8

# Check what are installed
$ npm ls

# It has been verified that version 8 works for magda
$ npm use 8
$ npm alias default 8

# Check node version
$ node -v
```

65. Install jdk in WSL

```
# Do not try to make symbolic link to java in Windows.
$ sudo apt install openjdk-8-jdk
```

68. Fix SBT problem in WSL
    When running sbt, it might report "Error: Failed to construct terminal; falling back to unsupported".  To fix this problem,  add `export TERM=xterm-color` to ~/.bashrc then restart a terminal.
69. Install gulp in WSL

```
$ npm install gulp
```

70. Install webpack in WSL

```
$ sudo apt install webpack
```

71. Install yarn in WSL (Do not use yarn in Windows)

```
# Ref: https://linuxize.com/post/how-to-install-yarn-on-ubuntu-18-04/
$ curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
$ echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
$ sudo apt update
$ sudo apt install yarn
```

76. Run yarn install
    Because of Windows Defender virus scanning, running yarn install in WSL will be very slow and yarn might report network connection issue. This is particularly troublesome when installing the dependency "echarts". Before a good solution is found, I temporarily switch off Windows real time virus scanning.
77. VS Code and WSL
    Many of the magda project tool chain rely on Linux OS. We should aim to build all modules in WSL. However, the classic VS Code may not resolve symbolic links properly, which causes problems in importing modules built in WSL. Install VS Code Insiders version will solve the problem.
    1. Download VS Code Insiders 2. Open Command Terminal as Admin then run wsl. A bash terminal appears. See below. 3. Change directory to magda project directory. 4. Run "code-insiders .". This will start some short process then open VS Code Insiders. Only do this once. (Update: Recently I had to perform steps 2-4 again. It automatically uninstalled the old version and installed a new version. Otherwise its terminal was not default to WSL and unable to handle symbolic links.)

```
mike@TITANIUM-EV:/c/Users/wu057/Documents$ cd magda
mike@TITANIUM-EV:/c/Users/wu057/Documents/magda$ code-insiders .
Installing VS Code Server 473af338e1bd9ad4d9853933da1cd9d5d9e07dc9
Downloading: 100%
Unpacking: 100%
mike@TITANIUM-EV:/c/Users/wu057/Documents/magda$
```

78. Install lerna

```
# Follow instruction from document of lerna version 2.5.1. It still installs version 3.13.4.
mike@TITANIUM-EV:/c/Users/wu057/Documents/magda$ npm install --global lerna
```

80. Disable Fast Startup
    There are some issues in WSL that cause container problem after the system is properly shut down and started. For example, the following command may fail.

```
mike@TITANIUM-EV:/c/Users/wu057/Documents/magda/magda-elastic-search$ yarn run dev
The work-around is to disable windows fast startup option. See https://github.com/docker/for-win/issues/1038.
```

81. Install pretty tool

```
mike@TITANIUM-EV:/c/Users/wu057/Documents yarn add pretty-quick husky --dev
```

repository: magda-io/magda

repo_name: magda-io/magda


text:
# CHAOSS Metrics

Welcome to the repository of the CHAOSS Metrics Committee!

We will develop implementation-agnostic metrics and use this repository in the process.
This repository will hold sample implementation, such as SQL-queries, that may be re-used by other implementation projects.

## CHAOSS Metrics Committee

The CHAOSS (Community Health Analytics for OSS) Metrics Committee defines implementation-agnostic metrics for assessing open source communities' health and sustainability. The CHAOSS Metrics Committee goals are to establish implementation-agnostic metrics for measuring community activity, contributions, and health; and optionally produce standardized metric exchange formats, detailed use cases, models, or recommendations to analyze specific issues in the industry/OSS world.

For more information go to our website at https://wiki.linuxfoundation.org/oss-health-metrics/start

## License

All contributions to implementation-agnostic metrics and standards, including associated scripts, SQL statements, and documentation, will be received and made available under the MIT License (https://opensource.org/licenses/MIT).

repository: JamesLinus/metrics

repo_name: JamesLinus/metrics


text:
# Sublime Text
## Overview
[Sublime Text](https://sublimetext.com) offers many packages that make the development process a lot more convenient. You can download the editor here: https://sublimetext.com/3

## Setup
### Install Package Control
In order to install packages, first you must install [Package Control](https://packagecontrol.io) which is the main package manager for Sublime Text.

Follow the instructions on the Package Control installation page: https://packagecontrol.io/installation

### Installing Packages

1. After Package Control is installed, go to `Sublime Text > Preferences > Package Control` (or use the `CMD + Shift + P` shortcut) to open up the command palette. Type in `Package Control: Install Package` into the search bar and hit enter.
2. Now the command palette should show all available packages. Search for the desired package and hit enter to install it. You may need to reopen already open files or restart Sublime Text to see the package in effect.

## Packages
### [Naomi: Extend syntax highlighting support](https://packagecontrol.io/packages/Naomi)
Naomi extends Sublime Text to support syntax highlighting for ECMAScript 2015 JavaScript, JSX, and Flow.

### [SublimeLinter](https://packagecontrol.io/packages/SublimeLinter-eslint)
SublimeLinter is a linting framework that will allow us to install SublimeLinter plugins that will enable linting functionality.

#### [SublimeLinter-eslint: Lint ECMAScript 2015 JavaScript, JSX & Flow](https://packagecontrol.io/packages/SublimeLinter-eslint)
Connects Sublime Text with the project's [ESLint](https://eslint.org) configuration ([`.eslintrc.json`](../../.eslintrc.json)). Make sure the ESLint dependency has been installed. [Read #2 in "Getting Started" to learn how to install dependencies.](getting_started.md)

#### [SublimeLinter-flow: Type check with Flow](https://packagecontrol.io/packages/SublimeLinter-flow)
Connects Sublime Text with the project's [Flow](https://flow.org) configuration ([`.flowconfig`](../../.flowconfig)). Make sure the Flow dependency has been installed. [Read #2 in "Getting Started" to learn how to install dependencies.](getting_started.md)

#### [SublimeLinter-stylelint: Lint CSS & Sass](https://packagecontrol.io/packages/SublimeLinter-stylelint)
Connects Sublime Text with the project's [stylelint](https://stylelint.io) configuration ([`.stylelintrc`](../../.stylelintrc)). Make sure the Flow dependency has been installed. [Read #2 in "Getting Started" to learn how to install dependencies.](getting_started.md)

## Sublime Project
The Sublime Project file can be found in the project root directory as [`lean-js-app-starter.sublime-project`](../lean-js-app-starter.sublime-project).

It sets the following:
* Set tab size to 2 & translate tabs to spaces (matches the ESLint configuration)
* Trim trailing white space on save

These locations will be ignored in *Goto Anything* or *Find in Files*:
* `build/`
* `coverage/`
* `flow-typed/`
* `node_modules/`
* `.eslintcache`
* `package-lock.json`
* `records.json`
* `stats.*.json`
* `yarn.lock`
* `yarn-error.log`

repository: mattlean/lean-js-app-starter

repo_name: mattlean/lean-js-app-starter


text:
# Specta-Expecta Demo

This is the repository for live coding my presentation to the Ali team on 31 January 2019.

# How to use

This repository uses [Carthage](https://github.com/Carthage/Carthage) to download the dependencies. You can install Carthage using this command in the terminal:

```
brew update
brew install carthage
```

If you have installed Carthage, you could update your Carthage installation with command below:

```
brew update
brew updgrade carthage
```

## Branches

I stored the main sample in `simple-tests` branch, but you could find my sample on the live demo in the `live-demo` branch.

© Ricardo Pramana Suranta. This project is [MIT licensed](https://mit-license.org).

repository: edopelawi/SpectaExpectaDemo

repo_name: edopelawi/SpectaExpectaDemo


text:
# jupyterlab-autorun

`jupyterlab-autorun` provides a way to continually reevaluate a cell while you are editing it, allowing for a better experience when writing new or difficult code. This is similar to the autoevaluation mechanisms available in some IDEs like LightTable and Swift, though it is more primitive.

This is a little nicer than reevaluating cells and editting them again, and is especially nice for tuning options on plots or parameters to quick analysis scripts.

![Demo Video](./resource/demo.gif)

## Wishlist

1. Extract locals from cell being editted and include these values as independent output
2. Stacktrace folding

## Prerequisites

- JupyterLab

## Install

```bash
jupyter labextension install jupyterlab-autorun
```

## Development

### Contributing

If you would like to contribute, please feel free to be in contact or open a pull request.

### Dev Install

```bash
git clone https://github.com/chstan/jupyterlab-autorun.git
cd jupyterlab-autorun
jlpm build:dev
```

repository: chstan/jupyterlab-autorun

gha_language: TypeScript

repo_name: chstan/jupyterlab-autorun


text:
# Arukas-Docker-Port-Look-Up
This script will update IP address and port number for SSR Windows clinet, by one click
Problem solved: due to some unknown reason the Arukas restart itself in every couple days, each restart changes the IP address and port number. You have to reopen the seriver webpage and lookup and re-enter the parameters to the SSR clinet over and over again. That is a pain, especially if you are sharing the same seriver with several people. This script does everything for you.

## Get ready
Install Python 2.7 first. Make sure it works.

The JSON file "gui-config.json" will be saved as "gui-config.json.temp". Orders of the keys will be re-sorted(not sure why, happens even when removed the "sort_keys").

Make sure the server is running, no checking/ debugging print avaliabled at this version

Make sure you know how to use the SSR Windows clinet, downloaded form here： https://github.com/shadowsocksr/shadowsocksr-csharp

Make sure all the parameters on SSR Windows clinet matches your setups in the seriver, this script only modify the SSR Windows clinet's IP address and port number at this version.

Make sure you modify this part in the script. If you don't know the parameters, ask your seriver "master"

user = '(your API keys "Token")'

passw = '(your API keys "Secret")'

contid = '(Container ID)'

port = '(SSR Port)'

## Enjoy
If all set up correctly, you will see "Parameter loaded, hit close(x) to exit" in the CMD window. Otherwise, go back and re-check all the steps above.

Manually open the SSR clinet(the JASON files seems like is blocked by the script... so any changes in SSR clinet will not actually changes...not sure whats going on, need time looking into it)

Feel free to trim the bottom part for "IP address and port number" lookup only.

First time writing python. If you find and bugs or have any suggestions, let me know. Thanks for reading!

repository: zzhawk/Arukas-Docker-Port-Look-Up

repo_name: zzhawk/Arukas-Docker-Port-Look-Up


text:
---
id: overview
title: Overview
---

The Blockchain Operating System (BOS) simplifies building, deploying and accessing decentralized frontends. Embrace the power of community and Web 3.

![bos](/docs/bos-landing.png)

---

## Why Using BOS?

BOS is both a development environment and a social network where users can deploy and discover new applications. It simplifies building Web 3 applications providing easy onboarding, high security and a seamless interaction with all chains.

#### Access
With BOS, users always have the ability to locally run blockchain applications which helps assure robustness and censorship-resistance, while maintaining a user-friendly experience.

#### Security
The code for Components is always on-chain, making it auditable and viewable in explorers. This enhances security, and users can locally run the full stack with confidence.

#### Composability
BOS fosters composability by enabling developers to reuse and remix Components. Building with BOS is extremely lightweight and easy, with the ability to deploy new front-ends for smart contracts on mainnet in less than 10 minutes.

#### Chain Agnostic
Additionally, BOS is chain-agnostic, making it a flexible solution for developers working with different blockchains.

---

## The Pillars of BOS

BOS is based on three pillars:
- [Components](#components): Composable frontends that solve specific problems.
- [Blockchains](#blockchains): To store the component's code, as well as their assets and data.
- [Gateways](#gateways): A simple way to render components anywhere.

### Components

[Components](home.md) are small web 3 applications (think [Lido](tutorial/hello-lido.md), Uniswap, Aave) that are stored **entirely on-chain**.

Developers can fork these apps and [compose them](./home.md#composing-components) to create full web applications.

### Blockchains

Components can call functions on any blockchain, with current support for all EVM chains (e.g. Polygon, zkSync) and NEAR.

The source code for the apps is on NEAR, due to it's ability to very cheaply store HTML/CSS/JS (a few cents).

### Gateways

Gateways make locally-run, decentralized front-ends available to the masses. A gateway consists of a specially designed virtual machine that loads and runs frontends for protocols built on Ethereum, L2s, and other Layer 1s like NEAR. The code for these frontends is stored on the NEAR blockchain.

Examples of gateways include [near.org](https://near.org), [bos.gg](https://bos.gg), [near.social](https://near.social), [Cantopia](https://cantopia.pages.dev), and [nearpad.dev](https://nearpad.dev).

repository: think-in-universe/docs

repo_name: think-in-universe/docs


text:
# Datenbanken Umfragen Webseite
This repository holds the code for the Baden-Wuerttemberg Cooperative State University (DHBW) Ravensburg project for the lecture *Datenbanken*.

## Documentation
This web-application is used for the lecture *Datenbanken* and the code is documented.

## Technologies
The web-application uses the following technology stack:
* HTML
* PHP
* MySQL/MariaDB

## Authors
* **Lukas Fink** - *Implementation / Planning / Concept* - [xSpeed1337](https://github.com/xSpeed1337)
* **Elena Deckert** - *Implementation / Planning / Concept* - [ElenaDeckert](https://github.com/ElenaDeckert)
* **Antonia Gabriel** - *Implementation / Planning / Concept* - [A1Gabriel](https://github.com/A1Gabriel)

## License

This project is licensed under the Apache 2.0 License - see the [LICENSE.md](LICENSE.md) file for details.

## Acknowledgments
* Was created as a Baden-Wuerttemberg Cooperative State University (DHBW) Ravensburg project for the lecture *Datenbanken*
* Thanks to my teammates Elena and Antonia

gha_language: PHP


text:
---
title: Free digital certificates from StartCom/StartSSL
categories: IT Related
tags: Useful Links
date: 2007-04-19 12:52:32 +10:00
---

I was doing some reading a while ago about digital certificates. The kind of certificate I was after was so that I could use HTTPS in IIS. It looked like the certificates were all very expensive for what they are. I finally came across a post that referred to StartCom. These guys offer free digital certificates for domain or email validation. At this stage, the only hitch is that the CA certificate is not on client machines by default, but can be installed from the StartCom site.

Check it out [here][0].

[0]: http://cert.startcom.org/

license: MIT,Unlicense

detected_licenses: MIT Unlicense

gha_language: SCSS


text:
# Text Analysis of Hamlet Book

## Requirements
- Python 3
- num2words
- nltk
- Sklearn
- gensim (Word2Vec)

## Description

This project is in charge of learning to work with text when manipulating words as numbers to put the data in the correct way to work with machine/deep learning techniques. We processed big files of text (books) in order to clean the text, remove strange characters and punctuations and convert all the words in a suitable form to use bag of words or word2vec techniques.

One book have been proposed (Hamlet.txt). The main idea is to clean the text and analyse it using the techniques proposed in the description. To be, or not to be, that is the question...


text:
# Reactive Systems Plugin

This plugin contains some pre-made systems that cover common use cases where you want to use rx streams to govern your system processing.

## ISetupSystem

This interface implies that you want to setup entities, so it will match all entities via the group and will run a `Setup` method once for each of the entities. This is primarily there for doing one off setup methods on entities, such as instantiating `GameObject` or complex object types.

## ITeardownSystem

This is similar to `ISetupSystem`, but is used when a matched entity's group is removed.

## IReactToEntitySystem

This interface implies that you want to react to individual changes in an entity. It will pass each entity to the `ReactToEntity` method to setup the observable you want, such as Health changing, input occurring, random intervals etc. This only happens once per matched entity, here is an example of the sort of thing you would do here:

```csharp
public IObservable<IEntity> ReactToEntity(IEntity entity)
{
    var colorComponent = entity.GetComponent<RandomColorComponent>();
    return colorComponent.Color.DistinctUntilChanged().Select(x => entity);
}
```

Once you have setup your reactions the `Process` method is triggered every time the subscription from the reaction phase is triggered, so this way your system reacts to data rather than polling for changes each frame, this makes system logic for succinct and direct, it also can make quite complex scenarios quite simple as you can use the power of **Rx** to daisy chain together your observables to trigger whatever you want.

It is also worth looking at the groups documentation as there are some features in groups which allow you to automatically constrain entities based upon predicates so that can push some constraining logic up to the *group* rather than your specific system, however sometimes it makes sense to be in the system, so its your call.

## IReactToGroupSystem

This is like the `IReactToEntitySystem` but rather than reacting to each entity matched, it instead just reacts to something at the group level. The `ReactToGroup` is generally used as a way to process all entities every frame using `Observable.EveryUpdate()` and selecting the group, however you can do many other things such as reacting to events at a group level or some other observable notion, here is a simple example:

```csharp
public IObservable<GroupAccessor> ReactToGroup(GroupAccessor group)
{
    return Observable.EveryUpdate().Select(x => group);
}
```

The main benefit of this interface vs the `IReactToEntitySystem` approach is that this one will only generate a single subscription to trigger all entities in the group to be processed, where the other interface would generate a subscription per entity, so this is a much more performant way of reacting to the same thing for the entire group.

## IReactToDataSystem<T>

So this is the more complicated and lesser used flavour of system. It is basically the same as the `IReactToEntitySystem` however it reacts to data rather than an entity. So for example lets say you wanted to react to a collision event and your system wanted to know about the entity as normal, but also the collision event that occurred. This system is the way you would do that, as its subscription passes back some data rather than an entity, here is an example:

```csharp
IObservable<CollisionEvent> ReactToEntity(IEntity entity)
{
    return MessageBroker.Receive<EntityCollisionEvent>().Single(x => x.collidee == entity);
}
```

So this offers a bit more power as the `Process` method takes both the entity in the pool and the returned data from the subscription allowing you to work with external data when processing.

This is still a fairly new system type so will possibly have some minor changes as we move forward.

### EventReactionSystem<T>

This is a custom implementation of the IManualSystem which acts as a shorthand way of waiting for an event then
once it occurs running some logic based purely upon the logic within that event. (However you can override
it a bit to provide additional entities outside of the group if needed).

```csharp
public class SomeEventSystem : EventReactionSystem<SomeEvent>
{
    public SomeEventSystem(IEventSystem eventSystem) : base(eventSystem)
    { }

    public override void EventTriggered(SomeEvent eventData)
    {
        // Do something with eventData
    }
}
```

repository: EcsRx/ecsrx

gha_language: C#

repo_name: EcsRx/ecsrx


text:
---
layout: post
title: Second Post
summary: New post written completely from JekyllEx.
categories: jekyllex
---

Hello there!

This is a post that I wrote completely from the JekyllEx Android app without even touching my github repository.

**Isn't that cool?**

JekyllEx(built by [Gourav](https://github.com/gouravkhunger)) is the easiest way to manage your jekyll blog directly from you android device!

JekyllEx is launching soon, stay updated at the [Genics Blog](https://genicsblog.com).

---------

Date: 05-09-2021

This section was added from the JekyllEx Android app to make a _demo video_...

repository: jekyllex/demo-jekyll-blog

repo_name: jekyllex/demo-jekyll-blog


text:
go: update dependencies depending on<EMAIL_ADDRESS>
Due to a vulnerability in<EMAIL_ADDRESS>CWE-190.

Updated libraries:

-<EMAIL_ADDRESS>to<EMAIL_ADDRESS>
-<EMAIL_ADDRESS>to<EMAIL_ADDRESS>
- replace `github.com/gorilla/websocket` with
`github.com/gorilla/websocket v1.4.2`

repository: sshyran/oasis-core

repo_name: sshyran/oasis-core


text:
---
title: "About"
description: ""
featured_image: ''
---
{{< figure src="./../images/gsri.png" title="" >}}

_GSRI_ is an interactive learning experience where you will meet like-minded graduate students and a bevy of mentors representing the UW Libraries, Graduate School Core Programs, and beyond. This experience will utilize multiple online platforms for engagement and interaction including _Zoom_, _Microsoft Teams_, and _this website_. Additional online environments may be leveraged to enhance the learning experience.

repository: liblearn/gsri

gha_language: HTML

repo_name: liblearn/gsri


text:
# Example Paginator Bot

This is just a quick example to demonstrate how the paginator works.

## Setup

```
npm install
```
Update the `config.js` file with your own bot token, client id and guild id.
```
npm run start
```
### ReactionPaginator

### SelectPaginator

## Usage

The following commands are available as demonstrations:

### /advanced-pagination
This is an action row pagination example that uses two select menus and a row of buttons for navigation. It makes API calls to dynamically create page content.

![advanced-pagination-gif](https://i.imgur.com/n9O0OH1.gif)
### /action-row-pagination
This is an action row paghination example that uses both buttons and a select menu for navigation. It makes API calls to dynamically create page content.

![action-row-pagination-gif](https://imgur.com/hidL14k.gif)
### /custom-react-pagination
A customised reaction example.

![react-paginator-gif](https://imgur.com/A9Wj2fX.gif)
### /custom-button-pagination
A customised button example.

![button-paginator-gif](https://imgur.com/Esqo43Z.gif)
### /custom-select-pagination
A customised select example.

![select-paginator-gif](https://imgur.com/lgNuRWC.gif)
### /dynamic-button-pagination
A dynamic button example where the pages are not provided up front.
### /react-pagination
A basic reaction example.
### /button-pagination
A basic button example.
### /select-pagination
A basic select example.


text:
## 安全协议

- [OAuth](http://oauth.net/2/)
- [oath](https://openauthentication.org/specifications-technical-resources/)
- [kerberos](http://web.mit.edu/kerberos/)

# Common Attack Methods 常见攻击方式

## DNS Spoofing

### Principle

- [DNS spoofing/DNS Poisoning](https://en.wiki2.org/wiki/DNS_cache_poisoning)
- [揭露DNS劫持，偷梁换柱的把戏](http://tech.ccidnet.com/zt/dns/)
- [中国DNS缓存污染](http://blog.csdn.net/yyf_ad/article/details/16357473)

### Defense

- [DNS Cache Poisoning And Prevention](https://techtrigger.wordpress.com/2012/01/11/dns-cache-poisoning-and-prevention/)
- [DNSPod](https://www.dnspod.cn/)

## DDOS

### Principle

- [DDos - Distributed denial-of-service](https://en.wiki2.org/wiki/DDos)

### Defense

- [360网站卫士](http://wzb.360.cn/)

## SQL Injection

### Principle

- [SQL injection](https://en.wiki2.org/wiki/SQL_injection)
- [sqlinjection.net](http://www.sqlinjection.net/)

### Defense

- [How To: Protect From SQL Injection in ASP.NET](https://msdn.microsoft.com/en-us/library/ff648339.aspx)
- [Avoid SQL Injection attacks](http://forums.asp.net/p/777624/2942861.aspx#2942861)
- [Coding techniques for protecting against Sql injection](http://forums.asp.net/t/1254125.aspx)

## XSS

### Principle

- [XSS - Cross-site scripting](https://en.wiki2.org/wiki/Cross-site_scripting)

### Defense

- [AntiXSS Library](http://www.asp.net/aspnet/overview/aspnet-and-visual-studio-2012/whats-new#_Toc318097382)
- [AntiXSS Library in .NET v4.5](http://forums.asp.net/t/1863558.aspx?AntiXSS+Library+in+NET+v4+5)
- [.NET AntiXSS Library](https://www.owasp.org/index.php/.NET_AntiXSS_Library)

## CSRF

### Principle

- [CSRF/XSRF - Cross-site request forgery](https://en.wiki2.org/wiki/Cross-site_request_forgery)
- [XSS和CSRF的比较](http://www.cnblogs.com/songhan/archive/2012/07/16/2592917.html)

### Defense

- [csurf](https://www.npmjs.com/package/csurf): Node.js CSRF protection middleware.
- [Preventing Cross-Site Request Forgery (CSRF) Attacks in ASP.NET Web API](http://www.asp.net/web-api/overview/security/preventing-cross-site-request-forgery-attacks)

## HTTP Packet Sniffer

### Principle

- [HTTP Packet Sniffer](https://en.wiki2.org/wiki/Packet_Sniffer)
- [SSL](https://en.wiki2.org/wiki/Transport_Layer_Security):
- [Https(SSL/TLS)原理详解](http://www.fenesky.com/blog/2014/07/19/how-https-works.html)
- [What is an SSL (Secure Sockets Layer) Certificate?](https://www.instantssl.com/ssl.html)

### Defense

Set Up SSL on IIS 7+

- [How to Set Up SSL on IIS 7](http://www.iis.net/learn/manage/configuring-security/how-to-set-up-ssl-on-iis)
- [IIS 7.0 SSL 部署指南](http://www.wosign.com/Support/iis7.html)

Programming

- [在 ASP.NET MVC 中使用 HTTPS (SSL/TLS)](http://www.cnblogs.com/snowdream/archive/2009/08/07/use-https-in-aspnet-mvc.html)
- [SSL pages under ASP.NET MVC](http://stackoverflow.com/questions/156748/ssl-pages-under-asp-net-mvc)

## Man-in-the-middle attack

### Principal

- [Man-in-the-middle attack](https://en.wiki2.org/wiki/Man-in-the-middle_attack)
- [Github被墙事件](https://en.greatfire.org/blog/2013/jan/china-github-and-man-middle)

# Reference

- [The Open Web Application Security Project Foundation](https://www.owasp.org/index.php/Main_Page)
- [SSL 证书服务，大家用哪家的？](http://www.zhihu.com/question/19578422)
- [Why is it possible to sniff an HTTPS / SSL request?](http://security.stackexchange.com/questions/19616/why-is-it-possible-to-sniff-an-https-ssl-request)
- [Hyperfox](https://hyperfox.org/): Hyperfox is a security tool for proxying and recording HTTP and HTTPs communications on a LAN.

repository: EthanCai/programming-learning-notes

repo_name: EthanCai/programming-learning-notes


text:
---
layout: post
title: "Machine Learning"
excerpt: "It can do it for you ... before it does you in"
tags: [Python, Machine Learning]
shorturl: "https://goo.gl/iR7sMJ"
image:
# pic silver robot white skin handshake 1900x500
  feature: https://cloud.githubusercontent.com/assets/300046/14622149/306629f0-0585-11e6-961a-dc8f60dadbf6.jpg
  credit:
  creditlink:
comments: true
---
{{ page.excerpt }}

{% include _toc.html %}

Google CEO Sundar Pichai said in 2016:

> "Machine learning is a core, transformative way by which we’re rethinking how we’re doing everything. We are thoughtfully applying it across all our products, be it search, ads, YouTube, or Play. And we’re in early days, but you will see us — in a systematic way — apply machine learning in all these areas."

Machine learning is a type of AI (Artificial Intelligence)
that enables computers to do things without
being explicitly programmed by human developers.
Rather than explicit programming, Machine Learning algorithms identify
rules through "training” based on many examples.

Recommended by Elon Musk: Nick Bostrom "SuperIntelligence: Paths, Dangers, Strategies"

Computers are also dealing with less structured data than before, such as
(natural language text, images, and videos)
rather than neat rows in tables.

## Skill-building from games

In 1997, when Deep Blue beat world chess champion Gary Kasparov,
it did so by "brute force", by using a supercomputer to
analyze the outcome of every possible move, looking further ahead than any human possibly could.

In 2016, IBM's Watson software beat top Jeopardy game champions
by "learning" from books and encyclopedias.
IBM only created the program that enables the computer to learn.
The software makes use of a "model" from example
inputs to make predictions
vs. following strictly static program instructions (logic defined by human developers).

In 2017, the top ranked player in the Chinese game Go was

defeated by Google's AlphaGo, which is based on Google's DeepMind acquisition.
The software made moves that many considered illogical.
BTW, Go is considered the most complex game ever invented.
Whereas chess players have, at any given turn, an average of 35 possible moves,
on a Go board's 19-by-19 grid, there are 250 possible moves.

Also in 2017, all top-ranked poker players were

bested by software named Libratus from Tuomas Sandholm at CMU.
The software adjusted its strategies during the tournament.
And its algorithms for strategy and negotiation are game-independent,
meaning they're not just about poker, but a range of adversarial problems.

21 August 2017, Elon Musk tweets: "OpenAI first ever to defeat world's best players in competitive eSports [dota2]. Vastly more complex than traditional board games like Chess & Go". VIDEO.

Todai Robot scored among the upper 20% of students in university entrance exams. Writing essays using a pen on paper. In Japanese. It knows 8,000 Japanese words, 2,000 mathematical axioms, and uses "symbolic computation" for "automatic reasoning" on 15 billion sentences.
Back in 2014.

## Use Cases

For ordinary people:

* Estimate the price of a house given real estate data (multiple regression)
* Classify movie reviews from imdb.com into positive and negative (binary) categories
* Classify news wire articles by topic (multi-class classification)

For small businesses:

* Sort vegetables using computer vision

For enterprises:

* Data Security - identify malware by detecting minute variations in file signatures.
* Fraud detection - fight money laundering by finding anomalies in transactions.
* Financial trading
* Health care - understand risk factors and spot patterns in medical info
* Marketing - personalize ads based on interests identified for individuals
* Smart cars
* Insurance - identify risks in smaller populations

## Algorithms

Use of hard-coded (static) "rules" crafted by human programmers is
called "symbolic AI", used in "expert systems" fashionable during the 1980s.

Machine learning algorithms identify information from data fed through
"generic" (general purpose) algorithms which build their own logic from detecting patterns within the data.

Patterns are recognized by neural network algorithms. This cycle of "learning" is implicit in a definition of Machine Learning by Mitchell (in 1997): "A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experience E".

The "network" in a neural network has multiple "layers" or data-processing modules that "distill" data.
Models using a small number of layers are called "shallow learning”.
GoogleNet in 2014 used 22 layers. In 2016, one network uses over 1,000 layers.

By contrast, the human brain has 300 million modules that recognize patterns,
each having 100 neurons.

At the input layer,
the network trains on a specific set of "features"
and then sends that information to the next layer.
That next layer combines previous conclusions with other features and passes it to the next layer, and so on, sequentially.

![ml-model-google-878x356-61242](https://user-images.githubusercontent.com/300046/30997127-ee8083b6-a482-11e7-8b7a-0dd4ea518fdc.jpg)
*

Models are trained using examples so they can be used to make predictions. The result of training is a mathematical model used to make predictions, such as taking an unlabled photo and deriving the label "cat".

## Terminology

An example of the last layer which presents the output/decision
(from recognizing what number is written down in a photo from the MNIST collection)
is a 10-way "softmax" layer returning an array of 10 probability scores (all summing to 1).
Each of the 10 contains the probability that the digit image belongs to one of
10 digit classes.

"Supervised" Machine Learning systems learn how to combine input to produce useful predictions on never-before-seen data.

Machine learning uses some terms that have alternate meanings for words also used by traditional programmers and statisticians:

* (In statistics, a "target" is called a dependent variable.)
   In machine learning, a target is also called a label,
   what a model should ideally have predicted,
   according to an external source of data.

* A "label" is a specific instance of a class.
   A label can be an answer (target) for a prediction task --
   either the answer produced by a machine learning
   system, or the right answer supplied in training data.
   It's represented as "y" in mathematical notation.

* Data points are called "samples" or "examples", represented as "x" in mathematical notation.

* Unlabeled examples are used for making predictions on new data.

* A "category" of a classification problem is called a class.
   For example, when classifying pictures, "puppy" and "muffin" are two of the classes.
   Each class describes a set of possible labels to choose from.

* A model maps examples to predicted labels.
   It's defined by internal parameters, which are learned.

* The prediction error, also called loss value,
  the measure of the distance between a model’s prediction and the target.

  It's a metric of measurements that you care about,
  which may or may not be directly optimized.

* An instance is the aspect about which you want to make a prediction.
   (An instance is called an "objective" in other fields.)

* A feature is a property of an instance used in a prediction task.
   For example, a web page might
   have a feature "contains the word 'cat'".

   A feature in machine learning is called a "variable" in statistics.
   A feature is an input variable—the x variable in simple linear regression.
   (Feature creation in machine learning is called a "transformation" in statistics.)

* A feature Column is a set of related features,
   such as the set of all possible countries in which users might live.
   An example may have one or more features present in a feature column. A
   feature column is referred to as a “namespace” in the VW system (at Yahoo/Microsoft), or a field.
   Example: An instance (with its features) and a label.

* An objective is a metric that an algorithm is trying to optimize.

* A pipeline is the infrastructure surrounding a machine learning algorithm. A particular pipeline can include:

   1. gathering the data from the front end,
   2. putting it into training data files,
   3. training one or more models, and
   4. exporting the models to production.

## Types of machine learning

### Supervised learning

The supervised learning approach aims to predict an output given an input
by making use of a labeled data set of training data (examples)
to which we know the "answer" or target.
It gets feedback on what is correct or not.
The training correlates features
to outputs in order to predict outputs based on new inputs.

   * For time series analysis, use a "recurrent net".
   Recurrent nets process data that changes with time,
   using a feedback loop that acts as a forecasting engine.

   Recurrent networks have a network topology in which data flow itself is recycled.
   It is used, for example, to generate sequences of words from the vectors describing the "meaning" of pictures. Each word depends on all that have already been produced.

   * For text processing (Sentiment Analysis)
   and speech recognition: use a
   Recurrent Net or a
   RNTN (Recursive Neural Tensor Network)
   that operates at individual character level.
   RNTNs were conceived by Richard Socher of

   MetaMind.io as part of his PhD thesis at Stanford.
   In 2016 it became part of Salesforce Einstein Predictive Services at
   https://metamind.readme.io/v1/docs

   RNTNs are better than feed-forward or recurrent nets with
   data with a hierarchical structure (binary trees),
   such as the parse trees of a group of sentences.

   * For image (object) recognition is DBN (Deep Belief Network) or
   Convolutional Net.

   Convolution means use of the same filter (also called a "kernel")
   across the whole image, just like use of a filter in Photoshop.

   Clarifai uses a convolutional net to recognize things and concepts in a digital image.
   It then presents similar images.

   Kernel methods are a group of algorithms for classification.
   Support Vector Machine (SVM)
   being the best known from Vapnik and Cortes in the early 1990s at Bell Labs and Vapnik and Chervonenkis as early as 1963.
   SVM classifies by finding in training data "decision boundaries" between sets of points belonging to different categories. Classifying new data points involves checking which side of the decision boundary they fall on.

Used in both feed-forward and (unrolled) recurrent networks is

Backpropagation to update weights so networks learn (improve).
*
The Deep Visualization Toolkit illustrates the intermediate process.

### Unsupervised learning #

The "unsupervised" part of unsupervised learning
means it makes use of a dataset without labels.

It uses "feature extraction" to discover a good internal representation of the input.
It identifies the structure of data to solve some given task.

It uses "Pattern recognition" to classify patterns and clusters data.

   * Autoencoders encode their own structure.
   They are feature extractors with the same number of input and output nodes.

   A hidden layer in Autoencoders provides a bottleneck
   of nodes reconstruction of input layer.
   So they can be used for image compression.

   * RBM (Restricted Boltzmann Machine) is an autoencoder
   with one visible and one hidden layer.
   Each visible node connects to all nodes in the hidden layer.

   The "restricted" is because there is no connection among nodes within its layer.

   RBM is like a two-way translator.
   RBM determines the relationship among input features.

   In training loops,
   RBM reconstructs inputs in the backward pass.

   RBM uses a measure called "KL Divergence" that compares actual to recreation of data.
   RBM makes decisions about what features are important based on weights and
   overall bias.

   All this means that data in
   RBM does not need to be labeled.

   As an example, to determine whether an animal is acerous or not acerous (has horns), it
   look at features such as color, number of legs, horns.

   "Principal Component Analysis" is a widely used linear method for finding
   a low-dimensional representation.

### Reinforcement learning

Reinforcement learning does not provide feedback until a goal is achieved.
Its objective is to select an action to maximize payoff.

In Dec 2013, Volodymyr Mnih and others at

DeepMind.com (a small company in London)
uploaded to Arxiv a paper titled

“Playing Atari with Deep Reinforcement Learning”.
The paper describes how a "convolutional neural network" program
learned to play several games on the Atari 2600 console (Pong, Breakout, SpaceInvaders, Seaquest, Beam Rider)
by observing raw screen pixels of 210 × 160.
The program learned by receiving a reward when the game score increased.

This was hailed as "general AI" because the games and the goals in every game were very different but, without any change, learned seven different games, and performed better than some humans in three of them.

*
Their

Feb. 2015 article in Nature gave them widespread attention.
Then Google bought DeepMind (for $400 million).

   * https://www.youtube.com/watch?v=e3Jy2vShroE&index=17&list=PLjJh1vlSEYgvZ3ze_4pxKHNh1g5PId36-

   * https://www.nervanasys.com/demystifying-deep-reinforcement-learning/

### Transfer learning

   * https://www.youtube.com/watch?v=Yx6Wv_SCKjI&index=16&list=PLjJh1vlSEYgvZ3ze_4pxKHNh1g5PId36-

### One-shot learning

(aka Probabilistic Programming)
is where a neural network
learns from one (or a few) examples,
as opposed to a large amount of data.

### Deep Learning

"Deep learning" is a specific subfield of machine learning.
The "deep" in "deep learning" is not about "deeper” understanding, but
about the depth of various filters in a multi-stage information distillation operation.
Each stage extracts some meaning from different representations of the input.

Deep learning learns
all layers of representation "greedily", in parallel at the same time,
rather than each layer in succession.
Intermediate incremental representations are learned jointly.
Each layer is updated to follow the representational needs of layers both
above and below itself.

See https://www.wikiwand.com/en/Deep_learning

## Tools for Machine Learning

From http://www.infoworld.com/article/3163525/analytics/review-the-best-frameworks-for-machine-learning-and-deep-learning.html

Python or R is supported.

Many corporations have efforts using Machine Learning:

SAP's "Leonardo" platform for ML

courseuses
the R language.

Tool:

Neural Network Playground.

### Python

Python is extended by libraries:

* for handling csv,
textblob and other data formats
* for visualization: Matplotlib, Seaborn, Bokeh, Yellowbrick
* for matrix operations,
Numpy (Quickstart)
* for statistics and linear algebra: SciPy,
   tweepy
* for vectorized computing: Pandas and
SFrame, which is not limited to datasets that can fit in memory, so it can deal with large datasets, even on a laptop.
* for Machine Learning, see below:

### Machine Learning frameworks

Scikit-learn 0.18.1
from Scikit-learn

   * http://www.infoworld.com/article/3158509/analytics/review-scikit-learn-shines-for-simpler-machine-learning.html

   * Mature documentation and libraries

   * Python-based, but does not support PyPy compiler

Spark MLlib 2.01
from Apache Software Foundation

   * http://www.infoworld.com/article/3141605/artificial-intelligence/review-spark-lights-up-machine-learning.html

   * Written in Scala and uses the linear algebra package Breeze which uses netlib-java.

   * Get data easily from Spark big-data clusters

   * Supported in the Databricks cloud

Keras (at https://keras.io)
is a model-level modular library,
providing high-level building blocks for developing deep learning models.

   * Keras runs on both CPU and GPU through
   backend engines Google's TensorFlow
   and Theano developed by the MILA lab at Universite de Montreal.

   * PROTIP: Bypass manual installation by using the Docker image for Keras at

   https://github.com/fchollet/keras/tree/master/docker
   based on Ubuntu.

In 2007, Nvidia launched CUDA, a C++ programming interface for its line of
GPUs (Graphic Processing Units) begins to replace clusters of less efficient CPUs.

   * In 2017, NVIDIA provides GPU supercomputers used in many self-driving cars.

In 2017, basic Python scripting skills suffice to do advanced deep learning research.

### Java in DL4j

Although Python is vastly more popular, there is a Java library
that can run on both Scala and Clojure.

Adam Gibson at
SkyMind
developed
Deeplearning4j (referred to as DL4j)
to be commercial-grade library to run on a distributed, multi-node setup.
It comes with GPU support for distributed training.

The DL4j team built a vectorization library called Canova.

In it one can select values for its hyper parameters.

DL4j supports most of the deep nets – RBM, DBN, Convolutional net, Recurrent net, RNTN, autoencoders, and vanilla MLP.

### Deep Learning Frameworks

![analysis-python-ml-650x248-53428](https://user-images.githubusercontent.com/300046/30065446-164df49c-9212-11e7-88e9-95ba672fc9e2.jpg)
NOTE:
Here is a ranking of Deep Learning frameworks based on GitHub star count.

TensorFlow AI platform from Google

   * http://www.infoworld.com/article/3127397/artificial-intelligence/review-tensorflow-shines-a-light-on-deep-learning.html

   * When running on CPUs, TensorFlow wraps itself over a low-level library for tensor operations called Eigen. BLAS
   * When running on GPUs, TensorFlow wraps itself over a library of optimized deep learning operations called cuDNN (with CUDA drivers developed by NVIDIA).
   * Google TPU (Tensor Processing Units)

   *
   Google's Machine Learning Crash Course

Microsoft Cognitive Toolkit v2.0 Beta 1 (aka CNTK 2), as of 2017:

   * http://www.infoworld.com/article/3138507/artificial-intelligence/review-microsoft-takes-on-tensorflow.html

   * uses a declarative BrainScript neural network configuration language
   * No MacOS support yet
   * CNTK 2 models on GPU-equipped N-series family of Azure Virtual Machines.
   * Azure Machine Learning is part of the larger Microsoft Cortana Analytics Suite offering.

Caffe 1.0 RC3
from Berkeley Artificial Intelligence...

   * http://www.infoworld.com/article/3154273/analytics/review-caffe-deep-learning-conquers-image-classification.html

MXNet v0.7
from Distributed Machine Learning Amazon...

   * http://www.infoworld.com/article/3149598/artificial-intelligence/mxnet-review-amazons-scalable-deep-learning.html

   * Amazon's DNN framework of choice, combines symbolic declaration of neural network geometries with imperative programming of tensor operations.

   * MXNet scales to multiple GPUs across multiple hosts with a near-linear scaling efficiency of 85 percent and boasts excellent development speed, programmability, and portability.

   * Its  dynamic dependency scheduler allows mixing symbolic and imperative programming flavors:
   Python, R, Scala, Julia, and C++

   * trained MXNet models can also be used for prediction in Matlab and JavaScript.

   * Ahead of TensorFlow with embed imperative tensor operations.

## Data representation

Tensors generalize matrices to an arbitrary number of axes (3 or more "dimensions").

A tensor that contains only one number is called a "scalar"
(aka "scalar tensor"), also called a "0D tensor" because
in Numpy the `ndim` attribute of a scalar tensor has 0 axis (ndim == 0).
(Yes, it confuses axis with dimensions)

An array (list) of several numbers is called a vector
or 1D tensor having one axis.
As Lecun explains it:

   A vector of 5 entries is called a "5-dimensional vector".
   Do not confuse a 5-dimensional vector with a 5D tensor!
   A 5D vector has only one axis and has 5 dimensions along its axis,
   while a 5D tensor has 5 axes (and may have any number of dimensions along each axis).

An array of vectors is a matrix, or 2D tensor of two axes.

x = np.array([[1, 2, 3, 4, 5],
              [10, 11, 12, 13, 14],
              [100, 101, 102, 103, 104]])

Each vertical "column" is like a variable. The first one has values 1, 10, and 100 from the three lines (rows), which are akin to a set of observations for a particular point in time.

The "shape" of the above is (3,5) for 3 vectors each containing 5 values.

A 3D tensor is defined as a "cube" of numbers.
The MNIST library of 60,000 sample images, each consisting of 28x28 black-and-white pixels
(each pixel represented by a number range from white to black),
has a shape of (60000,28,28).

4D tensors are used to process matrices of color pictures :

(samples, width, height, channels)

5D tensors are used when processing video data:

(samples, frames, width, height, color_depth).

In Python Numpy, a data type of "float32" or "float64" is a scalar tensor (or scalar array).

"Selecting" specific elements in a tensor is called "tensor slicing".

### Axies for Shape

2D tensors containing simple vector data of shape (samples, features)
are typically processed in "fully-connected" ("densely-connected" or "dense") layers
using the Dense class in Keras.

3D tensors containing sequence data of shape (samples, timesteps, features)
are typically processed by "recurrent" layers such as a LSTM layer.

4D tensors containing image data is typically processed by Convolution2D layers.

5D tensors containing video data use the reshape layer.
See https://github.com/anayebi/keras-extra
for using Extra Layers for Keras to connect a RNN (Recurrent Neural Network)
to a Convolutional Neural Network (CNN) by allowing the CNN layers to be time distributed.

### Pandas

The Padas Python library reads databases directly into
these Pandas structures:

| Dimensions | Name | Description |
| ---------- | ---- | ----------- |
| 1 | "Series" | Indexed 1 dimension data structure |
| 2 | "Timeseries" | Series using time stamps as an index |
| 3 | "DataFrame" | A two-dimensional table |
| 4 | "Panel" | A three-dimensional table data structure |

* A scalar regression task is one where the target is a continuous scalar value.
   For example, when predicting house prices, the different target prices form a continuous space.

* A vector regression task is one where the target is a set of continuous values
  (e.g. a continuous vector) while doing regression against multiple values (e.g. the coordinates of a bounding box in an image).

## Metrics

The goal is to derive models that generalize.

When using its training data, a "loss function"
is used to measure how well a neural network layer meets its objective.
Thus, some call it an "objective function".

It is called a function because it's the result of changes, and thus used to
determine whether changes are moving in the right direction.
Changes are made to minimize the loss function.

The number of data values is typically split into a
training set, a validation set, and a test set. This is so we don't evaluate data
using what was used for training in order to avoid overfitting.

An "optimizer" is the mechanism a network uses to update itself
(based on data received and loss function values).
It implements a specific variant for stochastic gradient descent.

## Gradient Boosting

To address shallow learning problems,
where structured data is available,
"gradient boosting machines" have been used.

Practitioners of gradient boosting make use of the XGB library,
which supports both the two most popular languages of data science: Python and R.

## Introductory Resources

YOUTUBE:

A friendly introduction to Deep Learning and Neural Networks
by Luis Serrano

Machine Learning is Fun! (article on Medium)
self-proclaimed "the world’s easiest introduction to Machine Learning".

Adam Geitgy's introduction of Machine Learning
https://medium.com/@ageitgey/machine-learning-is-fun-80ea3ec3c471/
"for anyone who is curious about machine learning but has no idea where to start"

https://www.youtube.com/watch?v=dYT2LSuemgg
Apr 26, 2017 by Dr. Joseph Reger, CTO of Fujitsu Technology Solutions

Enterprise Machine Learning in a Nutshell
free class from SAP.

## More #

This is one of a series on AI, Machine Learning, Deep Learning, Robotics, and Analytics:

{% include ai_links.html %}


text:
+++
title = "Using IAM and SCP Effectively"
weight = 500

+++
---

## What are we going to do?
This article will explain how to restrict access to the AWS services for your users but still allow them to provision
resources using *launch constraints*.

It will also cover how you can use resource naming conventions and conditional IAM and SCP statements to ensure
users cannot modify AWS Service Catalog products or the resources within them.

## Using launch constraints
You may wish to use a service catalog to provide a set of approved resource templates for consumption.  In this instance,
you may want to allow consumers to provision a product that contains an AWS S3 bucket but you may not want to allow these
users to provision AWS S3 buckets directly via the console or cli.

To achieve this, you could use a launch constraint and an IAM role assumable by the AWS Service Catalog service.  The IAM
role you provide as a launch constraint is used to provision the resources in the product.  The role provisioning the product
must have the *iam:passrole* permission.

## Resource naming conventions along with conditional IAM and SCP statements
When writing your AWS Service Catalog products you can have a parameter to all of your products that is used as a prefix
to the names of the resources provisioned.  You can then use IAM boundaries to prohibit the modifying of all resources
that have the given prefix or you can use a conditional SCP to prohibit the modifying of all resources that have the given
prefix except for the *PuppetRole*.

Within the Service Catalog Tools all provisioning occurs using an IAM role */servicecatalog-puppet/PuppetRole*.

When writing your products you should use globally unique parameter names to avoid clashes.  If you use the same name for
the resource prefixes you can set the parameter value once in the manifest file as a [global parameter](https://aws-service-catalog-puppet.readthedocs.io/en/latest/puppet/designing_your_manifest.html#parameters).

{{% notice note %}}
If you would like to share your experiences please raise a [github issue](https://github.com/aws-samples/aws-service-catalog-tools-workshop/issues)
{{% /notice %}}

gha_language: CSS


text:
---
title: TiDB Computing
summary: Understand the computing layer of the TiDB database.
---

# TiDB Computing

Based on the distributed storage provided by TiKV, TiDB builds the computing engine that combines great capability of transactional processing with that of data analysis. This document starts by introducing a data mapping algorithm that maps data from TiDB database tables to (Key, Value) key-value pairs in TiKV, then introduces how TiDB manages metadata, and finally illustrates the architecture of the TiDB SQL layer.

For the storage solution on which the computing layer is dependent, this document only introduces the row-based storage structure of TiKV. For OLAP services, TiDB introduces a column-based storage solution [TiFlash](/tiflash/tiflash-overview.md) as a TiKV extension.

## Mapping table data to Key-Value

This section describes the scheme for mapping data to (Key, Value) key-value pairs in TiDB. Data to be mapped here includes the following two types:

- Data of each row in the table, hereinafter referred to as table data.
- Data of all indexes in the table, hereinafter referred to as index data.

### Mapping of table data to Key-Value

In a relational database, a table might have many columns. To map the data of each column in a row to a (Key, Value) key-value pair, you need to consider how to construct the Key. First of all, in OLTP scenarios, there are many operations such as adding, deleting, changing, and searching for data on a single or multiple rows, which needs the database to read a row of data quickly. Therefore, each key should have a unique ID (either explicit or implicit) to make it quick to locate. Then, many OLAP queries require a full table scan. If you can encode the keys of all rows in a table into a range, the whole table can be efficiently scanned by range queries.

Based on the considerations above, the mapping of table data to Key-Value in TiDB is designed as follows:

- To ensure that data from the same table is kept together for easy searching, TiDB assigns a table ID to each table represented by `TableID`. Table ID is an integer that is unique throughout the cluster.
- TiDB assigns a row ID, represented by `RowID`, to each row of data in the table. The row ID is also an integer, unique within the table. For row ID, TiDB has made a small optimization: if a table has an integer type primary key, TiDB uses the value of this primary key as the row ID.

Each row of data is encoded as a (Key, Value) key-value pair according to the following rule:

```
Key:   tablePrefix{TableID}_recordPrefixSep{RowID}
Value: [col1, col2, col3, col4]
```

`tablePrefix` and `recordPrefixSep` are both special string constants used to distinguish other data in Key space. The exact values of the string constants are introduced in [Summary of mapping relationships](#summary-of-mapping-relationships).

### Mapping of indexed data to Key-Value

TiDB supports both primary keys and secondary indexes (both unique and non-unique indexes). Similar to the table data mapping scheme, TiDB assigns an index ID to each index of the table represented by `IndexID`.

For primary keys and unique indexes, it is needed to quickly locate the corresponding `RowID` based on the key-value pair, so such a key-value pair is encoded as follows.

```
Key:   tablePrefix{tableID}_indexPrefixSep{indexID}_indexedColumnsValue
Value: RowID
```

For ordinary secondary indexes that do not need to satisfy the uniqueness constraint, a single key might correspond to multiple rows. It needs to query corresponding `RowID` according to the range of keys. Therefore, the key-value pair must be encoded according to the following rule:

```
Key:   tablePrefix{TableID}_indexPrefixSep{IndexID}_indexedColumnsValue_{RowID}
Value: null
```

### Summary of mapping relationships

`tablePrefix`, `recordPrefixSep`, and `indexPrefixSep` in all of the above encoding rules are string constants that are used to distinguish a KV from other data in the Key space, which are defined as follows:

```
tablePrefix     = []byte{'t'}
recordPrefixSep = []byte{'r'}
indexPrefixSep  = []byte{'i'}
```

Also note that in the above encoding schemes, no matter table data or index data key encoding scheme, all rows in a table have the same key prefix, and all data of an index also has the same prefix. Data with the same prefixes are thus arranged together in TiKV's key space. Therefore, by carefully designing the encoding scheme of the suffix part to ensure that the pre-encoding and post-encoding comparisons remain the same, the table data or index data can be stored in TiKV in an ordered manner. Using this encoding scheme, all row data in a table is arranged orderly by `RowID` in the TiKV's key space, and the data of a particular index is also arranged sequentially in the key space according to the specific value of the index data (`indexedColumnsValue`).

### Example of Key-Value mapping relationship

This section shows a simple example for you to understand the Key-Value mapping relationship of TiDB. Suppose the following table exists in TiDB.

```sql
CREATE TABLE User (
     ID int,
     Name varchar(20),
     Role varchar(20),
     Age int,
     PRIMARY KEY (ID),
     KEY idxAge (Age)
);
```

Suppose there are 3 rows of data in the table.

```
1, "TiDB", "SQL Layer", 10
2, "TiKV", "KV Engine", 20
3, "PD", "Manager", 30
```

Each row of data is mapped to a (Key, Value) key-value pair, and the table has an `int` type primary key, so the value of `RowID` is the value of this primary key. Suppose the table's `TableID` is `10`, and then its table data stored on TiKV is:

```
t10_r1 --> ["TiDB", "SQL  Layer", 10]
t10_r2 --> ["TiKV", "KV  Engine", 20]
t10_r3 --> ["PD", " Manager", 30]
```

In addition to the primary key, the table has a non-unique ordinary secondary index, `idxAge`. Suppose the `IndexID` is `1`, and then its index data stored on TiKV is:

```
t10_i1_10_1 --> null
t10_i1_20_2 --> null
t10_i1_30_3 --> null
```

The above example shows the mapping rule from a relational model to a Key-Value model in TiDB, and the consideration behind this mapping scheme.

## Metadata management

Each database and table in TiDB has metadata that indicates its definition and various attributes. This information also needs to be persisted, and TiDB stores this information in TiKV as well.

Each database or table is assigned a unique ID. As the unique identifier, when table data is encoded to Key-Value, this ID is encoded in the Key with the `m_` prefix. This constructs a key-value pair with the serialized metadata stored in it.

In addition, TiDB also uses a dedicated (Key, Value) key-value pair to store the latest version number of structure information of all tables. This key-value pair is global, and its version number is increased by `1` each time the state of the DDL operation changes. TiDB stores this key-value pair persistently in the PD server with the key of `/tidb/ddl/global_schema_version`, and Value is the version number value of the `int64` type. Meanwhile, because TiDB applies schema changes online, it keeps a background thread that constantly checks whether the version number of the table structure information stored in the PD server changes. This thread also ensures that the changes of version can be obtained within a certain period of time.

## SQL layer overview

TiDB's SQL layer, TiDB Server, translates SQL statements into Key-Value operations, forwards the operations to TiKV, the distributed Key-Value storage layer, assembles the results returned by TiKV, and finally returns the query results to the client.

The nodes at this layer are stateless. These nodes themselves do not store data and are completely equivalent.

### SQL computing

The simplest solution to SQL computing is the [mapping of table data to Key-Value](#mapping-of-table-data-to-key-value) as described in the previous section, which maps SQL queries to KV queries, acquires the corresponding data through the KV interface, and performs various computations.

For example, to execute the `select count(*) from user where name = "TiDB"` SQL statement, TiDB needs to read all data in the table, then checks whether the `name` field is `TiDB`, and if so, returns this row. The process is as follows:

1. Construct the Key Range: all `RowID` in a table are in `[0, MaxInt64)` range. According to the row data `Key` encoding rule, using `0` and `MaxInt64` can construct a `[StartKey, EndKey)` range that is left-closed and right-open.
2. Scan Key Range: read the data in TiKV according to the key range constructed above.
3. Filter data: for each row of data read, calculate the `name = "TiDB"` expression. If the result is `true`, return to this row. If not, skip this row.
4. Calculate `Count(*)`: for each row that meets the requirements, add up to the result of `Count(*)`.

**The entire process is illustrated as follows:**

![naive sql flow](/media/tidb-computing-native-sql-flow.jpeg)

This solution is intuitive and feasible, but has some obvious problems in a distributed database scenario:

- As the data is being scanned, each row is read from TiKV via a KV operation with at least one RPC overhead, which can be very high if there is a large amount of data to be scanned.
- It is not applicable to all rows. Data that does not meet the conditions does not need to be read.
- From the returned result of this query, only the number of rows that match the requirements is needed, not the value of those rows.

### Distributed SQL operations

To solve the problems above, the computation should be as close to the storage node as possible to avoid a large number of RPC callings. First of all, the SQL predicate condition `name = "TiDB"` should be pushed down to the storage node for computation, so that only valid rows are returned, which avoids meaningless network transfers. Then, the aggregation function `Count(*)` can also be pushed down to the storage nodes for pre-aggregation, and each node only has to return a result of `Count(*)`. The SQL layer will sum up the `Count(*)` results returned by each node.

The following image shows how data returns layer by layer:

![dist sql flow](/media/tidb-computing-dist-sql-flow.png)

### Architecture of SQL layer

The previous sections introduce some functions of the SQL layer and I hope you have a basic understanding of how SQL statements are handled. In fact, TiDB's SQL layer is much more complicated, with many modules and layers. The following diagram lists the important modules and calling relationships:

![tidb sql layer](/media/tidb-computing-tidb-sql-layer.png)

The user's SQL request is sent to TiDB Server either directly or via `Load Balancer`. TiDB Server will parse `MySQL Protocol Packet`, get the content of requests, parse the SQL request syntactically and semantically, develop and optimize query plans, execute a query plan, get and process the data. All data is stored in the TiKV cluster, so in this process, TiDB Server needs to interact with TiKV and get the data. Finally, TiDB Server needs to return the query results to the user.


text:
# task_stimuli

This software is a set of cognitive tasks developed in psychopy and a system to schedule sets of tasks during a session.

Tasks are classes defined in `src/tasks`, and are instantiated in `src/sessions` files that describe a set of tasks in the session.

Material for the task (images/movies/lists...) is stored mainly in `data`

Outputs (logs, responses) are stored in the `output` folder and try to mimic a BIDS structure.

When used with option `--fmri` tasks waits for a new TTL character to start.

When used with the option `--eyetracking` this software will start Pupil, and trigger the recording of the eye movie and detected pupil position, which outputs to the `output` folder in a BIDS-like way.
Note that eyetracking data would require offline post/re-processing to be used and shared.

`utils` contains scripts to prepare movies in a reproducible way using the melt command line video editor in singularity.

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

## Attributions

...

## INSTALL

```
apt install python3-pip git
mkdir git
cd git

# this section is optional, only if using eyetracking
git clone https://github.com/pupil-labs/pupil.git
# follow instructions at https://docs.pupil-labs.com/#linux-dependencies

pip3 install git+https://github.com/psychopy/psychopy.git
# modify the file in psychopy that crashes
pip3 install scikit-video

git clone git@github.com:courtois-neuromod/task_stimuli.git
cd task_stimuli
mkdir output
```

## how to launch a session

`python3 main.py --subject test --session video003 --tasks videoshorttest --eyetracking --fmri -o /path/to/dataset/`

- --subject: can be whatever, will be used to save data in a bids-like structure
- --session: a session identifier that will be used to save the data in the BIDS
- --tasks: must match the name of a session script in `src/ses-<session_name>.py`, which contains the tasks to be ran on that session
- --eyetracking: turn on eyetracking, start pupil software and recording of eye
- -o : specifies the path to the root of the dataset where to output the data (in sourcedata or BIDS )

- --fmri: will wait for TTL (can be emulated with character `5` on the keyboard) to start the tasks that are labeled as fmri dependent. When not using that flag, tasks will run back to back. It will also append a video loop at the beginning of the session in order for the participant to have sound and visual stimuli to test the setup (then skip to start the session).

- --meg: TODO!

If you run multiple time this command, there are no risks of overwriting, the data will be suffixed by the date and time of start of the session.

## Creating session files

You can create new sessions by adding a `ses-xxxx.py` file in `src/sessions` folder.
Each file only create a `TASKS` list of task subclasses instances, that is loaded by the script and ran in the provided order.
Check the existing files for examples.

## How to interact with the software:

### stimuli

- `5`: emulate the trigger of MRI and start task "by hand" (can be changed in `src/shared/fmri.py`)
- `<ctrl>-c` : abort and **skip** the current task and move to the next one
- `<ctrl>-n` : abort the task and restart it, showing again the instruction
- `<ctrl>-q` : quit the session, saves and close the eyetracking software

If (and only if) the software stop responding and you cannot quit, switch to the terminal and kill the software with `<ctrl>-c`.

### eyetracking

There are "hotkeys in the pupil software to trigger actions", use the buttons with these letters or type.
C (<shift>-c): launch the calibration of the eyetracking, showing markers to the participant
T (<shift>-t): a test of the calibration accuracy, also showing markers on the screen

**Important: there are two softwares running, Psychopy and Pupil, when done with calibration, click on the Stimuli window to give the focus back to Psychopy, otherwise it will not get the TTL and the task will not start with the scanner.**

This is a problem that has to be fixed in the future to avoid failed acquisition start.
Update: should be fixed now, the software takes focus when task is loaded.

# source code

psychopy scripts for stimuli presentations

src/tasks contains scripts for tasks

src/shared folder should factorize the code common across tasks

## eyetracking

The eyetracking part is managed by launching pupil capture software and launching a single recording for the whole session.

### calibration

Run a short calibration task where the subjects have to look at points shown on the screen

### gazemapping

Once the calibration has been run (though it seems that pupil reload previous calibration), pupil produces gaze information that corresponds to position on the screen.
We then display that information in almost real-time on the experimenter screen.


text:
---
layout: default
---

# 1997-07-18 - Re: A.Jensen doesn't like Sympatico.ca

## Header Data

From: Toto<EMAIL_ADDRESS>To<EMAIL_ADDRESS>Message Hash: e524bb1969181dcb6231502a342a941937f8898c685736a9ea79ba8ad5be8264

Message ID<EMAIL_ADDRESS>Reply To<EMAIL_ADDRESS>UTC Datetime: 1997-07-18 07:26:01 UTC

Raw Date: Fri, 18 Jul 1997 15:26:01 +0800

## Raw message

```
{% raw  %}From: Toto<EMAIL_ADDRESS>Date: Fri, 18 Jul 1997 15:26:01 +0800
To<EMAIL_ADDRESS>Subject: Re: A.Jensen doesn't like Sympatico.ca
In-Reply-To<EMAIL_ADDRESS>Message-ID<EMAIL_ADDRESS>MIME-Version: 1.0
Content-Type: text/plain

Michael C Taylor wrote:
> On Thu, 17 Jul 1997, Dr.Dimitri Vulis KOTM wrote:
>
> > They hate Toto's ISP:

  So does Toto.

> Funny thing is that Sympatico.ca is NOT some fly-by-night ISP, but one of
> the larger ISPs and has the deepest pockets.
> Each province buys the basic 'francise' from MediaLink (??) in Upper
> Canada, and resells it and packages it how they want.

> The problem is with spam is not they do more of it, they are most
> likely underqualified to fix the problem of non-Sympatico users using the
> various Sympatico SMTP servers as a relay.

  Sympatico is a hacker's dream. Not only are you able to access their
systems all across Canada once you get an inside track into one of them
but most of the provincial admins seem to have little idea of what the
concept of security entails. As long as you remain low-key you can
set up your own little cyberkingdom across the country.
 {If you hack Toto's website you will find a file in the home directory
which explains how to hack Sympatico's mail system. In return Toto asks
that you use unlimited accounts instead of those like his, which cost
the user on a time-of-use basis.}

  Passive-aggressive hacker fans of "The True Story of the InterNet"
sometimes use Toto's account in ways that are meant to cause trouble
for him but he has not received even a whisper from Sympatico admin
despite loud and voriciferous protests from the victims of the hackers'
shenanigans.
  There was an annual MC hacker's contest going on for the last month
on the Sympatico system and none of the Webmasters seemed to notice
even though all the provincial systems were compromised from top to
bottom. When the Evil-1 set up shop on the Sympatico system he had
to go through it and _improve_ the security in order to keep the
neophyte hackers out of his hair.

  The Saskatchewan Sympatico system is the easiest to use for general
purpose mischief. The British Columbia system is best for hiding large
volumes of use and CPU time. The Ontario system is the most fun to
lurk in because all of the Sympatico employees are perverts of one
kind or another and their private email puts the sex sites to shame.
{It's best not to mess with the government accounts until you've
figured out where the Mountie and CIS spooks are lurking but their
monitoring methods are real blatant and easy to route around by using
blind telnet, deleted mail mirroring, etc.}

  Sympatico's motto is "Internet Service For Everyone."
  I doubt that they realize how true it is.

Toto #6UALDV8

{% endraw %}
```

## Thread

+ Return to [July 1997](/archive/1997/07)

+ Return to<EMAIL_ADDRESS>(Dr.Dimitri Vulis KOTM)](/authors/dlv_at_bwalk_dm_com_dr_dimitri_vulis_kotm_)"
+ Return to "[Michael C Taylor <mctaylor@mta.ca>](/authors/michael_c_taylor_mctaylor_at_mta_ca_)"
+ Return to "[Toto <toto@sk.sympatico.ca>](/authors/toto_toto_at_sk_sympatico_ca_)"

+ 1997-07-17 (Fri, 18 Jul 1997 01:39:04 +0800) - [A.Jensen doesn't like Sympatico.ca](/archive/1997/07/b7eada0a07a4cdf44e7e07e622681e6c4e90928c3804987737e842d920707ef2) -<EMAIL_ADDRESS>(Dr.Dimitri Vulis KOTM)_
  + 1997-07-17 (Fri, 18 Jul 1997 04:13:30 +0800) - [Re: A.Jensen doesn't like Sympatico.ca](/archive/1997/07/0968de7a524c87ead618fa34aa265fad20e17c518b1ae301df2a92f962cf0845) - _Michael C Taylor<EMAIL_ADDRESS>    + 1997-07-18 (Fri, 18 Jul 1997 15:26:01 +0800) - Re: A.Jensen doesn't like Sympatico.ca - _Toto<EMAIL_ADDRESS>