text:
---
id: module-shopify
title: Modules-Shopify
permalink: modules/Shopify
sectionid: docs-modules
---

# Shopify

Paymentwall has partnered with Shopify to allow its server owners accept payments easily via Paymentwall using credit cards, SMS payments, prepaid payment methods (such as Paysafecard, TicketSurf, Boleto), bank transfers and eWallets.

To connect Paymentwall with your Shopify store simply follow the 2 easy steps below:

## Project configuration in Paymentwall system
* Create your project with [Shopify connect button](https://api.paymentwall.com/developers/connect?source=shopify).

* You will be taken to **Paymentwall Account Registration Page**. **Sign Up** if you don't have an account or **Sign In** if you already have one. When you **Sign Up**, use your Shopify webstore URL for the **Website URL** field (e.g. *https://mystore.myshopify.net*).

* You will be taken to your merchant area page with your project already set up (a new project will be created if you had an account before).

* Go to **My Projects** tab. You will see that your new project has already been created. **Project Key** and **Secret Key** will be needed later to finish the project setup on Shopify admin panel.

* In **Settings** section, please set **Your API type** to  **Digital Goods**.

* Turn on the **Widget signature is required**.

* Change the **Pingback URL** to your Shopify store URL (e.g. *https://mystore.myshopify.net*)

* Choose the **Pingback Signature Version** to version 3.

  > Remember to save changes at the bottom of **Settings** section.

* In **Widgets** section, create a widget that you prefer. And save changes.

## Setup Paymentwall module on your platform
* Please authorize **Paymentwall** on your Shopify store via this [link](https://www.shopify.com/login?redirect=authorize_gateway%2F1030414).

* After you log in to your shop, you will be prompted to add **Paymentwall** as a gateway.

* Click the **Add payment gateway** button. Paymentwall should be available on your admin Payments settings page.

* In your dashboard, please go to **Settings** --> **Payments** section, and look for **Alternative payments** tab.

* Choose **Paymentwall** from the dropdown list.

* Fill all the required fields and click **Activate** to enable the gateway.

  > The **Project Key** and **Secret Key** can be found in your Paymentwall Account in **My Projects** tab.

## Version support

Paymentwall provide supports for all up-to-date Shopify version.

> Contact [module<EMAIL_ADDRESS>if you find the version of your Shopify module is not supported.

## Next step

You can start to test payments of your projects.

> **[Refer to test payments with test method](/sandbox/test-payment)**

Once you make sure everything works smoothly, please submit your project to go live.

> **[See how to live your project](/go_live-home)**


text:
<!---
Group:drug era
Name:DER16 Counts of drug era records per person
Author:Patrick Ryan
CDM Version: 5.0
-->

# DER16: Counts of drug era records per person

## Description
This query is used to count the number of drug era records (drug_era_id) for all persons. The input to the query is a value (or a comma-separated list of values) for a number of records per person. If the input is ommitted, all possible values are summarized.

## Query
```sql
SELECT
  count(1) AS s_count,
  t.person_id
FROM @cdm.drug_era t
group by t.person_id
having count(1) in (3, 4);
```

## Input

|  Parameter |  Example |  Mandatory |  Notes |
| --- | --- | --- | --- |
| list of drug_type_concept_id | 3, 4 | Yes |   |

## Output

|  Field |  Description |
| --- | --- |
| s_count | number of drug era records for all persons. |
| person_id | Person unique identifier |

## Sample output record

|  Field |  Description |
| --- | --- |
| s_count |  4 |
| person_id | 10015532 |

## Documentation
https://github.com/OHDSI/CommonDataModel/wiki/

repository: mi-erasmusmc/QueryLibrary

branch_name: refs/heads/master

extension: md

gha_language: HTML

license_type: permissive

repo_name: mi-erasmusmc/QueryLibrary


text:
# Docker Node.js Proof of Concept

Proof of concept on how build a docker container for Node.js apps.
This is just for learning a bit about docker. There might be 🐜🐛🐞🕷🐝 and 🐲🐉🀄.

__Makes use of:__

[![](http://dockeri.co/image/mhart/alpine-node)](https://hub.docker.com/r/mhart/alpine-node/)

__License:__

[![license](https://img.shields.io/github/license/mashape/apistatus.svg?maxAge=2592000)](https://fabiantheblind.mit-license.org/)

----

You need Docker installed to use this. Luckily there is a App for Windows & macOS. Get it at [docker.com](https://www.docker.com/). You also should install [Kitematic](https://kitematic.com/) for easier acces to your running containers.

```bash
# Edit the package.json to reflect your project
# e.g. change the name of the container and user in
# scripts
#
# start docker app
# then run:
git clone https://github.com/FH-Potsdam/docker-node-proof-of-concept.git ./docker-poc/
cd docker-poc/
npm run docker:simple
# to get the ID of the container run
# docker ps
# to see the output run
docker logs [CONTAINER ID]
```

open: [localhost:61428](http://localhost:61428) to see the express app.

The above commands:

- build a basic the docker image
- execute the docker container defined in `Dockerfile`

To have a smaller container you can bundle all your *.js files into a single one. This can be done by running:

```bash
npm run docker:browserify
```

it will:

- build the basic image
- bundle the js files into a single bundle.js file using browserify
- remove the node_modules folder
- execute the container defined in `Dockerfile.browserify`

Note: There might be problems with native modules.

## Useful Docker commands

```bash
# list all images
docker images
# list all containers
docker ps
# remove a container
docker rm [CONTAINER ID or TAG]
# reomve an image
docker rmi [IMAGE ID OR TAG]
# stop all running containers
docker stop $(docker ps -a -q)
#
# see what your app is logging
docker logs [CONTAINER ID]

# build from docker file not called Dockerfile
# -t is the tag -f is the filename
#
#  the last argument is the working directory root
docker build -t [USER]/[FANCY NAME] -f My-Special-Dockerfile-name ./
#
# run a container with some useful settings
#
# -p is the port to map first is the actual port second the port the node app uses
#
# -e sets some environment variables
#
# -m the max memory
# Memory limit (format: <number>[<unit>]). Number is a positive integer. Unit can be one of b, k, m, or g. Minimum is 4M.
#
# --memory-swap
# Total memory limit (memory + swap, format: <number>[<unit>]). Number is a positive integer. Unit can be one of b, k, m, or g.
#
# -d sets detached mode
# To start a container in detached mode, you use -d=true or just -d option. By design, containers started in detached mode exit when the root process used to run the container exits. A container in detached mode cannot be automatically removed when it stops, this means you cannot use the --rm option with -d option.
#
docker run -p 61428:8080 -u "app" -e "NODE_ENV=production" -m "300M" --memory-swap "1G" -d user/fancy-name

# see the package.json scripts how they are used in this project
```

## Recommended Readings

- [Dockerfile reference](https://docs.docker.com/engine/reference/builder/)
- [Dockerizing a Node.js web app Node.js](https://nodejs.org/en/docs/guides/nodejs-docker-webapp/)
- [Best practices for writing Dockerfiles](https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/)
- [Best Practices for Non-root User · Issue #48 · mhart/alpine-node · GitHub](https://github.com/mhart/alpine-node/issues/48)
- [Getting Started with Docker Scotch](https://scotch.io/tutorials/getting-started-with-docker)
- [option to not install devDependencies · Issue #1434 · npm/npm · GitHub](https://github.com/npm/npm/issues/1434)
- [docker-node/BestPractices.md at master · nodejs/docker-node · GitHub](https://github.com/nodejs/docker-node/blob/master/docs/BestPractices.md)
- [odino.org || Docker run your Nodejs App in 24mb of an Image](http://odino.org/minimal-docker-run-your-nodejs-app-in-25mb-of-an-image/)
- [Issues · docker/docker-bench-security · GitHub](https://github.com/docker/docker-bench-security/issues)

__nexe (Not working yet)__

- [Docker for Mac: standard_init_linux.go:175: exec user process caused "exec format error" · Issue #23865 · docker/docker · GitHub](https://github.com/docker/docker/issues/23865)
- [deployment - How do I deploy Node.js applications as a single executable file? - Stack Overflow](http://stackoverflow.com/questions/14314038/how-do-i-deploy-node-js-applications-as-a-single-executable-file)

license: MIT

repository: FH-Potsdam/docker-node-proof-of-concept

repo_name: FH-Potsdam/docker-node-proof-of-concept


text:
# binding [![Build Status](https://travis-ci.org/go-macaron/binding.svg?branch=master)](https://travis-ci.org/go-macaron/binding) [![Sourcegraph](https://sourcegraph.com/github.com/go-macaron/binding/-/badge.svg)](https://sourcegraph.com/github.com/go-macaron/binding?badge)

Middleware binding provides request data binding and validation for [Macaron](https://github.com/go-macaron/macaron).

### Installation

	go get github.com/go-macaron/binding

## Getting Help

- [API Reference](https://gowalker.org/github.com/go-macaron/binding)
- [Documentation](http://go-macaron.com/docs/middlewares/binding)

## Credits

This package is a modified version of [martini-contrib/binding](https://github.com/martini-contrib/binding).

## License

This project is under the Apache License, Version 2.0. See the [LICENSE](LICENSE) file for the full license text.

gha_language: TypeScript


text:
# vue-todo

Welcome to the Vue-Todo documentation. This is a single page application written in ES6 which demonstrates Vue.js and other related libraries.

## Summary of Technologies Used

The application has been built using the following key technologies:

- Vue.js
- Vue Router
- Vuex
- Webpack

## Build Setup

``` bash
# install dependencies
npm install

# serve with hot reload at localhost:8080
npm run dev

# build for production with minification
npm run build

# run unit tests
npm run unit

# run e2e tests
npm run e2e

# run all tests
npm test
```

For detailed explanation on how things work, checkout the [guide](http://vuejs-templates.github.io/webpack/) and [docs for vue-loader](http://vuejs.github.io/vue-loader).

## Copyright and Licence
This application has been written by Tom Humphris and is released under the MIT License.

repository: tomeightyeight/vue-todo

repo_name: tomeightyeight/vue-todo


text:
# Azure Workshop Demo Script

This demo script contains short descriptions of demos used in [Microsoft Azure Networking Workshop](https://www.ipspace.net/Microsoft_Azure_Networking_(Workshop)) and [webinar](https://www.ipspace.net/Microsoft_Azure_Networking).

You will need an Azure subscription (account) to work through them.

## Prepare the environment

* Start Azure shell by opening a browser window and logging shell.azure.vm
* Create a set of SSH keys with **ssh-keygen**
* Clone the demo repository with **git clone https://github.com/ipspace/azure**
* Change working directory to **azure**
* Log into Azure (if needed) with **az login** and configure Azure CLI with **az configure**

## List the available Azure locations

```
simple/list-locations
```

The script executes **az account list-locations** command. You can execute various versions of this command, for example using **-o table** option to create a table output.

Note: all demos use scripts with one or several **az** commands. The first command in every script enables bash command tracing so you'll be able to see the exact commands being executed.

## Create a resource group and start a VM

The first demo creates a resource group in East US location (hardcoded in the script) and starts a VM to demonstrate the variety of objects needed to support a single VM. Note that all those objects are created automatically.

The **create-rg** script creates a resource group and saves the resource group name into **~/.rg** file. All subsequent scripts read the resource group name from that file.

```
setup/create-rg Simple
simple/create-vm-a
```

After the VM is created, the Azure CLI prints a JSON object describing the VM. Find the public IP address in that data and log into the VM using **ssh azure@public-ip-address**. Use [Azure Portal](https://portal.azure.com/#home) to explore various objects created to support the VM.

Note: You can also use **. setup/get-public vm-name** command (for example, **. setup/get-public A**) to get the public IP address of a VM from Azure and set a corresponding *bash* environment variable (that's why you need . in front of the command). After using the above command you could SSH into A using **ssh azure@$A**.

### Create another VM in the same subnet

Create the second VM in the same resorce group:

```
simple/create-vm-b
```

Note that while Azure CLI created a new VM, new disk, new public IP address, and new network security group (NSG) it did not create a new virtual network (VNet) or subnet.

## Cleanup

Destroy all objects used in this demo by deleting the resource group with **delete-rg** script or  with **az group delete** command.

```
setup/delete-rg
```

## Create application environments with two subnets

This demo creates:

* A new resource group
* Virtual network *AppNet* with IP address space <IP_ADDRESS>/16 and two subnets: *AppSubnet* (<IP_ADDRESS>/24) and *DBSubnet*
* Create a virtual machine in each subnet.

```
setup/create-rg Net
network/create-vnet
network/create-vm
```

After the virtual machines have been created you can log into the *Web* virtual machine but not in the *DB* virtual machine as it has no permanent public IP address. You can, however, ping the private IP address of *DB* virtual machine from *Web* virtual machine, or reach external destinations from *DB* over TCP or UDP (use **curl** or **wget** from DB to download a few public web pages).

Don't forget to clean up afterwards:

```
setup/delete-rg
```

## Create Network Security Groups

This demo creates:

* Virtual network *AppNet* with the same subnets as the previous demo
* Unprotected VMs in the virtual network (with no per-VM Network Security Group). *Web* VM is reachable from the Internet, *DB* VM has no public IP address and is thus reachable only from within the virtual network

```
setup/create-rg NSG
network/create-vnet
nsg/create-vm
```

After creating the virtual machines, log into the *Web* VM and verify that you can ping *DB* VM and reach it over SSH (you won't be able to log in unless you fix the SSH keys)

Next, create *Web* and *DB* Network Security Groups and apply them to the *Web* and *DB* subnets:

```
nsg/create-web-nsg
nsg/create-db-nsg
nsg/apply-nsg
```

Use **az network nsg list -o table** command to list the contents of the newly-created NSG.

### Test NSG

Log into *Web* VM and check whether you can reach *DB* VM over SSH, ping, or TCP port 80 (using **curl http://db** or **telnet db 80**). Please note that when the attempts to connect to web server on port 80 return an error it's generated by the host (because there's no web server running on the host) not by the Azure virtual network (which quietly drops the packets).

If your NSG doesn't work as expected use the following commands to figure out what the problem might be:

```
az network nsg rule list -g NSG --nsg-name DB-NSG -o table
az network nic list-effective-nsg --resource-group NSG --name DBVMNic -o table
az network nsg rule list -g NSG --nsg-name DB-NSG -o table --include-default
```

Remove SSH entry from DB-NSG using portal. Recheck the connectivity with SSH.

### Fixing the NSG

If you can't fix the NSG for the *DB* subnet yourself, use this command to fix it:

```
nsg/fix-db-nsg
```

### Cleanup

Don't forget to delete the resource group after completing the demo.

```
setup/delete-rg
```

## Application Security Groups Demo

In the Application Security Groups (ASG) demo we'll repeat the steps from the NSG demo but replace the subnet prefixes used in previous demo with ASG objects. The initial steps are almost identical to the previous demo:

* Create a resource group;
* Create a virtual network with two subnets;
* Create an application security group (ASG);
* Create two VMs (one in each ASG)

```
setup/create-rg ASG
network/create-vnet
asg/create-asg
asg/create-vm
```

When creating NSG rules we'll use ASG objects instead of subnet prefixes:

```
asg/create-db-nsg
asg/create-web-nsg
nsg/apply-nsg
```

After setting up the demo, connect to *Web* VM and explore which TCP ports on *DB* VM you can reach. Also, try to figure out which IP addresses belong to a particular ASG (**az network nic list-effective-nsg -g ASG -n DBVMNic** doesn't seem to return usable information).

Don't forget to delete the resource group after completing the demo.

```
setup/delete-rg
```

## Route table demo

This demo will replace the default route table in database subnet with a custom route table where the system default route (pointing to Internet) is replaced with a custom default route with *drop* next hop (effectively blocking Internet access for *DB* VM).

We'll reuse the steps from virtual networking demo to create the virtual network, the two subnets, and the *DB* and *Web* VMs.

```
setup/create-rg rt
network/create-vnet
network/create-vm
```

To test the outbound connectivity from *DB* VM you have to be able to log into it, and you can only do it from the *Web* VM, so you have to copy your private SSH key to the *Web* VM (never do that in production).

```
. setup/get-public Web
scp ~/.ssh/id_rsa azure@$Web:.ssh/
```

Now you can use *Web* VM as a SSH jump host to get to the *DB* VM and test outbound Internet connectivity.

```
ssh azure@$Web
Web> ssh<EMAIL_ADDRESS>DB> curl www.example.com
```

You can display routing table in *DBSubnet* to verify that it includes the default route pointing toward the Internet.

```
az network nic show-effective-route-table -g rt -n DBVMNic -o table
```

Next, create a custom route table and apply it to *DBSubnet*

```
rt/create-rt
rt/apply-rt
```

Check whether the routing table applied to *DBVMNic* has been modified:

```
az network nic show-effective-route-table -g Net -n DBVMNic -o table
```

Finally, log into *DB* VM and try to fetch a web page from the public Internet.

## Peering demo

In the VNet Peering demo we'll create two virtual networks and three VMs (two in network A, one in network B).

```
setup/create-rg peer
peer/create-net-a
peer/create-net-b
peer/create-vm
```

Display private and public VM IP address with these commands:

```
az network nic list -g peer --query "[ [*].name,[*].ipConfigurations[*].privateIpAddress ]" -o table
az vm list -g peer -d -o table
```

Log into A1, try to ping A2 and B1. You should be able to ping A2, but not B1.

### Create network peering

Use these commands to create and verify VNet peering between A and B.

```
peer/create-peer
az network vnet peering list -g peer --vnet-name Net-A -o table
az network vnet peering show --name A2B --vnet-name Net-A --resource-group peer
```

Log into A1, try to ping A2 and B1. You should not be able to ping B1 even though the VNet peering has been successfully established.

### Troubleshooting

Check the routing tables on A1 and B1 NICs:

```
az network nic show-effective-route-table -g peer -n A1VMNic -o table
az network nic show-effective-route-table -g peer -n B1VMNic -o table
```

You'll notice that the routing tables include prefixes from the peered network. Try to figure out why the traffic is not exchanged between A1 and B1 (for example, using *IP Flow Verify* feature in GUI portal).

Check the effective NSG applied to VM NICs, in particular the actual prefixes used in packet filters. You'll notice that the *VirtualNetwork* service tag does not include prefixes from the peered virtual network.

```
az network nic list-effective-nsg -g peer -n A1VMNic
az network nic list-effective-nsg -g peer -n B1VMNic
```

### Enable virtual network access

Fix the NSG by setting the *allowVirtualNetworkAccess* flag on VNet peering.

```
az network vnet peering update --name A2B --vnet-name Net-A --set allowVirtualNetworkAccess=true --resource-group peer
az network vnet peering update --name B2A --vnet-name Net-B --set allowVirtualNetworkAccess=true --resource-group peer
az network nic list-effective-nsg -g peer -n A1VMNic
az network nic list-effective-nsg -g peer -n B1VMNic
```

You might want to change the flag on one side of the peering and check effective NSG on VM NIC before changing the flag on the other side of the peering.

## Cleanup

Congratulations, you completed the last demo. Don't forget to clean up all resources and resource groups you created - Microsoft will happily bill you if you keep them running.

repository: khangvd/pubcloud

path: /Azure/CLI/Demo Script.md

filename: Demo Script.md

repo_name: khangvd/pubcloud


text:
<!-- MarkdownTOC -->

- [Linux Commands Examples](#linux-commands-examples)

<!-- /MarkdownTOC -->

[![commandlinefu](images/commandlinefu.com)](http://commandlinefu.com/)

[![climagic youtube](images/climagic-icon.png)](https://www.youtube.com/user/climagic)

[![linux skill](images/linuxskill.png)](https://www.youtube.com/channel/UCmzTzKZT3yWaIiFvEyIHrOg)

<center>

|[![climagic twiiter](images/climagic.png)](https://twitter.com/climagic)|[![unixtooltip twitter](images/unixtooltip_twitter.png)](https://twitter.com/UnixToolTip)|
|:---:|:---:|

|[![commandlinefu 0 star](images/commandlinefu0star.png)](https://twitter.com/commandlinefu)|[![commandlinefu 3 star](images/commandlinefu3star.png)](https://twitter.com/commandlinefu3)|[![commandlinefu 10 star](images/commandlinefu10star.png)](https://twitter.com/commandlinefu10)|
|:---:|:---:|:---:|

</center>

# Linux Commands Examples
- [Linux Commands Tweets](linux-commands-tweets.md)
- [nixCraft: My 10 UNIX Command Line Mistakes](http://www.cyberciti.biz/tips/my-10-unix-command-line-mistakes.html)

- Make your prompt safer with # so that if you accidentally copy & paste it, it doesn't run. 🌟🌟🌟🌟🌟

		PS1="# $PS1"

- In bash, cd - takes you back to your previous directory.

- Tired of repeatedly pressing 'y' through some shell process? Try the 'yes' command: 🌟

		yes | rm *.txt
		yes | fsck /dev/FOO

- Edit a file on a remote server using vim from your local *nix desktop

		vim<EMAIL_ADDRESS>
- In vim this will search all lines (%s) for wlan0 and replace all the instances per line (g) with eth3 and confirm (c)

		:%s/wlan0/eth3/gc

- Using +F option or pressing F in less is similar to `tail -f filename.log` but can use less's features. 🌟🌟

		less +F filename.log

- Show the first and last 5 lines of the file 'log'.

		(head -5; tail -5) < log

- Make less more like more, but still more than more. Percent of file in prompt, etc.

		export LESS='-sCmqPm--Less--(?eEND:%pb\%.)'

- This environment variable will invoke 'less' with these options when it is used. Like when viewing a man page.

		export LESS="-S -j10 -i"

- Find out which of your directories(below the current directory) occupy at least 1GB of space. 🌟🌟

		du -h . | grep "^[0-9\.]\+G"

- Total bytes used by 5+ year old directories in CWD:

		find . -mtime +$((365*5)) -maxdepth 1 -exec du -sb {} \; |awk '{s+=$1}END{print s}'

- List of empty subdirectories of current directory.

		find . -empty -type d

- Find and long list mp3 files in Music dir older than a year and larger than 10MB.

		find music -name '*.mp3' -mtime +365 -a -size +10M -ls

- Like ping, but it uses an ARP request to check, which gets around host firewalls blocking ICMP. Only works on same subnet

		arping <IP_ADDRESS>

- Show directory size and sort by human readable amount (MB, GB, etc.). Requires GNU sort for -h option. 🌟🌟

		du -sh */ | sort -h

- Show the total space used on all your local disk partitions. 🌟

		df -lP |awk '{sum += $3} END {printf "%d GiB\n", sum/2**20}'

- Print disk space used on all ext3 or 4 FS in GiB.

		df -Pl -t ext3 -t ext4 | tail -n+2 | awk '{ sum+=$3 } END { print sum/2**20 }'

- Quickly remove duplicate lines from a log/text file on Unix/Linux system:

		awk '!seen[$0]++' app.logfile

- Life is too short to run the same command twice.

		export HISTSIZE=0

- Want to increments all numbers in input.txt?

		perl -pe 's/(\d+)/ 1 + $1 /ge' input.txt

- cp - u will only copy files that don't exist, or are newer than their existing counterparts, in the destination directory.
- Print server serial/product/manufacturer:

		for i in serial-number manufacturer product-name; do
		echo "$i $(dmidecode -s system-${i})";
		done

- Copy a file using "ionice -c 3" to give it idle priority to reduce load on the system.

		ionice -c 3 cp vm1.img vm1-clone.img

- Remove the prefix 'unwanted' from the beginning of each filename with .jpg suffix in CWD.

		rename 's/^unwanted//' *.jpg

- Find file duplicates in 'dir' recursively based on size and mdsum and log to dupes.txt

		fdupes -r dir > dupes.txt

- Quickly find the largest 5 files in the CWD tree without crossing filesystem boundaries 🌟🌟🌟

		find . -xdev -ls | sort -n -k 7 | tail -5

- counts files in the current path by modification month

		find . -maxdepth 1 -type f -printf '%TY-%Tm\n' | sort | uniq -c

- Use the */ trick to get only the directories, then use ${dir%/} to remove the trailing / you get

		for dir in */ ; do echo "${dir%/}" ; done

- Move current year pics to 2015 directory.

		find . -maxdepth 1 -daystart -type f -name '*.jpg' -mtime -$( date +%j ) -exec mv -v {} 2015/ \;

- Lower case all files in a folder.

		for f in *; do b=$(echo "$f" | tr '[A-Z]' '[a-z]'); mv "$f" "$b"; done

- Print the day of the year. Can be useful with things like find.

		date +%j

- Make month histogram of dates of files in current directory.

		ls -la --full-time |tr -s " " |cut -f6 -d " "|cut -c1-7 | sort | uniq -c

- Use cut to print out columns 1, 5 and 10 through 15 in data.csv and write that to new.csv

		cut -d, -f1,5,10-15 data.csv > new.csv

- Want to store log of your ssh session? Try:

		ssh -l user server1 | tee -a ~/myssh.log

- Want to find out which file is the oldest in tar ball archive?

		tar -tvf xkcd-tar-bomb.tar.gz | sort -k 4 -r | head

- Add ssh key to remote host. Wrong:

		cat key | ssh usr@box 'cat >> .ssh/authorized_keys'

- Add ssh key to remote host. Correct: 🌟🌟

		ssh-copy-id usr@box

- Create a dynamic SOCKS5 proxy on port 8989 using an SSH connection. Some apps can be configured to use this.

		ssh -D 8989 you@remotehost

- After all the host keys and auth, you'll be on server3.

		ssh -t user1@server1 'ssh -t user2@server2 "ssh -t user3@server3"'

- The awk variable $1 contains the first field in a record, $2 the second, $3 the third, etc. $0 contains all fields.

- Print all from 3rd field to end of line. Very useful for log parsing.

		awk '{ print substr($0, index($0,$3)) }' mail.log

- List top 20 404's URLs in descending order by reqs.

		awk '$9 == "404" {print $7}' access.log |sort|uniq -c|sort -rn| head -n 20

- List 10 largest open file on Unix:

		lsof /|awk '{ if($7>1048576) print $7/1048576 "MB" " " $9 " " $1 }'|sort -nu|tail

- *"Attempt"* to recover an accidentally removed file. 🌟🌟

		fgrep --binary-files=text -C 2000 "string in file" /dev/sda > recovereddata.out

- Replace foo with bar only on lines that contain 'bang'. Use in pipeline or with file args. 🌟

		sed "/bang/ s/foo/bar/"

- Recursive search and replace old with new string, inside files. recursively traverse the directory structure from . down, look for string "oldstring" in all files, and replace it with "newstring", wherever found 🌟🌟

		$ grep -rl oldstring . |xargs sed -i -e 's/oldstring/newstring/'

		also:

		grep -rl oldstring . |xargs perl -pi~ -e 's/oldstring/newstring'

- Watch web server access log for HTTP status code 500 & display entry generated that code: 🌟

		tail -f access.log | awk '$9 == 500 { print $0 }'

- You can actually follow more than one log at once and get new updates on them. Use -q to not print filename header.  🌟

		tail -f *.log

- list top 50 404's in descending order.

		awk '$9 == "404" {print $7}' access.log |sort|uniq -c|sort -rn| head -n 50

- Request by hour graph.

		awk '{print $4}' apache_log|sort -n|cut -c1-15|uniq -c|awk '{b="";for(i=0;i<$1/10;i++){b=b"#"}; print $0 " " b;}'

- Count syslog hits per minute in your messages log file. Useful for doing quick stats.

		awk -F: {'print $1 ":" $2'} messages |uniq -c

- Get list of top URLs from all logs combined 🌟🌟🌟

		zcat access_log*.gz |cat - access_log |awk '{print $7}' |sed 's/\?.*//' |sort|uniq -c|sort -nr

- For Mar 22nd, print the req fields for hosts from two letter gTLD

		fgrep 22/Mar/2015 access_log |awk '$1~/\.[a-z][a-z]$/{print $6 " " $7}'

- Want to extract files to another directory using tar command? Try

		tar xvf file.tar -C /path/to/dir

- Compress files with xz in PWD according to size, starting with smallest.

		ls -Sr1 | while IFS=$'\n' read -r file; do xz "$file"; done

- The utility 'file' reports a file's type. With the option -z it will try to look inside zipped files.

- Compare the contents of 2 dirs. Show only 2 columns, each for files unique to the directory:

		comm -3 <(ls -1 dir1) <(ls -1 dir2)

- Want to see your FreeBSD server cpu temperature? Try:

		sysctl -a |grep temper

- Want to delete a single command from history on a Linux, OS X & Unix Bash shell? 🌟

		history -d N

- Want to resume failed download on Linux/Unix/OSX/BSD?

		curl -LOC - url
		wget -c url

- Quick and easy way to make a mirror of a website

		wget -m http://www.example\.com/

- Is my HTTPD using Gzip?

		curl -sILH 'Accept-Encoding: gzip,deflate' http://cyberciti.biz | grep Content-Encoding

- Want to use the curl command with proxy username/password on Linux/Unix?

		curl -x usr:pwd@proxy:port url

- You can use the text-based web browser to browse the Internet in console

		w3m cyberciti.biz
		lynx cyberciti.biz

- Search (grep) for multiple error message on Linux/Unix: 🌟🌟🌟

		egrep -w 'warning|error|critical' /var/log/messages

- Search for names and build a frequency count for each name.

		egrep -wo "(Donnie|Frank|Roberta|Grandma)" story.txt |sort|uniq -c|sort -r

- Determine if a port is open with bash:

		: </dev/tcp/<IP_ADDRESS>/80

		For times when netcat isn't available.

		Will throw a Connection refused message if a port is closed.

		Scriptable:
		(: </dev/tcp/<IP_ADDRESS>/80) &>/dev/null && echo "OPEN" || echo "CLOSED"

- [The following is Juniper screenOS authentication backdoor - master ssh password:](https://t.co/IQOGT33oTC)

		<<< %s(un='%s') = %u

- [Want to list all iptables rules with line numbers? Try](http://www.cyberciti.biz/faq/linux-viewing-all-iptables-rules-with-numbers-command/)

		iptables -L -n -v --line-numbers

- Capture all port 123 traffic to a file on a Linux/Unix like system:

		tcpdump -s 1514 port 123 -w file

		Read captured traffic:
		tcpdump -r file

- Show only up to the first 10 packets by each source IP:

		tcpdump -nn ip | awk '{s=$3;sub(/\.[0-9]+$/,"",s);if(a[s]++<10){print}}'

- Want to see whether there was an error after executing a command but no error message was displayed? $? shows the exit status. Try

		echo $?

- Want to see disks attached to Linux server? Try

		lsscsi

- Want to list SCSI devices (or hosts) and their attributes under Linux operating systems? Try

		lsscsi -g

- Want to delete existing SAN LUNs on Linux? 🌟

		multipath -f map # Flush a multipath devicemap
		echo 1>/sys/block/sdc/device/delete # Delete sdc

- List both running privileged and unprivileged lxc containers:

		netstat -ax | egrep<EMAIL_ADDRESS>
- Show the TCP and UDP ports being listened on and if you're root, also show the process associated, user, etc. 🌟🌟🌟

		netstat -lepunt

- Want to find out the MAC addresses of the KVM powered VM? Try

		virsh dumpxml VM_NAME | grep 'mac address'

- Want to list block devices on #Linux easily? Try

		lsblk
		lsblk -m
		lsblk -f

- Start a web service on port 8000 that uses the current directory as its document root

		python -m SimpleHTTPServer

- Use perl regex (negative look-behind/look-ahead assertions) to get URLs.

		grep -P -o '(?<=href=")http:\S+(?=")' *.html

- WP abuse

		grep -h "POST /.*wp-login.php" *-access_log |awk '$1!~/^my.ip.addr$/{print $1}' |sort|uniq -c|sort -nr |head -50> wp-abusers.txt

- Make slideshow from *.jpg

		for p in *.jpg; do ffmpeg -loop_input -f image2 -i $p -t 3 -r 4 -s 1080x720 -f avi - >> slides.avi; done

- Wrap the lines of draft.txt at 72 characters wide, doing so at spaces, not middle of word (-s)

		fold -w 72 -s draft.txt > newdraft.txt

- tail log & highlight errors (if your grep supports --color)

		tail -f foo.log|egrep --line-buffered --color=auto 'ERROR|WARN|$'

- Random color per log line.

		tail -F syslog |while read -r line;do printf "\033[38;5;%dm%s\033[0m\n" $(($RANDOM%255)) "$line";done

- Instead of typing the user & group, if they are the same you can just type the user followed by a colon

		chown -R www-data: *

- Show the query and results of 'select' queries going to your mysql server. Won't work on socket conns

		ngrep -d eth0 -i 'select' port 3306

- Show what processes are using port 80 either locally or remotely. Need to be root for unowned processes. 🌟

		lsof -i TCP:80

- On Linux, print out a list of the process IDs that are in the zombie state.

		ps aux | awk '{if ($8=="Z") { print $2 }}'

- What I've done this week

		git log --author=$USER --format="- %B" --since=-7days --reverse |mail -s "What I've done this week"<EMAIL_ADDRESS>
- Which days I've worked...

		git log --date=short --format="%ci"|awk '{print $1}'|uniq

- I find brace expansion useful for renaming files. This cmd expands to "mv Picture.jpg Picture-of-my-cat.jpg"

		mv Picture{,-of-my-cat}.jpg

- Show % reports of CPU statistics for every active task in the server at two second intervals. 🌟

		pidstat 2 5

- Suspend and reattach a process to screen

		longcmd ; [Ctrl-Z] ; bg ; disown ; screen ; reptyr $( pidof longcmd )

- Want to clean metadata from images and other files on Linux? Try

		mat -d file.jpg # Display it
		mat file.jpg # Clean it

- Apparently according to testing, this is the fastest way to delete millions of small files.

		rsync -a -delete empty/ foo/

- See top & bottom 5 log entries on unix/linux

		(echo "--[head]---"; head -5; echo "--[bottom]--"; tail -5) </var/log/httpd/php.log

- Test reachability of a cloud DNS server & measure the round-trip time 🌟

		for i in {1..4};do ping -q -c 4 ns-cloud-c${i}.googledomains.com;done

- To find in which file an event has been logged in use following which will display the last modified logs on linux

		ls -ltr /var/log | tail

- Want to detect execution in a virtualized environment on Linux?

		systemd-detect-virt

		OR

		dmidecode -s system-manufacturer

- TCP vulnerability? Easy to mitigate with ansible:

		- sysctl: name=net.ipv4.tcp_challenge_ack_limit value=999999999 sysctl_set=yes

- Live dmesg output in human readable format (colors, timing)

		dmesg -wH

- List open ports on Linux:

		netstat -an --inet | grep LISTEN | grep -v <IP_ADDRESS>

		ss -l     (all open ports)

		ss -nlp

		ss -4nlp  (ipv4)

		ss -6nlp  (ipv6)

		ss -s     (summary)

		ss -lp | grep 4949  (who is responsible for opening socket/port #4949)

		ss -t -a    (All TCP sockets)

		ss -x -a    (unix sockets)

		ss -4 state time-wait

		ss -o state fin-wait-1

		ss -4 state time-wait -o

- Display HTTP connections:

		ss -o state established '( dport = :http or sport = :http )'

- Compare local and remote log file with vim. Edit a file and show differences:

		vimdiff /path/local/log scp://<IP_ADDRESS>/path/log

- The sed command p prints. For example, print lines 3 through 7 of a file:

		sed -n '3,7p' somefile

- Quickly find the largest 5 files in the CWD tree without crossing filesystem boundaries:

		find . -xdev -ls | sort -n -k 7 | tail -5

- List the 20 largest files or folders under the current working directory:

		du -ma | sort -nr | head -n 20

- Search the file system, but don't descend into the /sys or /proc directories:

		find / \( -path /proc -o -path /sys \) -prune -o -print

- Scan your internal network for hosts listening on TCP port 22 (SSH protocol):

		nmap --open -p T:22 <IP_ADDRESS>/24   (prints closed/filtered ports)
		nmap <IP_ADDRESS>/24 -p22

- Display top bandwidth hogs on website:

		awk '{a[$1] += $10} END {for (h in a) print h " " a[h]}' access_log | sort -k 2 -nr | head -10

- Determine what lines two different files have in common. The comm program requires sorted files:

		comm -12 <(sort names1) <(sort names2)

- Add this to .vimrc and all searches in vim will use "very magic" mode which acts like egrep:

		nnoremap / /\v

- Long list (-l) only the directories in the current directory. .*/ and */ are utilizing your shell's glob matching ability:

		ls -ld .*/ */

- Show only the processes matching httpd, ignoring the line of the grep process itself (regex trick):

		ps auxww | grep "[h]ttpd"

- Show tabs in a file:

		cat -T

- strace is a sysadmin godsend. This will follow pid 927 and its children, writing to "smtpd.<pid>":

		strace -p 927 -o smtpd -ff -tt

- While in #vim insert mode, insert the filename of the current file into the buffer or vim command prompt:

		[Ctrl-R] %

- In #vim, re-select the previous selection:

		gv

- In vim, this will left shift indentation of selected lines by one tab. > for right:

		v (select text with cursor movement) <

- Comment out selected lines in vim using block select mode. :

		[Ctrl-V] (move cursor down across beginning of lines) [I] [#] [Esc]

- In vim, depending on your term color scheme, these can help you w/ syntax highlighting:

		:set bg=dark or :set bg=light

- Open all your JS source files in vim tabs. Then use :tabn and :tabp to switch between them. :tabe newfile to open new tab:

		vim -p *.js

- This vim command will turn off the current search highlighting until the next search. Use :set hlsearch to turn on:

		:noh

- Check net before slow scan:

		for d in {1..254};do ping -c3 <IP_ADDRESS> > /dev/null ||{ echo "VPN is down";break; }; nmap -T1 10.3.0.$d ; done

- You can use the -n option with make to preview what it will do first before actually doing it:

		make -n install

- Who can see your <IP_ADDRESS> requests?:

		traceroute <IP_ADDRESS>|awk -F[\(\)] '$2~/[0-9]/{print $2}'|while read i;do echo $i;geoiplookup $i;done

- The sed command p prints. For example, print lines 3 through 7 of a file:

		sed -n '3,7p' somefile

- Filter out lines containing 'spam':

		sed '/spam/d' somefile

- Show some full file paths referenced in the process table to try to find out where systems are storing data:

		ps auxww | grep -o "/[^\ ]*"

- Did they really only change what they say they did?:

		diff <(docx2txt < agreement.docx) <(docx2txt < newagreement.docx)

- Handy Bash feature: Process Substitution:

		$ some-command <(another-command)
		$ diff <(curl http://somesite/file1) <(curl http://somesite/file2)
<EMAIL_ADDRESS>
- Use the -w option in diff to ignore differences in whitespace (tabs instead of spaces, etc.)

		diff -w index.html bookexample/index.html

- mping nic.uz:

		mping(){ ping $@|awk -F[=\ ] '/time=/{t=$(NF-1);f=2000-14*log(t^18);c="play -q -n synth 1 pl "f"&";print $0;system(c)}';}

- Prefix each line with the local time based on epoch time in 1st column:

		|gawk '{printf("%s %s\n",strftime("%Y-%m-%d_%T", $1),$0)}'

- Optimize all png files for faster loading site:

		find . -type f -iname "*.png" -print0|xargs -I {} -0 optipng "{}"

- Backslashing the * glob instead of quoting the expression:

		find ./music -name \*.mp3 -exec cp {} ./new \;

- Find files under /var/log that are readable by the current user. Takes groups and ACLs into account:

		find /var/log -readable -ls

- Prefix the epoch time in column 1 with the local time:

		tail -f udp.log|gawk '{printf("%s %s\n",strftime("%Y-%m-%d_%T", $1),$0)}'

- Want to display date/time for each bash command history on a Linux/Unix?

		HISTTIMEFORMAT="%d/%m/%y %T "

- Regular Expression: will match one of the first five letters of the alphabet

		[a-eA-E]

- Regular Expression: The -f option in grep lets you specify a file of regular expressions, one per line.

- Regular Expression: The awk variable NR contains the current record number.

- curl supports numeric ranges. This is the full 14 days of unix-mas from 2012:

		curl -Ns http://www.climagic\.org/uxmas/[1-14]

- Skip first 2 fields & print rest of the line on Linux/Unix:

		awk '{ $1=""; $2=""; print}' input

- Automatic Security Updates in RHEL/CentOS 7:

		yum --security update

- The GNU version of date lets you do things like the following:

		date -d "today + 90 days"
		date -d "+90 days"

- Search man pages for a given string:

		man -k string

- Output a file, displaying non-printing characters:

		cat -v
		cat -A

- Print lines of file without printing ones already seen. $0 means whole line in awk. 'a' is an array:

		awk '!a[$0]++' file

- Use jq to pretty print some json data with ANSI color coded syntax and use -R in less to process the color:

		jq -C '.' data.json | less -R

- Replace all variables $val with $pid in getproc.pl, except on commented lines. *FIXED*:

		sed -i -e '/^\s*#/!s/$val\>/$pid/g' getproc.pl

- Gen 20 random users/passwords:

		for i in {1..20} ; do rig|head -1 |tr A-Z a-z;done |while read f l;do echo ${f:0:1}${l}:$(pwgen 12 1);done

- Quickly create a large file on a linux system:

		fallocate -l 10G 10Gigfile
		truncate -s 10M output.file
		truncate -s 10G output.file
		https://www.cyberciti.biz/faq/howto-create-lage-files-with-dd-command/

- How to reboot a systemd linux box forcefully:

		systemctl reboot --force --force

- Shell scripts run in their own subshell, so a cd in a script effects only the subshell. To script a cd, use an alias instead.

- Directly attach screen/tmux over ssh session & save a useless parent bash process:

		ssh -t box screen -r s
		ssh -t box tmux a
		https://github.com/brejoc/ssht

- alias this to `ssh` and it will (re)open a screen session named like your current user on the remote host:

		#!/usr/bin/env bash

		###
		# call `SSH_SCREEN=1234 ./ssh user@host` to connect to screen 1234
		# for example when there are multiple detached sessions with the same name
		###

		#set -o xtrace #debug

		set -o nounset
		set -o errexit
		set -o pipefail

		/usr/bin/ssh -t $@ "screen -R ${SSH_SCREEN:-${USER}}"

- How the Network Time Protocol will handle the leap second:

		2016-12-31 23.59.59
		2016-12-31 23.59.60 <-- leap second
		2017-01-01 00.00.00

- Analyze your whole lastlog to see the different remote hosts for each user:

		last -da | awk '{print $1 " " $NF}' | sort | uniq -c

- Show top count for col3, 2016-07 - 2017-01:

		sed -n '/^2017-01/,/^2016-06/p' logs.csv |sed '$d' |awk -F, '{print $3}' |sort|uniq -c|sort -rn

- grep gziped log files from .gz files in dirs starting with 2017-01-1 & 1 wild character:

		zgrep SSH::Password_Guessing 2017-01-1?/notice.*gz

- Make URLs from data before foo:

		grep -B3 "foo" multilinedata.txt |egrep "^Id:" |while read x x id; do echo "https://hostname/${id}"; done

- Filename with colons giving you problems with scp or rsync? Use the ./ path prefix B4 filename:

		scp ./file:with:colons.txt user@otherhost

- SSH ProxyCommand Examples. Use "ProxyCommand" ~/.ssh/config to easily access servers hidden behind port knocking and jump hosts:

		Host unreachable_host
			ServerAliveInterval 10
			ForwardAgent yes
			ForwardX11 yes
			User username
			IdentityFile ~/.ssh/id_rsa
			#ProxyCommand ssh -i ~/.ssh/id_rsa username@gateway_host exec /usr/bin/nc %h %p
			#Since OpenSSH 5.4, the ssh client has it's own way of reproducing the Netcat behaviour:
			ProxyCommand ssh -i ~/.ssh/id_rsa -W %h:%p username@gateway_host

- Verify ssl cert and get info the CLI:

		true | openssl s_client -connect www\.cyberciti\.biz:443 -showcerts

- Remove wanted Debian/Ubuntu/Mint Linux kernel and free up disk space:

		#Sysadmin apt-get --purge autoremove

- Remove wanted CentOS/RHEL Linux kernel and free up disk space

		yum -y remove $(rpm -qa | grep ^kernel | grep -v $(uname -r) | grep -v kernel-firmware)

		Next to the package-cleanup you can add to /etc/yum.conf:

		installonly_limit=2

- Get a very fast list of the ip addresses of hosts on your local network:

		cat /proc/net/arp | awk '{print $1}'

- List traffic from inside to outside:

		awk '$3~/^172\.16\./ && $5!~/^172\.16\./{printf("%s %s\n",strftime("%Y-%m-%d_%T", $1),$0)}' conn.log

- Know your limits:

		getconf -a | sort -k2nr | grep MAX

[![largest open files](images/largest_open_files.png)](https://twitter.com/nixcraft)

<div class="container">
<iframe width="560" height="315" src="https://www.youtube.com/embed/GaAfhO1kpUk?rel=0" frameborder="0" allowfullscreen class="video"></iframe>
</div>
<br/>

<div class="container">
<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/UvZY7bYt2Lo?rel=0" frameborder="0" allowfullscreen class="video"></iframe>
</div>
<br>

<div class="container">
<iframe src="//www.slideshare.net/slideshow/embed_code/key/zyZZiqphwk4WEs" width="668" height="714" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" allowfullscreen class="video"> </iframe> <div style="margin-bottom:5px"> <strong> <a href="//www.slideshare.net/prakashrockz/red-hat-lvm-cheatsheet" title="Red hat lvm cheatsheet" target="_blank">Red hat lvm cheatsheet</a> </strong> from <strong><a href="//www.slideshare.net/prakashrockz" target="_blank">Prakash Ghosh</a></strong> </div>
</div>
<br/>

<center>
<blockquote class="imgur-embed-pub" lang="en" data-id="a/Ia5mz"><a href="//imgur.com/a/Ia5mz">Got inspired and made this wallpaper today. Hope you enjoy.</a></blockquote><script async src="//s.imgur.com/min/embed.js" charset="utf-8"></script>
</center>

repository: cyrilbois/awesome-tech

gha_license_id: NOASSERTION

repo_name: cyrilbois/awesome-tech


text:
---
layout: post
title: Kindlers
tags:
- game
- unity
- project
date: 2016-01-28 00:00
---

<iframe src="https://player.vimeo.com/video/167897879" width="640" height="360" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe>

### Kindlers is a single-player, puzzle game prototype
<!--more-->

Built for the [Global Game Jam 2016](http://globalgamejam.org/about), Kindlers was Developed over the course of 48 hours. I was the sole programmer and technical artist for a team of five (Game Designer, 3D Modeller, 2D Artist, Sound Designer).

The puzzle mechanic involved aligning multiple fire birds, slightly offset in position and rotation, while each following the same input. The game consisted of five fully playable levels.

### Programming

As the sole programmer on the team, I had a hefty list of tasks:

* Grid-based Movement system
* Building a level creation scene for the designer
* The Gameplay loop (Winning, Losing, ect...)
* Various shaders and effects

![](/blog/assets/kindlers/levelExample02.gif)

The goal of the game was to align each character on glowing targets at the same time

![](/blog/assets/kindlers/character.gif)

I also worked on the cloth simulation for the characters

![](/blog/assets/kindlers/levelExample01.gif)

In addition to programming, I also acted as the Art Director for the game, leading the overall aesthetic choices for the game. While most of the assets were created by teammates, the visuals were assembled and implemented by myself.


text:
[![Build Status](https://travis-ci.org/pezra/hash-selector.svg)](https://travis-ci.org/pezra/hash-selector)

# HashSelector

Select values from deeply nested/complex hashes with ease.

## Usage

Consider a complex and deeply nested hash such as the following:

```ruby
config =
{ databases: [
    { name: "myapp_production",
      user: "myapp",
      host: "db1"
    },
    { name: "legacy_db",
      host: "db2"
    }
  ],
  "redis": { host: "redis_server" }
}
```

Say we want to find the user name of legacy db but if we do not find it use a default value. With just the basic hash behavior this can be rather onerous. With `HashSelector` is it a breeze.

```ruby
selector = HashSelector.new[:databases].find{|db| db[:name] == "myapp_production"}[:user]
selector.find_in(config) # => "myapp"
```
The selector definition line should be read as "a new selector that chooses the `:databases` item, finds an entry in it whose name is `myapp_production`, and selects its `:user`"

If the location specified by a `HashSelector` does not exist the hash it will raise a `KeyError` unless a block is provided. If a block is provided the block will be evaluated and its return value will returned from `#find_in`.

```ruby
selector = HashSelector.new[:databases].find{|db| db[:name] == "myapp_test"}[:user]
selector.find_in(config) { "myapp" } # => "myapp"
```

In this example, there find fails because there is not entry in `:databases` with a name of `myapp_test` so the default (`myapp`) is returned instead.

`HashSelector`s are immutable and may be reused any any number of different hashes.

## Installation

Add this line to your application's Gemfile:

```ruby
gem 'hash-selector'
```

And then execute:

    $ bundle

Or install it yourself as:

    $ gem install hash-selector

## Contributing

1. Fork it ( https://github.com/pezra/hash-selector/fork )
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Add some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
5. Create a new Pull Request

repository: pezra/hash-selector

repo_name: pezra/hash-selector


text:
# Telomere-to-telomere genome for annotating common alleles & abnormally avoided stop-codons

## Contributors
-  Elbay Aliyev - Lead
-  Muhammad Sohail Raza - Writer
-  Shangzhe Zhang, Anastasia Illarionova, Prashant Ranjan, Tiancheng Xu, Aditi Sammi - Tech support
-  Bryce Kille, ChunHsuan LO - SysAdmin

## Introduction

After two decades of refinements, the human reference genome (GRCh38) has become more accurate and mostly complete. However, there are still hundreds of unresolved gaps persist, and no single chromosome has been finished from end to end because of the existence of highly repeated sequences (called transposable elements). Foutunatedly, by using the high-coverage & ultra-long-read technologies, several scientific groups have presented a new human genome assembly that surpasses the continuity of GRCh38, along with a gapless, telomere-to-telomere assembly of a human chromosome, which is called CHM13 reference genome.

## Goal
- To identify rare variants in CHM13 and recorrecting some of them as common variants. By Checking the fasta file directly.
- To screen out in-frame stop codons sites that disagree with Ribo-seq analysis for validting the annotation.
- Propose a more representative reference genome by 1000 Genomes Project

## Flowchart

by Anastasia Illarionova

## Dependancies

- python3
- conda
- snakemake
- [ClinVar](https://www.ncbi.nlm.nih.gov/clinvar/)
- [bcftools](https://samtools.github.io/bcftools/)
- [gffread](https://github.com/gpertea/gffread)
- [GATK](https://gatk.broadinstitute.org)

## Input:

- vcf files (using CHM13_v1.0 as reference already)

  https://s3-us-west-2.amazonaws.com/human-pangenomics/index.html?prefix=T2T/CHM13/assemblies/variants/1000_Genomes_Project/by_chr/
- CHM13_v1.0 Fasta file

  https://s3-us-west-2.amazonaws.com/human-pangenomics/T2T/CHM13/assemblies/chm13.draft_v1.0.fasta.gz

- CHM13_v1.0 GFF file

  https://s3-us-west-2.amazonaws.com/human-pangenomics/T2T/CHM13/assemblies/annotation/chm13.draft_v1.0.gene_annotation.v4.gff3.gz

## Outputs:

- New reference Fasta file (for all samples in 1GP)
- (Optional) New reference Fasta files for 5 subpopulations
- ClinVar annotation of common alleles

## Usage

Install evironment

```
conda env create -f utils.ymal
```
Activate evironment

```
conda activate popchrom
```

Dry run (See what will happen)

```
snakemake -np
```

Use snakemake to run the workfolw

```
snakemake --cores 10
```

## Sample detailed Workflow (chr22)

### I. Data Acquisition and Preprocessing:

1. Downloading the raw CHM13 fasta file.

2. Downloading the raw population based VCF (CHM13 fasta based).

3. Downloading the raw Annotation file (gff3).

### II. Core tasks:

**1.** Identifying stop codon sites in CHM13 fasta file (Shangzhe, Muhamad, Bryce)

  -  Extract the protein sequences from GFF annotation and FASTA file

  ```
  gffread -g chm13.draft_v1.0.fasta chm13.draft_v1.0.gene_annotation.v4.gff3 -y chm13.v1.0.pep.fasta
  ```

  -  Costom script to extract the positions of in-frame stop codons (Only for chr22)

  ```
  python3 determine.in-frame.stop-codon.py chm13.v1.0.pep.fasta chm13.draft_v1.0.gene_annotation.v4.gff3 | grep "chr22" > chm13.draft_v1.0.chr22.in-frame.stop-codon.bed
  ```

**2.** Identifying common variants from Chr22 VCF file (Aditi, Muhamad, Bryce, Tiancheng)

  -  Variant call filtering criteria (SNP, AAF > 5%)

  ```
  bcftools view -i "INFO/AF > 0.05" 1kgp.chr22.recalibrated.snp_indel.pass.withafinfo.vcf > 1kgp.chr22.recalibrated.snp_indel.pass.withafinfo.filtered_5%.vcf
  ```

**3.** Checking for overlaps between common variants from chr22 VCF file with stop codon sites identified from CHM13 fasta. Also, it will be checked if there are inconsistent nonsense variants or ORFs between CHM13 & hg38, which requirs RiboSeq validation (Anastasia, ChunHsuan)

  -  Picking up common variants

  ```
  By checking the variant's allele frequency in population and the mutation type (non-synonymous/synonymous/nonsense).
  ```

  -  Riboseq-validation (for the ORF & inframe-stop-codon sites where variants located)

  ```
  To download paired RNASeq.fastq and RiboSeq.fastq
  ```

  ```
  To quality control fastq files
  (adapter trimming, remove low quality reads, etc.)
  ```

  ```
  To align the reads to CHM13 refgenome
  (get RNASeq.bam and RiboSeq.bam)
  ```

  ```
  To peak calling for read coverage of aligned bam files
  (This will reveal real ORF and stop-codon sites by comparing RNAseq & Riboseq at same time.)
  ```

  ```
  To validate our targeted variant sites by the peak calling results,
  and to clasify them into true ones and false ones.
  ```

**4.** Annotate common variants with ClinVar (Anastasia)

### IIII. Results:

**1.** Statistics of allele frequency

**2.** Postions of in-frame stop codons

    https://github.com/collaborativebioinformatics/popchrom/blob/main/data/chm13.draft_v1.0.chr22.in-frame.stop-codon.bed

**3.** Biologically annotated variants (CHM13 based).

**4.** Statistical visualization

## References

- Sergey Aganezov et.al, A complete reference genome improves analysis of human genetic variation. bioRxiv 2<IP_ADDRESS>2063; doi: https://doi.org/10.1101/2<IP_ADDRESS>2063
- T2T Consortium: https://sites.google.com/ucsc.edu/t2tworkinggroup/technology?authuser=0
- Lifting over variants from GRCh38 to T2T-CHM13: https://github.com/mccoy-lab/t2t-variants/tree/main/liftover_vcfs
- Ribosome-Profiling: https://github.com/FDA/Ribosome-Profiling

## Acknoledgement

- Fritz J. Sedlazeck
- Ben Busby

branch_name: refs/heads/main


text:
## HTTP-DB

http-db is a simple in-memory key-value database. All operations can be performed by the http-methods GET, PUT and DELETE.

There is no build-in authentication and authorization mechanism.

### Examples

Start http-db as a docker container:

```sh
docker run --name=http-db -d -p 8080:8080 psanetra/http-db
```

Save plain text "Hello World" at absolute path /hello/world

```sh
curl -v -XPUT -H 'Content-Type: text/plain' --data "Hello World" http://localhost:8080/hello/world
*   Trying <IP_ADDRESS>...
* Connected to localhost (<IP_ADDRESS>) port 8080 (#0)
> PUT /hello/world HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.47.0
> Accept: */*
> Content-Type: text/plain
> Content-Length: 11
>
* upload completely sent off: 11 out of 11 bytes
< HTTP/1.1 201 Created
< Location: /hello/world
< Date: Sat, 21 Jul 2018 15:51:48 GMT
< Content-Length: 0
<
* Connection #0 to host localhost left intact
```

Get the resource

```sh
curl -v http://localhost:8080/hello/world
*   Trying <IP_ADDRESS>...
* Connected to localhost (<IP_ADDRESS>) port 8080 (#0)
> GET /hello/world HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: text/plain
< X-Content-Type-Options: nosniff
< Date: Sat, 21 Jul 2018 15:56:35 GMT
< Content-Length: 11
<
* Connection #0 to host localhost left intact
Hello World
```

Delete the resource

```sh
curl -v -XDELETE http://localhost:8080/hello/world
*   Trying <IP_ADDRESS>...
* Connected to localhost (<IP_ADDRESS>) port 8080 (#0)
> DELETE /hello/world HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 204 No Content
< Date: Sat, 21 Jul 2018 15:58:53 GMT
<
* Connection #0 to host localhost left intact
```

repository: PSanetra/http-db

repo_name: PSanetra/http-db


text:
# getpivots (pivot-point-calculator)
=====================================================

Getpivots can be used for identifying areas of support and resistance for the next day's trading session.  Getpivots calculates pivot points for a given ticker symbol using the most recent closing price data, and leverages several different algorithms which include the Floor/Classic formula, Woodie’s Formula, as well as Kirk's formula.

![getpivots-screencap1](https://cloud.githubusercontent.com/assets/12847315/11052885/6ae05404-872a-11e5-91c1-13bfb17f50d2.jpg)

Install
-------

    pip install getpivots

Usage
-----

    getpivots -t tickersymbol -f -w -k -c

Options
-----
    -h, --help          Print this help text and exit
    -t, --ticker        Specify a ticker symbol to run against (e.g. YHOO)
    -f, --floor         Uses the Floor/Classic calculation
    -w, --woodie		Uses the Woddy calculation
    -k, --kirk			Uses Kirk's calculation (thekirkreport.com)
    -c, --current       Inserts indicator for current price if run during an active session (e.g. -> 123.00)

See Also
--------

For more information: https://pypi.python.org/pypi/getpivots


text:
## LPegLabel - Parsing Expression Grammars (with Labels) for Lua

---

### Introduction

LPegLabel is a conservative extension of the
[LPeg](http://www.inf.puc-rio.br/~roberto/lpeg)
library that provides an implementation of Parsing
Expression Grammars (PEGs) with labeled failures.
Labels can be used to signal different kinds of errors
and to specify which recovery pattern should handle a
given label. Labels can also be combined with the standard
patterns of LPeg.

Besides that, LPegLabel also reports the farthest
failure position in case of an ordinary failure
(which is represented by label **fail**).

This document describes the new functions available
in LpegLabel and presents some examples of usage.

With labeled failures it is possible to distinguish
between an ordinary failure and an error. Usually, an
ordinary failure is produced when the matching of a
character fails, and this failure is caught by ordered choice.
An error (a non-ordinary failure), by its turn, is produced
by the throw operator and may be caught by a recovery rule.

In LPegLabel, the result of an unsuccessful matching
is a triple **nil, lab, errpos**, where **lab**
is the label associated with the failure (a string or
an integer), and **errpos** is the input position being
matched when **lab** was thrown.

When **lab** is an ordinary failure and no error was thrown before,
**errpos** is the farthest position where an ordinary failure occurred.
In case **lab** is an ordinary failure and an error
was thrown before, **errpos** is the farthest input position
where an ordinary failure occurred after the last error.

Below there is a brief summary of the new functions provided by LpegLabel:

Function	Description

lpeglabel.T (l)
  Throws a label l to signal an error

%{l}
  Syntax of relabel module. Equivalent to lpeglabel.T(l)

p^l
  Syntax sugar available at relabel for p / %{l}

relabel.calcline(subject, i)
  Calculates line and column information regarding position i of the subject

### Functions

#### lpeglabel.T(l)

Returns a pattern that throws the label `l`, which
can be an integer or a string.

When a label is thrown, the current subject position
is used to set **errpos**, no matter whether it is the
fartherst failure position or not.

In case the PEG grammar has a rule `l`, after a label is thrown
this rule will be used as a recovery rule, otherwise the whole
matching fails.

The recovery rule will try to match the input from the subject
position where `l` was thrown. In case the matching of the recovery
rule succeeds, the regular matching is resumed. Otherwise, the
result of the recovery rule is the matching result.

When we have a predicate such as `-p` or `#p` and a label `l` is thrown
during the matching of `p`, this causes the failure of `p`, but does
not propagate `l`, or calls its associated recovery rule.

#### %{l}

Syntax of *relabel* module. Equivalent to `lpeglabel.T(l)`.

Label `l` must be a valid identifier name.

#### p^l

Syntax of *relabel* module. The pattern `p^l` is equivalent
to `p + lpeglabel.T(l)`.

Label `l` must be a valid identifier name.

#### relabel.calcline (subject, i)

Returns line and column information regarding position i of the subject.

### Examples

Below there a few examples of usage of LPegLabel.
The code of these and of other examples is available
in the *examples* directory.

#### Reporting the farthest failure

This example illustrates the new values returned
by the *match* function in case of an unsuccessful
matching. As no error is thrown in this example,
when the matching fails *errpos* represents the
farthest suffix where an ordinary failure occurred.

```lua
local m = require'lpeglabel'

function matchPrint(p, s)
  local r, lab, errpos = p:match(s)
  print("r: ", r, "lab: ", lab, "errpos: ", errpos)
end

local p = m.P"a"^0 * m.P"b" + m.P"c"
matchPrint(p, "abc")  --> r: 3    lab: nil   errpos: nil
matchPrint(p, "c")    --> r: 2    lab: nil   errpos: nil
matchPrint(p, "aac")  --> r: nil  lab: fail  errpos: 3
matchPrint(p, "xxc")  --> r: nil  lab: fail  errpos: 1
```

#### Matching a list of identifiers separated by commas

The following example defines a grammar that matches
a (possibly empty) list of identifiers separated by commas.
A label is thrown when there is no identifier after a comma,
or when the whole input is not matched.

```lua
local m = require'lpeglabel'
local re = require'relabel'

local terror = {
  ErrId   =  "expecting an identifier",
  ErrEnd  =  "expecting EOF",
  fail    =  "undefined"
}

local id = m.R'az'^1

local g = m.P{
  'S',
  S      =  m.V'List' * (-m.P(1) + m.T'ErrEnd'),
  List   =  m.V'Id' * (m.V'Comma' * (m.V'Id' + m.T'ErrId'))^0,
  Id     =  m.V'Sp' * id,
  Comma  =  m.V'Sp' * ',',
  Sp     =  m.S' \n\t'^0,
}

function mymatch (g, s)
  local r, e, pos = g:match(s)
  if not r then
    local line, col = re.calcline(s, pos)
    local msg = "Error at line " .. line .. " (col " .. col .. "): "
    return r, msg .. terror[e] .. " before '" .. s:sub(pos) .. "'"
  end
  return r
end

print(mymatch(g, "one,two"))              --> 8
print(mymatch(g, "one two"))              --> nil  Error at line 1 (col 4): expecting EOF before ' two'
print(mymatch(g, "one,\n two,\nthree,4")) --> nil  Error at line 3 (col 7): expecting an identifier before '4'
```

In this example we could think about writing rule List as follows:
```lua
List = m.V'Id' * ((m.V'Comma' + m.T'ErrComma') * (m.V'Id' + m.T'ErrId'))^0,
```

but when matching m.V'Comma' + m.T'ErrComma' against the end of input
we would get a failure whose associated label would be **errComma**,
and this would cause the failure of the *whole* repetition.

Below we rewrite the previous grammar to indicate an error when there is no
comma after an identifer. Before tyring to match a comma, we check if
we have reached the end of input:

```lua
local m = require'lpeglabel'
local re = require'relabel'

local terror = {
  ErrId     =  "expecting an identifier",
  ErrComma  =  "expecting ','",
  fail      =  "undefined"
}

local id = m.R'az'^1

local g = m.P{
  'S',
  S      =  m.V'List',
  List   =  m.V'Id' * (#m.P(1) * m.V'Comma' * (m.V'Id' + m.T'ErrId'))^0,
  Id     =  m.V'Sp' * id,
  Comma  =  m.V'Sp' * ',' + m.T'ErrComma',
  Sp     =  m.S' \n\t'^0,
}

function mymatch (g, s)
  local r, e, pos = g:match(s)
  if not r then
    local line, col = re.calcline(s, pos)
    local msg = "Error at line " .. line .. " (col " .. col .. "): "
    return r, msg .. terror[e] .. " before '" .. s:sub(pos) .. "'"
  end
  return r
end

print(mymatch(g, "one,two"))               --> 8
print(mymatch(g, "one two"))               --> nil  Error at line 1 (col 4): expecting ',' before ' two'
print(mymatch(g, "one,\n two,\nthree,4"))  --> nil  Error at line 3 (col 7): expecting an identifier before '4'
print(mymatch(g, " 1,2"))                  --> nil  Error at line 1 (col 2): undefined before '1,2'
```

#### Error Recovery

We can specify a recovery rule that should
be matched when a label is thrown. After matching
the recovery rule, and possibly recording the error,
the parser will resume the regular matching.
The recovery rule must have the same name (or number)
of the label that was thrown.

For example, in the example below we expect to match rule *A*,
but when a failure occur the label `Err` is thrown and then we
will try to match rule *Err*:
```lua
local m = require'lpeglabel'

local recp = m.P"oast"

local g = m.P{
  'S',
  S    =  m.V'A' * '.',
  A    =  m.P't' * (m.P'est' + m.T'Err'),
  Err  =  m.P'oast'
}

print(g:match("test."))   --> 6
print(g:match("toast."))  --> 7
print(g:match("oast."))   --> nil  fail  oast.
print(g:match("toward.")) --> nil  fail  ward.
```
When trying to match subject 'toast.', in rule *A* the first
't' is matched, then the matching of `m.P"est"` fails and label
`Err` is thrown, with the associated inpux suffix 'oast.'.
The recovery rule *Err* successfully matches 'oast', so
the regular matching continues, and pattern `'.'` matches
the rest of the input.

When matching subject 'oast.', pattern `m.P"t"` fails, and
the result of the matching is nil,  fail, 1.

When matching 'toward.', label `Err` is thrown after matching 't',
with the associated input suffix 'oward.'. As the matching of the
recovery pattern fails, the result is nil, fail, 3.

Usually, the recovery pattern is an expression that does not fail.
In the previous example, we could have used `(m.P(1) - m.P".")^0`
as the recovery pattern.

Below we rewrite the grammar that describes a list of identifiers
to use a recovery strategy, with the help of some auxiliary functions.
Function `record`, plus function `recorderror`, will help
us to save the input position where a label was thrown,
while function `sync` will give us a synchronization pattern,
that consumes the input while it is not possible to match a given
pattern `p`.

When the matching of an identifier fails, a defaul value ('NONE')
is provided.

```lua
local m = require'lpeglabel'
local re = require'relabel'

local terror = {
  ErrId     =  "expecting an identifier",
  ErrComma  =  "expecting ','",
  ErrList   =  "expecting a list of identifiers",
  fail      =  "undefined"
}

local subject, errors

function recorderror(pos, lab)
  local line, col = re.calcline(subject, pos)
  table.insert(errors, { line = line, col = col, msg = terror[lab] })
end

function record (lab)
  return (m.Cp() * m.Cc(lab)) / recorderror
end

function sync (p)
  return (-p * m.P(1))^0
end

function defaultValue ()
  return m.Cc"NONE"
end

local id = m.R'az'^1

local g = m.P{
  "S",
  S         =  m.V"List" + (m.P(1) * m.T'ErrList'),
  List      =  m.V'Id' * (#m.P(1) * m.V'Comma' * (m.V'Id' + m.T'ErrId'))^0,
  Id        =  m.V'Sp' * m.C(id),
  Comma     =  m.V'Sp' * ',' + m.T'ErrComma',
  Sp        =  m.S' \n\t'^0,
  ErrId     =  record'ErrId' * sync(m.P",") * defaultValue(),
  ErrComma  =  record'ErrComma' * sync(id),
  ErrList   =  record'ErrList' * sync(m.P(-1)) * defaultValue()
}

function mymatch (g, s)
  errors = {}
  subject = s
  io.write("Input: ", s, "\n")
  local r = { g:match(s) }
  io.write("Captures (separated by ';'): ")
  for k, v in pairs(r) do
    io.write(v .. "; ")
  end
  io.write("\nSyntactic errors found: " .. #errors)
  if #errors > 0 then
    io.write("\n")
    local out = {}
    for i, err in ipairs(errors) do
      local msg = "Error at line " .. err.line .. " (col " .. err.col .. "): " .. err.msg
      table.insert(out,  msg)
    end
    io.write(table.concat(out, "\n"))
  end
  print("\n")
  return r
end

mymatch(g, "one,two")
--> Captures (separated by ';'): one; two;
--> Syntactic errors found: 0

mymatch(g, "one two three")
--> Captures (separated by ';'): one; two; three;
--> Syntactic errors found: 2
--> Error at line 1 (col 4): expecting ','
--> Error at line 1 (col 8): expecting ','

mymatch(g, "1,\n two, \n3,")
--> Captures (separated by ';'): NONE;
--> Syntactic errors found: 1
--> Error at line 1 (col 2): expecting a list of identifiers

mymatch(g, "one\n two123, \nthree,")
--> Captures (separated by ';'): one; two; three; NONE;
--> Syntactic errors found: 3
--> Error at line 2 (col 1): expecting ','
--> Error at line 2 (col 5): expecting ','
--> Error at line 3 (col 6): expecting an identifier

```

##### *relabel* syntax

Below we write a grammar for a simple programming language
using the syntax supported by *relabel*, where `%{}` is the throw
operator, and the syntax `p^l` is syntatic sugar for
`p / %{l}` (given that *l* is a valid identifier name):

```lua
local re = require 'relabel'

local terror = {
  cmdSeq        =  "Missing ';' in CmdSeq",
  ifExp         =  "Error in expresion of 'if'",
  ifThen        =  "Error matching 'then' keyword",
  ifThenCmdSeq  =  "Error matching CmdSeq of 'then' branch",
  ifElseCmdSeq  =  "Error matching CmdSeq of 'else' branch",
  ifEnd         =  "Error matching 'end' keyword of 'if'",
  repeatCmdSeq  =  "Error matching CmdSeq of 'repeat'",
  repeatUntil   =  "Error matching 'until' keyword",
  repeatExp     =  "Error matching expression of 'until'",
  assignOp      =  "Error matching ':='",
  assignExp     =  "Error matching expression of assignment",
  readName      =  "Error matching 'NAME' after 'read'",
  writeExp      =  "Error matching expression after 'write'",
  simpleExp     =  "Error matching 'SimpleExp'",
  term          =  "Error matching 'Term'",
  factor        =  "Error matching 'Factor'",
  openParExp    =  "Error matching expression after '('",
  closePar      =  "Error matching ')'",
  eof           =  "Error, expecting EOF",
  undefined     =  "Undefined Error"
}

g = re.compile([[
  Tiny       <- CmdSeq (!. / %{eof})
  CmdSeq     <- (Cmd SEMICOLON^cmdSeq) (Cmd SEMICOLON^cmdSeq)*
  Cmd        <- IfCmd / RepeatCmd / ReadCmd / WriteCmd  / AssignCmd
  IfCmd      <- IF  Exp^ifExp  THEN^ifThen  CmdSeq^ifThenCmdSeq  (ELSE CmdSeq^ifElseCmdSeq / '')  END^ifEnd
  RepeatCmd  <- REPEAT  CmdSeq^repeatCmdSeq  UNTIL^repeatUntil  Exp^repeatExp
  AssignCmd  <- NAME  ASSIGNMENT^assignOp  Exp^assignExp
  ReadCmd    <- READ  NAME^readName
  WriteCmd   <- WRITE  Exp^writeExp
  Exp        <- SimpleExp  ((LESS / EQUAL)  SimpleExp^simpleExp / '')
  SimpleExp  <- Term  ((ADD / SUB)  Term^term)*
  Term       <- Factor ((MUL / DIV)  Factor^factor)*
  Factor     <- OPENPAR  Exp^openParExp  CLOSEPAR^closePar  / NUMBER  / NAME
  ADD        <- Sp '+'
  ASSIGNMENT <- Sp ':='
  CLOSEPAR   <- Sp ')'
  DIV        <- Sp '/'
  IF         <- Sp 'if'
  ELSE       <- Sp 'else'
  END        <- Sp 'end'
  EQUAL      <- Sp '='
  LESS       <- Sp '<'
  MUL        <- Sp '*'
  NAME       <- !RESERVED Sp [a-z]+
  NUMBER     <- Sp [0-9]+
  OPENPAR    <- Sp '('
  READ       <- Sp 'read'
  REPEAT     <- Sp 'repeat'
  SEMICOLON  <- Sp ';'
  SUB        <- Sp '-'
  THEN       <- Sp 'then'
  UNTIL      <- Sp 'until'
  WRITE      <- Sp 'write'
  RESERVED   <- (IF / ELSE / END / READ / REPEAT / THEN / UNTIL / WRITE) ![a-z]+
  Sp         <- (%s / %nl)*
]], terror)

local function mymatch(g, s)
  local r, e, pos = g:match(s)
  if not r then
    local line, col = re.calcline(s, pos)
    local msg = "Error at line " .. line .. " (col " .. col .. "): "
    return r, msg .. terror[e]
  end
  return r
end

local s = [[
n := 5;
f := 1;
repeat
  f := f + n;
  n := n - 1
until (n < 1);
write f;]]
print(mymatch(g, s))            --> nil  Error at line 6 (col 1): Missing ';' in CmdSeq

print(mymatch(g, "a : 2"))      --> nil  Error at line 1 (col 2): Error matching ':='
print(mymatch(g, "a := 2; 6"))  --> nil  Error at line 1 (col 8): Error, expecting EOF
```

### Caveats

Does not use the number **1** to specify a recovery rule,
since that the entry with index 1 in the grammar table defines
its initial rule.

In case your grammar has many regular and recovery rules,
you may get an error message such as grammar: has too many rules.
In this case, we need to change *MAXRULES* in `lpltypes.h`.

repository: sqmedeiros/lpeglabel

gha_language: C

gha_license_id: MIT

repo_name: sqmedeiros/lpeglabel


text:
![unittests](https://github.com/adegenna/HardSphereDynamics/actions/workflows/cmake.yml/badge.svg)

Solver to compute 2D hard sphere collision dynamics. Parallelized with OpenMP.

Copyright 2021 by Anthony M. DeGennaro (ISC License).

**Installation**

```sh
cd [/PATH/TO/HardSphereDynamics]
mkdir build
cd build
cmake ../
make
```

**Unit Tests**

```sh
cd [/PATH/TO/HardSphereDynamics]/build
./testall
```

**Example Driver**

```sh
cd [/PATH/TO/HardSphereDynamics]/examples
./run_example.sh
./plot_example.sh
```
![](https://github.com/adegenna/HardSphereDynamics/blob/master/figs/billiards.png)
*Snapshots in time of a billiard ball rack break*

license: ISC

repository: adegenna/HardSphereDynamics

gha_license_id: ISC

repo_name: adegenna/HardSphereDynamics


text:
Jellybeans-nvim: a lua port of jellybeans for neovim
===

![Screenshot](scrot.png)

Instructions
---

**Note**: This colorscheme requires at least Neovim 0.5 (it works fine on current master).

Add to your `init.lua` (this is using
[paq-nvim](https://github.com/savq/paq-nvim), adjust to your package manager):

```
local paq = require('paq-nvim').paq

paq { 'rktjmp/lush.nvim'}
paq { 'metalelf0/jellybeans-nvim' }

cmd('colorscheme jellybeans-nvim')
```

Notes
---

* I directly mapped original colors as strictly as possible; right now, only
  colors already defined in the original colorscheme have been mapped, to get a
  colorscheme as close as possible to the original one. I'll consider adding
  more highlight groups for plugins etc. if requested;

* Treesitter support is basic: it uses the default links defined in
  `nvim-treesitter`, and I didn't customize the treesitter groups. Please
  submit a PR or contact me on reddit if you think anything should be changed /
  improved;

* this was built with [lush.nvim](http://git.io/lush.nvim); for more
  information on Lush and a helper script to setup your repo clone.

repository: thiagopnts/jellybeans-nvim

repo_name: thiagopnts/jellybeans-nvim


text:
---
title: "SecKill Development Journey (I)"
lang: en
ref: seckill-development-journey-part-I
permalink: /docs/seckill-development-journey-part-I/
excerpt: "an introduction of how to build seckill demo project step by step"
last_modified_at: 2017-09-13T09:00:00+08:00
author: Yangyong Zheng
tags: [seckill]
redirect_from:
  - /theme-setup/
---

## Overview
  SecKill is a demo project that our ServiceComb team try to build a micro-service architecture from the perspective of domain-driven design ([DDD](https://en.wikipedia.org/wiki/Domain-driven_design)).

## Background

  SecKill idea comments from the Internet popular promotions, using coupons to obtain some discounts; because the number of coupons is limited, so we must provide a platform for everyone who wants to get it fair competition.

  As a demo project, so we don't want make the SecKill too complicated; we just use the most primitive strategy - first come first get. It is quite simple if you want to implements other strategies by yourself.

## First Design
   Now let's analyze requirement in customer bounded context:
1. Customer try to grab a coupon, SecKill return the coupon according to the coupon in inventory;
2. Customer cannot get the coupon anymore if he already send the request to the system.
3. Customer can query how many coupons he owned.

  Let's go! what? you say this description is to simple? because they are no registration system, and promotion is how come, how to start? which coupon the customer request for? Is the discount all same? etc.

  Actually, we are accustomed to thinking a lot at the beginning, so some larger system, the design stage may be a few months, but this time you can't consider too much, because we believe:

1. Over-design can't guarantee fully meeting the demand after construction completed, the only constant is the demand is always changing, so don't over-design;
2. The principle of the microservice architecture is that service provides single function, so each components(microservices) only need to provide small service, without having to design all the services in system one-time;
3. Should be solve the problem of the core domain as soon as possible.

  Then, we can follow the many principles of domain-driven design ([DDD](https://en.wikipedia.org/wiki/Domain-driven_design)). The core task of the design process is identify entities, value objects, services and other elements, and further search aggregation; because our requirements are very simple, so it's easy find them out:
*  Entities:Promotion
*  Value-Object:Coupon,PromotionEvent
*  Domain Service:SecKill

  Now we can find the aggregation between them, separated two microservice, now we direct use MySQL database for shared data (although microservice recommend use independent storage for each, we can revisit it in the next iteration)
1. Command : for SecKill domain service
2. Query : for query Coupon Value-Object

![Fig-1 Architecture](/assets/images/seckill-develop-course-part-I-arth-en.png){: .align-center}

## Core Domain Implement
  The source code at [Github](https://github.com/ServiceComb/seckill), Welcome to Watch and Star : )
### Implement of Command micro-service
  We create two class "SecKillCommandService" and "SecKillPersistentRunner" to accept seckill request and persist seckill result, considering persistence is slow, we use BlockQueue temporary cache requests. another class "SecKillCommandRestController" act as Endpoint; ServiceComb support a variety of programming models and communication model, also well support Spring Boot, and easy deploy to Huawei cloud.

![Fig-2 Command micro-service components](/assets/images/seckill-develop-course-part-I-command-en.png){: .align-center}

### Implement of Query microService
  Query logic is relatively simple, because PromotionEvent is Value-Object, we use PromotionEvent records to get Coupon Value-Object, you can read the source code and understand this process.

## Support Domain Implement
  As promotion is entity and seckill need it as preconditions, we need build and manage it in another domain, because it's not so urgent at current stage, we can use Spring injection autowire fix objects.

![Fig-3 Support domain object injection](/assets/images/seckill-develop-course-part-I-spring.png){: .align-center}

## Function Verification
  Finally, we use PostMan to check completed functions, Please ensure request body fill with JSON format, and set Content-Type to application/json:

### Request SecKill

![Fig-4 Request seckill](/assets/images/seckill-develop-course-part-I-request-seckill.png){: .align-center}

### Query owned coupons

![Fig-5 Query owned coupons](/assets/images/seckill-develop-course-part-I-query-coupons.png){: .align-center}

gha_language: CSS


text:
# 5. Integer Underflow in 6LoWPAN IPHC Header Uncompression
- Bug Description: Missing checks on network packet size in uncompress_IPHC_header leads to an integer underflow, resulting in corrupted net_buf bounds
- Bug Result: The size field of a net_buf_simple struct gets underflown, significantly enlarging the assumed size of a network buffer, leading to out-of-bounds accesses in IPv6 functionality.
- Bug Impact: Out-of-bounds accesses inside IPv6 parsing logic. I highly suspect this to be exploitable to Arbitrary Code Execution by an attacker sending IPv6 packets (such as maliciously fragmented ICMPv6 Ping requests).

## Bug Details
- Affected code: Header Uncompression logic in subsys/net/ip/6lo.c#net_6lo_uncompress->uncompress_IPHC_header.

High-Level reasoning for bug occurrence:
1. A recent fix of a bug which I reported earlier added a check to make sure enough buffer tail space was available and that the right amount of bytes were moved during in-place 6LoWPAN IPHC header compression(https://github.com/zephyrproject-rtos/zephyr/blob/d969aced6dd9ef7834d21ff8131aa561e71cc42d/subsys/net/ip/6lo.c#L1356)
2. Fuzzing has shown another bug in the handling of the opposite case where the header cannot be uncompressed into the existing buffer.
3. If not enough space is available in the current buffer, an additional buffer is allocated to hold the uncompressed contents
4. In this external-buffer uncompressed handling, the size of the compressed header is calculated based on the metadata of the initial parts of the payload
5. The expected header of the pre-computed size is then stripped from the original fragment using net_buf_pull.
6. It is not checked, however, if the buffer held enough bytes to contain this uncompressed header in the first place
7. As a result, net_buf_pull will try to remove more bytes from the buffer than are actually present, which makes the buffer's size go negative, which is a large number as it is later interpreted as an unsigned number.
8. This sets the buffer pointer of the net_buf_simple buffer out of bounds, such that out-of-bounds memory is treated as a way too large network buffer

https://github.com/zephyrproject-rtos/zephyr/blob/d969aced6dd9ef7834d21ff8131aa561e71cc42d/subsys/net/ip/6lo.c#L1366
Vulnerable code path:

0. Multiple code paths lead to this IP-layer uncompression logic, including Bluetooth, CAN bus, and IEEE 802.15.4 radio packets. I list a sample code path from ieee802154 packets, which were fuzzed using our research prototype.
1. ieee802154_recv->ieee802154_manage_recv_packet->ieee802154_reassemble->fragment_add_to_cache
	- Fragments are added via fragment_add_to_cache, and the full packet eventually reconstructed if all required fragments are present
	- Link: https://github.com/zephyrproject-rtos/zephyr/blob/d969aced6dd9ef7834d21ff8131aa561e71cc42d/subsys/net/l2/ieee802154/ieee802154_fragment.c#L517
2. fragment_add_to_cache->net_6lo_uncompress
	- After reconstructing the packet data, the compressed IPv6 header needs to be uncompressed
	- Link: https://github.com/zephyrproject-rtos/zephyr/blob/d969aced6dd9ef7834d21ff8131aa561e71cc42d/subsys/net/l2/ieee802154/ieee802154_fragment.c#L527
3. fragment_add_to_cache->net_6lo_uncompress->uncompress_IPHC_header->get_ihpc_inlined_size
	- To uncompress the header, first the length is determined from the header's 'iphc' tag
	- Link: https://github.com/zephyrproject-rtos/zephyr/blob/d969aced6dd9ef7834d21ff8131aa561e71cc42d/subsys/net/ip/6lo.c#L1330
4. fragment_add_to_cache->net_6lo_uncompress->uncompress_IPHC_header
	- Without checking the actual size of the incoming packet's buffer, the buffer is trimmed by the size which was computed from the 'iphc' tag
	- Link: https://github.com/zephyrproject-rtos/zephyr/blob/d969aced6dd9ef7834d21ff8131aa561e71cc42d/subsys/net/ip/6lo.c#L1366
5. uncompress_IPHC_header->net_buf_simple_pull
	- The network buffer API function net_buf_pull will then substract the size, underflowing it in the process
	- Link: https://github.com/zephyrproject-rtos/zephyr/blob/d969aced6dd9ef7834d21ff8131aa561e71cc42d/subsys/net/buf.c#L1126

## Proposed Fix
- After calculating the size of the expected uncompressed header based on the iphc metadata field, check that enough space is actually present within the buffer.

- Note that the single pkt->buffer fragment may not represent all data within the packet, which may consist of multiple fragments
	- Legitimate parties will probably not send a too small fragment as part of the fragments, so the first fragment not holding the full compressed header could be used as an indication to just drop the packet
	- Otherwise, the logic would have to support stripping the uncompressed header from across multiple network packet fragments/buffers

- If the uncompressed header is expected to be present in the first fragment, the following check could be implemented:
```
diff --git a/subsys/net/ip/6lo.c b/subsys/net/ip/6lo.c
index 736cf05839..f870abf4fc 100644
--- a/subsys/net/ip/6lo.c
+++ b/subsys/net/ip/6lo.c
@@ -1348,6 +1348,12 @@ static bool uncompress_IPHC_header(struct net_pkt *pkt)
                        nhc_inline_size;
        }

+       /* Proposed fix: Make sure the buffer holds the full compressed header */
+       if (compressed_hdr_size > pkt->buffer->len) {
+               NET_ERR("Too small packet to hold compressed IPHC header");
+               return false;
+       }
+
        if (net_buf_tailroom(pkt->buffer) >= diff) {
                NET_DBG("Enough tailroom. Uncompress inplace");
                frag = pkt->buffer;

```

repository: fuzzware-fuzzer/fuzzware-experiments

gha_language: Shell

repo_name: fuzzware-fuzzer/fuzzware-experiments


text:
[![Gitter](https://img.shields.io/badge/chat-gitter-purple.svg)](https://gitter.im/taymyr/taymyr)
[![Gitter_RU](https://img.shields.io/badge/chat-russian%20channel-purple.svg)](https://gitter.im/taymyr/taymyr_ru)
[![Build Status](https://travis-ci.org/taymyr/lagom-metrics.svg?branch=master)](https://travis-ci.org/taymyr/lagom-metrics)
[![Javadocs](https://www.javadoc.io/badge/org.taymyr.lagom/lagom-metrics-java_2.12.svg)](https://www.javadoc.io/doc/org.taymyr.lagom/lagom-metrics-java_2.12)
[![codecov](https://codecov.io/gh/taymyr/lagom-metrics/branch/master/graph/badge.svg)](https://codecov.io/gh/taymyr/lagom-metrics)
[![Maven](https://img.shields.io/maven-central/v/org.taymyr.lagom/lagom-metrics-java_2.12.svg)](https://search.maven.org/search?q=a:lagom-metrics-java_2.12%20AND%20g:org.taymyr.lagom)

# [Dropwizard Metrics](https://metrics.dropwizard.io) for [Lagom](https://www.lagomframework.com)/[Play](https://playframework.com)

TODO

## License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this project except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

repository: taymyr/lagom-metrics

gha_language: Kotlin

repo_name: taymyr/lagom-metrics


text:
# FileX.GetReadFileStream method

```csharp
public static FileStream GetReadFileStream(string file, bool async = true)
```

## See Also

* class [FileX](../FileX.md)
* namespace [Butterfly.Core.Util](../../Butterfly.Core.md)

<!-- DO NOT EDIT: generated by xmldocmd for Butterfly.Core.dll -->


text:
go-web-proxy
============

A simple web based proxy written in Go. Can be easily hosted on [OpenShift](https://openshift.redhat.com/app/login)

currently provides two options,

1] this is useful for a single file. It only pipes the response.

    ` <hostname>/p/?target=http://foo.com/bar.png`

2] this is useful to view a web page that contains other static images, css, js files

    `<hostname>/t/?target=http://foo.com/bar/`

how to host on OpenShift
------------------------

- fork this repo
- register an account on openshift.redhat.com, follow standard account activation procedure
- go to **Settings** tab and add public key for ssh to work
- go back to **Applications** tab, click **Add Application**
- scroll down and select Go language under Other types
- fill in the form, enter name/domain pair
- enter git repo address for forked repo. enter **master** as branch name
- keep default for rest but know that if you select **No scaling** then you won't be able to scale later
- Click **Create Application**
- Done

Add a bookmarklet to quickly use this proxy
-------------------------------------------

- Add a new bookmark in your browser
- name it anything, say **proxy**
- use this as its URL

    `javascript:location.href='http://<your-hostname-here>/t/?target='+location.href`

- now when you need to use this proxy, simply click the bookmark and it should work

License
-------

MIT license


text:
# Commands

Since commands are before events and have not happend yet they should be in present-tense.
Commands are send via RPC not as message.

`startFlow`

`stopFlow`

`continueFlow`

`cancelFlow`


text:
# Luci: The Discord Bot

![Luci](/bot/avatars/avatar2.png)

A General-Purpose Fun bot with lots and lots of features.

# Commands

## Moderation

| Commands                  | What it does?                                             |
| ---------                 | ---------------------------------                         |
| mute                      | Mute someone for a time interval                          |
| unmute                    | Unmute a muted person                                     |
| kick                      | Kick a member out!                                        |
| ban                       | Ban them! Good riddance                                   |
| unban                     | Unban a member                                            |
| bans                      | See list of members who are banned                        |
| baninfo                   | See info about a particular banned member                 |
| addrole                   | Add a role to a member                                    |
| removerole                | Remover a role from a member                              |
| purge             | Purge a member's messages (You can purge your own messages too)   |
| clean                     | Clean your own messages                                   |
| hackban                   | Ban someone who is not in the server                      |
| lock | Lockdown a channel or the server (members will not be able to send any message) |
| unlock                    | Unlock a locked server or channel                         |

## Games

| Commands                  | What it does?                                             |
| ---------                 | ---------------------------------                         |
| aki                       | Play the famous Akinator the game                         |
| tictactoe [alias = ttt]   | Play the classic Tic Tac Toe game against Luci            |
| quiz                      | Play a trivia quiz from over 20 categories                |

## Conversation games

| Commands                  | What it does?                                             |
| ---------                 | ---------------------------------                         |
| truth                     | Ask a truth question. No lie!                             |
| dare                      | Dare someone                                              |
| neverhaveiever [alias = nhie] | Never have I ever...                                  |
| wouldyourather [alias = wyr]  | Would you rather...                                   |

## Fun Commands

| Commands                  | What it does?                                             |
| ---------                 | ---------------------------------                         |
| meme                      | Meme goes brrrrr!                                         |
| memespam                  | So that you don't have to beg for meme 50 times           |
| avatar    [alias = av]    | See the avatar of a user (must be from the same guild)    |
| bigmoji   [alias = e ]    | Enlarge an emoji                                          |
| Comics                    | Get a ohno, Saturday Morning Breakfast Cereal (smbc), The Perry Bible Fellowship (pbf), Cyanide and Happiness (cah), XKCD (xkcd), Mr. Lovenstein (mrls), Chainsawsuit (chainsaw), Sarah's Scribbles (sarah), Dilbert (dilbert), Calvin and Hobbes (calvin), Garfield (garfield), Odd is out (oddones) Commic |
| poll                      | Make a poll                                               |
| nitro [alias = n] | Don't have nitro? Don't worry, send an animated emote through bot |
| insult                    | Evil insult 😈                                            |
| shout                     | Emojify words                                             |
| dm                        | DM someone through the bot (but why would you do that anyway 🤔) |
| alphanato [alias = nda]   | Get military phonetics                                    |
| photo                     | Get a photo from unsplash                                 |
| wallpaper                 | Get today's Bing Wallpaper                                |

## Animals Awwwww ❤

| Commands                  | What it does?                                             |
| ---------                 | ---------------------------------                         |
| dog                       | Get a cute doggo pic                                      |
| dogfact                   | Get a random fact about dogs                              |
| cat                       | Get a cute catto pic                                      |
| catfact                   | Get a random fact about cats                              |
| panda                     | Get a cute panda pic                                      |
| birb                      | Get a cute bird pic                                       |
| redpanda                  | Get a cute redpanda pic                                   |
| koala                     | Get a cute koala pic                                      |

## Nerd Commands

| Commands                  | What it does?                                             |
| ---------                 | ---------------------------------                         |
| math [alias = calc]       | Handy-dandy calculator (supports very complex stuff too)  |
| integrate                 | Integrate an expression (not fully functional yet 😬)     |
| derive                    | Differentiate an expression                               |
| factorise [alias = solve] | Find roots of a polynomial                                |
| tangent                   | Find tangent to a curve at a point                        |
| definite integrate        | Do definite integration on a curve                        |

The command that I am most proud of, however are these:

## Schedule Memes

You can schedule to send memes (from a particular subreddit; default is r\dankmemes) to a channel hourly/twice a day/daily/whatever. You can schedule the time interval in any way.

## Schedule Wallpaper

Send daily bing wallpapers to a channel.

# TODO

There are lots and lots of things that I want to add to the bot. You can open an Issue and suggest anything that you want me to add and I will try my best.

repository: luciferchase/luci

gha_language: Python

repo_name: luciferchase/luci