text:
# terraform-aws-autocluster-rabbitmq

This repo contains a [Terraform](https://terraform.io/) module to provision a rabbitmq cluster using docker, and autoscaling groups of AWS.

Enabled rabbitmq plugins
---------------

- Autocluster
- Consistent Hash Exchange
- Delayed Message Exchange
- Federation
- Federation Management
- Management
- Management Visualiser
- Message Timestamp
- MQTT
- Recent History Exchange
- Sharding
- Shovel
- Shovel Management
- Stomp
- Top
- WebStomp

What is included in this module
------------------------------
* Creation and provisioning of the rabbitmq cluster's nodes using docker image gavinmroy/alpine-rabbitmq-autocluster which supports the autocluster plugin of rabbitmq using aws autoscale groups
* Creation of autoscale group/ launch configurations (and the iam roles/policies required)
* ELB to access rabbitmq 5672 port and the admin console (thourgh port 80 and 15672)
* Posibility to setup the ELB to be internet-faced but it will be created private by default.
* Security groups for the ELB/instances -> by default, traffic is allowed from within inside the VPC CIDR blocks

Requirements
--------------
* VPC is required (see [terraform-multitier-vpc](https://github.com/grey-systems/terraform-multitier-vpc) to create an enabled HA vpc in AWS

Usage
--------
Module usage:

      provider "aws" {
        access_key = "${var.access_key}"
        secret_key = "${var.secret_key}"
        region     = "${var.region}"
      }

    # minimal setup, check Inputs to see the different customisation options
     module "rabbitmq" {
         source                                = "github.com/teamleadercrm/terraform-aws-autocluster-rabbitmq.git?ref=master"
         public_subnet_ids                     = "${var.public_subnet_ids}"
         private_subnet_ids                    = "${var.private_subnet_ids}"
         availability_zones                    = "${var.availability_zones}"
         aws_keypair_name                      = "${var.aws_keypair_name}"
         environment                           = "${var.environment}"
  }

Inputs
---------
| Name | Description | Type | Default | Required |
|------|-------------|:----:|:-----:|:-----:|
| availability_zones | The chosen availability zones *must* match the AZs the VPC subnets are tied to. | list | - | yes |
| aws_keypair_name | AWS Key pair name | string | - | yes |
| environment | Environment name, used as prefix to name resources | string | - | yes |
| rabbitmq_admin_elb_allow_cidr_blocks | List of additional cidrs block that will be granted to access the rabbitmq Admin console VPC's cidr_block is always granted. | list | `<list>` | no |
| rabbitmq_desired_nodes | Desired number of nodes for autoscaling | string | `1` | no |
| rabbitmq_dns_name | Dns record that will be created if route53_zone_id is provided The Dns record will be created as {rabbitmq_dns_name}.{route_zone_name} and it will be a CNAME to the rabbitmq ELB. | string | `rabbitmq` | no |
| rabbitmq_elb_allow_cidr_blocks | List of additional cidrs block that will be granted to access the rabbitmq broker ELB VPC's cidr_block is always granted. | list | `<list>` | no |
| rabbitmq_elb_private | Determines if the ELB to be created will be private | string | `true` | no |
| rabbitmq_instance_allowed_cidr_blocks | List of additional cidrs block that will be granted to access the rabbitmq isntances (ssh, rabbit ports) VPC's cidr_block is always granted. | list | `<list>` | no |
| rabbitmq_instance_type | AWS EC2 instance type for the instances | string | `t2.small` | no |
| rabbitmq_max_nodes | Max number of nodes for autoscaling | string | `5` | no |
| rabbitmq_min_nodes | Min number of nodes for autoscaling | string | `1` | no |
| route53_zone_id | Route53 zone id to create a dns record for rabbitmq. The record will be created if and only if this value is provided (non-empty) | string | `` | no |
| subnet_ids | List of subnets ids to spare the different rabbitmq nodes | list | - | yes |

Outputs
---------

| Name | Description |
|------|-------------|
| rabbitmq_autoscaling_group_id | Id of the autoscaling group |
| rabbitmq_elb_id | Id of the rabbitmq ELB |

Contributing
------------
Everybody is welcome to contribute. Please, see [`CONTRIBUTING`][contrib] for further information.

[contrib]: CONTRIBUTING.md

Bug Reports
-----------

Bug reports can be sent directly to authors and/or using github's issues.

-------

Copyright (c) 2017 Grey Systems ([www.greysystems.eu](http://www.greysystems.eu))

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.


text:
# Meeshkan Webapp

You can sign up for Meeshkan at [https://app.meeshkan.com/](https://app.meeshkan.com/).

Repositories in, bug fixes out. [Meeshkan](https://meeshkan.com/) is an automated testing tool designed to find bugs in your projects. In its current state, it's built to handle APIs and third-party services. Once installed, Meeshkan runs weekly and submits issues to your repository when it finds bugs.

We're actively building this webapp to facilitate the core service. Through the Meeshkan webapp, you can see what tests were run and resolve any issues that come up while executing your tests.

📖 Looking for an overview? [Read our announcement blog post](https://meeshkan.com/blog/open-source-webapp/).

![Example of the Meeshkan dashboard in dark mode.](https://user-images.githubusercontent.com/26869552/83249968-ab134a80-a1a7-11ea-9168-bf4864e6c680.png)

## What's in this document:

- [Documentation](#documentation)
- [Routing style](#routing-style)
- [Important dependencies](#important-dependencies)
- [Known issues](#known-issues)
- [Roadmap](#roadmap)
- [Contributing](#contributing)

## Documentation

If you're an existing Meeshkan user, everything from configuration instructions to frequently asked questions is available in the [Meeshkan documentation](https://meeshkan.com/docs/).

## Important dependencies

This repository makes heavy use of certain libraries. Without being familiar with the basics of how these libraries work, it will be difficult to understand the code base.

These libraries are, in no particular order:

- [`next.js`](https://github.com/zeit/next.js): Opinionated framework for writing server-rendered React apps.
- [`chakra-ui`](https://github.com/chakra-ui/chakra-ui): Accessible React component library.
- [`fp-ts`](https://github.com/gcanti/fp-ts): Typed functional programming for TypeScript.
- [`io-ts`](https://github.com/gcanti/io-ts): Run-time type system for IO decoding/encoding.
- [`monocle-ts`](https://github.com/gcanti/monocle-ts): Functional optics for TypeScript projects.

## Routing style

We optimize for user experience (clear URL structure) and only add an abstraction if absolutely necessary. The repercussion of this is we will have some reserved paths such as `settings`, `user`, and more. We percieve this to have minimal impact and aren't actively solving for these edge cases.

## Known issues

- We currently have no way to know if a user revokes an incoming webhook.
- GitHub often responds that a stored refresh token is invalid.

## Roadmap

We're still in the early stages and things are changing rapidly, but here are some larger goals we'd like to accomplish with this webapp. Please note that these are subject to change at any time.

- Allow you to control the frequency of your tests
- Give you access to explore the specification Meeshkan builds of your repository
- More team-related functionalities
- Burn-down chart of bugs

Have a suggestion? [File an issue](https://github.com/meeshkan/webapp/issues/new).

## Contributing

⚠ The purpose of this repository is to keep our work open and transparent. But, given the way our webapp is configured, **we're unable to take outside contributions directly**. [File an issue](https://github.com/meeshkan/webapp/issues/new) if you have a feature request or bug report. You can also [contact our team directly](https://meeshkan.com/contact/).

Please note that this project is governed by the [Meeshkan Community Code of Conduct](https://github.com/meeshkan/code-of-conduct). By participating, you agree to abide by its terms.

➡ If you're a member of the Meeshkan organization, you can find all of the local development instructions in the [contributing guide](./CONTRIBUTING.md).

repository: khadorkin/webapp

branch_name: refs/heads/main

extension: md

license_type: permissive

repo_name: khadorkin/webapp


text:
|   |   |
|---|---|
| **[Grid Protection Alliance](http://www.gridprotectionalliance.org "Grid Protection Alliance Home Page")** | **[Exercises](README.md)** |

***This document is an example procedure for illustrating workflow methods.***

---

# GitHub Fork Example

This is an *example* style exercise that illustrates GitHub forked project workflow methods. Although the exact exercise is not repeatable because it refers to specific accounts and repositories, the general concepts and methodology of this exercise are repeatable.

- [Platform](#platform)
    - [GitHub.com Account](#githubcom-account)
    - [Git Software](#git-software)
- [Forking a Project](#forking-a-project)
    - [Cloning Your Project Fork](#cloning-your-project-fork)
        - [Frequently Asked Questions (FAQ)](#cloning-frequently-asked-questions-faq)
        - [Configuring ssh](#configuring-ssh)
        - [Cloning Your Project from GitHub to Your Local Machine](#cloning-your-project-from-github-to-your-local-machine)
- [Synchronizing Your GitHub Fork with the Upstream Master Repository](#synchronizing-your-project-fork-with-the-upstream-master-repository)
    - [Configuring Your Cloned Project Remotes](#configure-your-cloned-project-remotes)
    - [Pulling Changes from Your GitHub Fork](#pulling-changes-from-your-github-fork)
    - [Pulling Changes from the Upstream Master](#pulling-changes-from-the-upstream-master)
    - [Pushing Changes to your GitHub Fork](#pushing-chagnes-to-your-gethub-fork)
- [Creating a Pull Request](#creating-a-pull-request)
- [Managing Merge Conflicts](#managing-merge-conflicts)

---

## Platform

This exercise uses a PC or virtual machine with Internet connection and Git command line software installed.  This exercise was tested in both Windows and Linux operating systems.  The GitHub website is designed to work with a modern web browser with a graphic user interface (GUI).

### GitHub.com Account

You need a [GitHub.com](https://github.com) registered account with a verifiable email address to perform the tasks in this exercise.  A free GitHub account can be used for this exercise.

This exercise will use a GitHub account named *W3PMU*.  To perform this exercise, replace *W3PMU* with your own GitHub account name.

### Git Software

- *Windows* - Download and install Git from [https://git-scm.com](https://git-scm.com/). This exercised was tested using a *Portable* version.
- *Linux* - Install Git using your distro's package manager or build Git from source code. For example: [Git instructions in Exercise_openPDC_in_Debian_Jessie](Exercise_openPDC_in_Debian_Jessie.md#git)

---

## Forking a Project

The folllowing demonstrates how to create a *Fork* of this DeveloperExercises project.

1. Run a web browser and Sign in to [https://github.com](https://github.com)
2. Navigate to the [DeveloperExercises](https://github.com/GridProtectionAlliance/DeveloperExercises) repository
3. Click the *Fork* button
    - screenshot: [GitHub project page showing *Fork* button in upper right corner](Exercise_GitHub_Fork.files/Fork_01.png)
    - screenshot: [Project *Forking* progress page](Exercise_GitHub_Fork.files/Fork_02.png)
4. After the *Fork* process is completed, your browser will display a new *forked* project page in your GitHub account.
    - screenshot: [Your account's new *Forked* project page](Exercise_GitHub_Fork.files/Fork_03.png)
5. Optional Setting: To avoid possible confusion, click the *Settings* button and give your project *Fork's* repository a new name.  For example, after *forking* **DeveloperExercises**, rename its repository to **DeveloperExercises_fork**
6. Optional Setting: Set the *Restrict editing to collaborators only* option to checked
    - screenshot: [Your *Forked* project's *Settings* showing *Repository name* changed and *Restricted editing* checked](Exercise_GitHub_Fork.files/Fork_04.png)

### Cloning Your Project Fork

For simplicity in the scope of this exercise: *Forks* are on GitHub and *Clones* are on your local machine.

*Clone* a project's repository to work with the project's files on your local machine.

#### Frequently Asked Questions (FAQ)

- What is a *clone*?

> A *clone*, in the scope of this exercise, is a copy of a *git* repository. *Cloning* a repository is the procedure for making a copy of the repository. Unlike a downloaded *zip* or *tar* file containing a copy of the repository's files, a clone includes git's content management metadata that enables you to pull updates and merge changes in your repository.

- What is a *fork*?

> A *fork*, in the scope of this exercise, is a *GitHub* repository that was *cloned* from another existing *GitHub* repository. *Forking* a repository is the procedure for cloning a GitHub repository from the main *upstream* source to your own GitHub repositories collection.

- Should I *clone* my *fork* or should I *clone* the mainstream or *upstream* repository?

> When you *clone* your *fork*, it is easier to push your local changes to your *fork* on GitHub. Using the GitHub *fork* and *pull request* workflow allows you and others to review your changes before committing them to the main *upstream* project.

- Can I *fork* my *clone*?

> No, not in the scope of this exercise.  GitHub cannot access your *clones* on your local machine; so GitHub cannot *fork* your *clones*.

#### Configuring **ssh**

The following procedure configures **ssh** to enable *git* to efficiently communicate between GitHub and your local machine. **ssh** client comes standard in most Linux distributions.  If you have a Windows machine and installed the [Git Software](#git-software) client as described in the beginning of this exercise, the *MINGW bash* terminal provides an **ssh** client.

1. On your local machine, open a *bash* compatible terminal. In Windows, run the Git client: `git-bash.exe`.
2. Review the directory listinf for the folder `~/.ssh` using the command: `ls -al ~/.ssh`
    - screenshot: [typical .ssh directory listing](Exercise_GitHub_Fork.files/ssh_01.png)
3. If your directory listing for the `~/.ssh` folder contains the files: `id_rsa` and `id_rsa.pub` then you can probably skip the **ssh-keygen** step.  If the directory does not have those files the run the **ssh-keygen** utility.
    - When running **ssh-keygen** accept all of the defaults by pressing the *Enter* key. A passphrase is not needed for this exercise.
        - screenshot: [ssh-keygen](Exercise_GitHub_Fork.files/ssh_02.png)
4. Copy the *ssh-rsa* contents from the `~/.ssh/id_rsa.pub` file to your machine *clipboard*
    - screenshot: [ssh-rsa key in id_rsa.pub](Exercise_GitHub_Fork.files/ssh_03.png)
5. Open a web browser and sign in to your [GitHub.com](https://github.com) account
6. Drop down your Profile menu and select *Settings*
    - screenshot: [Profile menu Settings](Exercise_GitHub_Fork.files/GitHub_Settings_01.png)
7. Select the *SSH and GPG keys* item from left side Personal settings menu
8. Click the *New SSH key* button in the upper right
9. Paste the *ssh-rsa* key from the id_rsa.pub file into the *Key* text box. If paste does not work, repeat step 4 above and try again.
10. Enter a *Title* for the key. This is typically set to the username@machinename found at the end of the key.
    - screenshot: [New SSH Key](Exercise_GitHub_Fork.files/GitHub_Settings_02.png)
10. Click the *Add SSH key*
    - screenshot: [Added SSH Key](Exercise_GitHub_Fork.files/GitHub_Settings_03.png)

#### Cloning Your Project from GitHub to Your Local Machine

1. On your local machine, open a *bash* compatible terminal. In Windows, run the Git client: `git-bash.exe`.
2. Navigate to or Create a convenient work folder
3. Clone the project with the command: `git clone git@github.com:W3PMU/DeveloperExercises_fork.git`
    - screenshot: [git clone](Exercise_GitHub_Fork.files/git_01.png)

You now have a local *clone* of your GitHub *forked* repository.

---

## Synchronizing Your GitHub Fork with the Upstream Master Repository

When the main upstream repository is changed by other contributors, your GitHub *fork* and your local *clones* may need to be synchronized with the main upstream repository.

The general procedure for synchornizing your GitHub *fork* and local *clones* can be a variation of the following steps:

1. Add and Commit changes to your local *clone*
2. Pull any changes from your GitHub *fork* to merge with your local *clone*
2. Pull changes from the main upstream repository to merge with your local *clone*
3. Push your local *clone* changes to your GitHub *fork*

### Configuring Your Cloned Project Remotes

Your local *clone* needs to have 2 **git remote** connections defined. The default **remote** named *origin* was defined when the repository was cloned from the *fork*. To pull from the main upstream project, add another **remote**.

1. On your local machine, open a *bash* compatible terminal. In Windows, run the Git client: `git-bash.exe`
2. Navigate to your *clone* folder
3. Run `git remote -v` to review existing git remotes
4. Add a new remote for the main upstream repository by running the following command:
    - `git remote add gpa https://githup.com/GridProtectionAlliance/DeveloperExercises.git`
    - screenshot: [Review remotes](Exercise_GitHub_Fork.files/git_02.png)

### Pulling Changes from Your GitHub Fork to Your Local Clone

1. Run `git pull origin master` to pull changes from your fork to your clone
    - screenshot: [Update your local clone from your GitHub fork](Exercise_GitHub_Fork.files/git_03.png)

### Pulling Changes from the Upstream Master

1. Run `git pull gpa master` to update your clone with changes from the main upstream repository
    - screenshot: [Pull changes from the main project](Exercise_GitHub_Fork.files/git_04.png)

### Pushing Changes to your GitHub Fork

1. Run `git push origin master` to push your local *clone's* changes up to up to your GitHub *fork*
    - screenshot: [Push to your GitHub fork](Exercise_GitHub_Fork.files/git_05.png)

---

## Creating a Pull Request

The folllowing demonstrates how to create a *Pull Request* to submit your contributions from your *Fork* to the main upstream project.

1. Run a web browser and Sign in to [https://github.com](https://github.com)
2. Navigate to the [DeveloperExercises](https://github.com/GridProtectionAlliance/DeveloperExercises) repository
3. Click the *Pull requests* tab button
    - screenshot: [GitHub project page showing *Pull requests* tab button](Exercise_GitHub_Fork.files/PullRequest_01.png)
4. Click the *New pull request* button in the upper right
5. Click the *compare across forks* link in the "Compare changes across branches..." text
6. Click and drop down the *head fork:* button and select your *fork* from the drop down list
    - screenshot: [Select your fork from the head forks drop down](Exercise_GitHub_Fork.files/PullRequest_02.png)
7. Check for the green *Able to merge* check message
    - screenshot: [Able to merge check](Exercise_GitHub_Fork.files/PullRequest_03.png)
8. Click the *Create pull request* button
    - screenshot: [Open a pull request](Exercise_GitHub_Fork.files/PullRequest_04.png)
9. Click the *Create pull request* button
    - screenshot: [Completed pull request](Exercise_GitHub_Fork.files/PullRequest_05.png)

---

Jan 8, 2017 - created by [aj](https://github.com/ajstadlin)

---

Copyright 2017 [Grid Protection Alliance](http://www.gridprotectionalliance.org)

license: MIT

branch_name: refs/heads/master


text:
# Snakemake workflow: Snakemake Pipeline for Illumina Raw Data

[![DOI](https://zenodo.org/badge/531835641.svg)](https://zenodo.org/badge/latestdoi/531835641)
[![Snakemake](https://img.shields.io/badge/snakemake-≥6.3.0-brightgreen.svg)](https://snakemake.github.io)
[![GitHub actions status](https://github.com/currocam/IlluminaSnakemake/workflows/Tests/badge.svg?branch=main)](https://github.com/currocam/IlluminaSnakemake/actions?query=branch%3Amain+workflow%3ATests)

A Snakemake workflow for Illumina raw reads to genome assemblies, annotate genomes and compare them.

## Setting up the pipeline

### Cloning repository

```bash
git clone https://github.com/currocam/IlluminaSnakemake
cd IlluminaSnakemake
```

### Installation

### Custom installation

There are two ways to install the pipeline. One is to create a conda enviroment with all the necessary programs.

### Containerized installation

However, you may find incompatibilities in the environment creation. In that case, you can make use of Conda and Singularity to run each of the programs in a separate environment. You will need a base installation of [Snakemake](https://snakemake.readthedocs.io/en/stable/#), [Singularity](https://docs.sylabs.io/guides/3.5/user-guide/introduction.html) and [Mamba](https://mamba.readthedocs.io/en/latest/installation.html).

Assuming that you have a preexisting Mamba installation:

```bash
mamba create -c conda-forge -c bioconda -n snakemake snakemake singularity
conda activate snakemake
```

> :warning: ****You will need to add `--use-conda` and `--use-singularity` to every command**.

### Fixing BUSCO requirement for parallelization

```bash
export NUMEXPR_MAX_THREADS=100
```

You can follow this [tutorial](https://guillaume-martin.github.io/saving-environment-variables-in-conda.html) to avoid running this step every single time.

## Quick start

Just the most common commands you may be using all the time. Given that raw data are in the `reads` directory inside `IlluminaSnakemake`, run the following command to rename the files:

```bash
snakemake -c8 rename
```

or, if using conda and singularity:

```bash
snakemake -c8 rename --use-conda --use-singularity
```

Then, run the following to display what would be done (taking only the file modification date into account for deciding if a substep should be re-run):

```bash
snakemake -n -rerun-triggers mtime
```

or, if using conda and singularity:

```bash
snakemake -n -rerun-triggers mtime --use-conda --use-singularity
```

If it looks okay, just run it by specifying the maximum number of threads the pipeline can use (although single steps are forced to a maximum of 0.75 * `MAX_CORES`)

```bash
snakemake -c75 -rerun-triggers mtime
```

or, if using conda and singularity:

```bash
snakemake -c75 -rerun-triggers mtime --use-conda --use-singularity
```

This will probably take a while. You may want to check the log files for each subprocess while are being written inside the `logs` directory.

## Usage

First of all, let's inspect the directory structure **before** you run Snakemake, just after you clone this repository:

```bash
$ tree -L 2
.
├── LICENSE
├── README.md # You're reading this file rigth now
├── config
│   ├── README.md # How to configurate the pipeline
│   └── config.yaml # Where you're going to configurate the pipeline
├── enviroment.yaml # List of dependencies
└── workflow # Where the actual code is written
    ├── Snakefile
    └── envs
```

The script expects a directory named `reads` where the raw Illumina `fastq.gz` files should be moved into. Please check [`config/README.md`](config/README.md) for changing this default behavior. Files should be named as follows:

### Renaming files

The first step is to rename the different files to just display the sample and the read. You can do this manually or run the following command. In case of renaming files manually, you should move them into the `results/renamed_raw_reads` (or edit the `config.yaml` file).

```bash
snakemake -c1 rename
```

or, if using conda and singularity:

```bash
snakemake -c1 rename --use-conda --use-singularity
```

### Running snakemake

Snakemake has a large `-help` documentation, which can be accessed as follows:

```bash
snakemake --help
```

Here, we'll see only the ones I have found more interesting. First of all, you may run a `dry-run` (to display what would be done)

```bash
snakemake -n --quiet
```

Or a more detailed one, including files and the shell commands that will be executed (nice for debugging):

```bash
snakemake -p -n
```

To run the pipeline, you must indicate the number of maximum cores you want to dedicate to the pipeline:

```bash
snakemake -c 100
```

Any Snakemake pipeline can be visualized using `--dag` as follows

```bash
snakemake --dag | dot -Tsvg > docs/dag.svg
```

![DAG](docs/dag.svg)

Let's inspect the directory structure **after** you run Snakemake. You may be aware that new directories have been created (in addition to `reads`, where we moved the raw data files previously).

```bash
tree -L 2
.
├── LICENSE
├── README.md
├── config
├── enviroment.yaml
├── logs # Log files inside are been updating with and stderr, in case you want to check how is everything going.
│   ├── busco
│   ├── fastqc
│   ├── prokka
│   ├── quast
│   ├── renaming
│   ├── spades
│   └── trim_galore
├── reads
├── results
│   ├── allfastqc
│   ├── busco
│   ├── prokka
│   ├── quast
│   ├── renamed_raw_reads
│   ├── spades
│   └── trimmed
└── workflow
    ├── Snakefile
    └── envs
```

### Comparing genomes

To compare genomes, you can use the `compare` rule. This rule uses dRep and checkM to compare different genomes. You can configure this tool in the config.yaml file (see [`config/README.md`](config/README.md) ).

There are 2 modes: by default, all samples are compared.

```bash
snakemake -c50 compare
```

or, if using conda and singularity:

```bash
snakemake -c50 compare --use-conda --use-singularity
```

However, it may be useful to compare only some of genomes with each other. In this case, the pipeline expects a file with as many lines as files and their respective addresses. For example,

```bash
cat subsample1.txt
results/spades/MST103_spades/MST103_contigs.fasta
results/spades/MST104_spades/MST104_contigs.fasta
results/spades/MST102_spades/MST102_contigs.fasta
```
Then, you need to indicate the file you are going to use (you can also do it by editing  [`config/config.yaml`](config/config.yaml))

```bash
snakemake -c50 compare --config SAMPLES_TO_COMPARE_FILEPATH="subsample1.txt"
```

Let's see the file structure resulting from running `compare` with all the files and with a subsample

``` bash
results/dRep/
├── all
│   ├── data
│   │   ├── Clustering_files
│   │   ├── MASH_files
│   │   └── fastANI_files
│   ├── data_tables
│   ├── dereplicated_genomes
│   ├── figures
│   └── log
└── subsample1
    ├── data
    │   ├── Clustering_files
    │   ├── MASH_files
    │   └── fastANI_files
    ├── data_tables
    ├── dereplicated_genomes
    ├── figures
    └── log
```

## Secondary metabolite biosynthetic gene clusters

To search a genome sequence for secondary metabolite biosynthetic gene clusters you can use [antismash](https://antismash.secondarymetabolites.org/#!/start). For running antismash against all genomes run:

```bash
snakemake -n antismash --use-singularity
snakemake -c100 antismash --use-singularity
```

This step relies on [singularity](https://docs.sylabs.io/guides/3.5/user-guide/introduction.html) to run antismash and BiG-SCAPE (it will not work without it).

Please check the [config file](config/config.yaml) for the flags you want to use and note that the ratio of the maximum number of threads indicated on the command line and threads indicated in the config file will determine whether the process will be parallelized or not.

> :warning: **This pipeline assumes that you want to use prokka annotated genomes for antismash**: That means there would be no gene finding in antismash. You can either modify this behaviour in [the configuration file](config/config.yaml) or run this step aside.

If you want to use [Big-Scape](https://bigscape-corason.secondarymetabolites.org/), you can also run the following code. Notice that .gbk files will be edited to have as an organism the filename (sample name).  Since this step relies in the previous one, the pipeline will figure out that it needs to run antimash before, **so you could simply run the following command**:

```bash
snakemake -n bigscape --use-singularity --singularity-args ' --writable'
snakemake -c100 bigscape --use-singularity --singularity-args ' --writable'
```

See [this issue](https://github.com/medema-group/BiG-SCAPE/issues/22) for more information about why we need the '--writable' argument.

## A few considerations for dealing with unexpected behavior

Because of Snakemake's way of deciding whether a step needs to be rerun, you may encounter undesired behavior. For example, rerunning an entire pipeline after modifying a single parameter (which only affects one of the final steps), moving a directory, or adding new reads.

If this happens (it will be reflected when using `snakemake -n` to view the schedule), there are some useful flags. As a rule of thumb, the desired effect is achieved by taking only the file modification date into account:

```bash
snakemake -n -rerun-triggers mtime
snakemake -c100 -rerun-triggers mtime
```

Another command of interest is `--touch`:

> Touch output files (mark them up to date without really changing them) instead of running their commands. This is used to pretend that the rules were executed, in order to fool future invocations of snakemake. Note that this will only touch files that would otherwise be recreated by Snakemake (e.g. because their input files are newer). For enforcing a touch, combine this with `--force`, `--forceall`, or` --forcerun`. Note however that you loose the provenance information when the files have been created in realitiy. Hence, this should be used only as a last resort. (default: False) (note that you can touch individual files, obtaining the same effect).

If you're 100% confident that all your files are up to date, you can run the next command to touch all files:

```bash
snakemake -R $(snakemake --list-code-changes --list-input-changes --list-params-changes --list-version-changes) --touch
snakemake -R $(snakemake --list-code-changes --list-input-changes --list-params-changes --list-version-changes) --touch -c1
```

As a last resort, you can always delete the folder with the results of the step you want to rerun and then run Snakemake normally.

## Tips to test different parameters

It is very important to note that, if we are only interested in getting one file, we can tell Snakemake directly.

For example, let's imagine that we want to rerun the 16s sequence fetch because, with the restrictions imposed, we have not obtained any sequence. To re-run the filtering step with a different minimum number of bp just for one file (without affecting other samples):

```bash
grep -c '>' results/barrnap/MST109_filtered_rrna.fa
1 # Just one sequence

snakemake -n results/barrnap/MST109_filtered_rrna.fa -rerun-triggers mtime
Building DAG of jobs...
Nothing to be done (all requested files are present and up to date). # Expected behavior

rm results/barrnap/MST109_filtered_rrna.fa
snakemake -c1 results/barrnap/MST109_filtered_rrna.fa -rerun-triggers mtime --config minumun_16s_allowed=0
grep -c '>' results/barrnap/MST109_filtered_rrna.fa
9
```

Another very useful tool is `--summary`, which prints a table associating each output file with the rule used to generate it, the creation date, and, optionally, the version of the tool used for creation. This flag, in addition, to grep, can be very useful for checking the status of files.

```bash
snakemake --summary -rerun-triggers mtime | grep 'contigs.fasta'

Building DAG of jobs...
results/spades/MST103_spades/MST103_contigs.fasta       Sat Sep 10 10:44:32 2022        assembling_genome       -       logs/spades/MST103.log    ok      no update
results/spades/MST109_spades/MST109_contigs.fasta       Sat Sep 10 09:57:56 2022        assembling_genome       -       logs/spades/MST109.log    ok      no update
results/spades/MST102_spades/MST102_contigs.fasta       Sat Sep 10 10:35:02 2022        assembling_genome       -       logs/spades/MST102.log    ok      no update
```

gha_language: Python

gha_license_id: MIT


text:
---
layout: post
title: "Tech Breakfast #31 - To Infinity & Beyond: A Case Study on Refactoring"
category: articles
tags:
  - Refactoring
  - Web
image:
  feature: refactor.jpg
---

When inheriting projects, we aren't always going to receive a code-base that is susceptible to change.  The code-base does everything the client wants, pleases the user and functions, but is unmaintainable, hard to read, and unpleasant to work on.  Something has to be done, but what? Refactor.

{% include speakerdeck.html deckId="f7d139fc57d6441cb3f50550a3c08f5e" %}

gha_language: CSS


text:
# php-login-register
Responsive ,efficent and robust  login/register php pages.

Features:
- Responsive page with complete validation
- Ajax for no page refresh
- Completely secured storage for data

Screenshots:

Login Page:
![image](https://cloud.githubusercontent.com/assets/20264110/16687154/6b4e21d2-4534-11e6-83ef-abdacae9fcdf.png)
Register page:
![image](https://cloud.githubusercontent.com/assets/20264110/16687139/58901956-4534-11e6-8c7c-2f4cc9d73b5d.png)
Validating the form:
![image](https://cloud.githubusercontent.com/assets/20264110/16687178/9676209e-4534-11e6-9c7a-8f689de69fcb.png)
Successfull Registeration:
![image](https://cloud.githubusercontent.com/assets/20264110/16687202/b94ace08-4534-11e6-9ad9-fa9a19db3c83.png)
Logging in:
![image](https://cloud.githubusercontent.com/assets/20264110/16687584/23d44f72-4537-11e6-963a-b02b240c2e21.png)


text:
Two lines that are not parallel meet in a single point.

If the equations for the lines are $y=mx+c$ and $y=nx+d$, then the point
of intersection is at: $$\begin{eqnarray*}
x&=&\frac{d-c}{m-n}\\
y&=&\frac{cn-dm}{n-m}
\end{eqnarray*}$$


text:
# ScopeJS
A JavaScript library for defining and working with lexical scopes.

[ScopeJS is hosted on github](http://github.com/dbrans/scope).

Latest version: [0.10.1](http://github.com/dbrans/scope)

## Table of Contents
- [Installation and Usage](#usage)
- [Theory](#theory)
- [API](#api)
- [Annotated Source](documentation/docs/scope.html)
- [CoffeeScript Support](#coffee)

## Overview

    // Create a new lexical scope
    var scope = Scope.create({
      locals: {foo: 3},
      literals: {
        multiplyFoo: function(val) {foo *= val;}
      }
    });

    // Call a local function.
    scope.multiplyFoo(4);

    // Evaluate an expression locally.
    scope.eval('foo++');

    // Get a local variable.
    log(scope.foo); // 13

    // Extend a scope.
    var inner = scope.extend({
      locals: {bar: 'bar'}
    });

    // Inner scope has access to the outer scope:
    inner.foo = 6;
    log(scope.foo));        // 6

    // But outer scope does not have access to the inner scope:
    log(scope.eval('bar')); // Error: bar is not defined.

~anchor:usage
## Installation

There are two ways to install ScopeJS: via npm or via git.

### NPM

1. If you haven't already, [install npm](http://howtonode.org/introduction-to-npm).
2. `npm install -g scope`
(Leave off the -g if you don't wish to install globally.)

### Git
To install from source, you should have [CoffeeScript](http://coffeescript.org) installed.

Then run:

    git clone git://github.com/dbrans/scope.git
    cd scope
    cake install

## Including ScopeJS in your project.

### CommonJS Environment (e.g., Nodejs)

    var Scope = require('scope').Scope;
    var scope = Scope.create(...);

### Browser Environment
Include `browser/scope.js` in your html page:

    <script type="text/javascript" src="PATH_TO_SCOPE/browser/scope.js"></script>

`browser/scope.js` defines `Scope` as a global variable. So you can just go
ahead and use it:

    var scope = Scope.create(...);

## Community
Please use the [github page for ScopeJS](http://github.com/dbrans/scope)
to discuss and raise issues about ScopeJS.

~anchor:theory
## Theory
### A 'scoped eval' function

Consider this function literal as a string:

    var EVAL_LITERAL = "(function (expr) {return eval(expr)})";

Eval-ing this string in some lexical scope creates a 'scoped eval':
a function that, if passed outside of the current scope, can evaluate
expressions inside that scope. Here's a demonstration:

    var scopeEval = (function() {
      var x = 3;
      return eval(EVAL_LITERAL);
    })();

    log(scopeEval('x')); // 3
    log(eval('x'));      // Error: x is not defined

In general, given a list of variable names `var1`, `var2`, ..., you can dynamically
create a lexical scope where those variables are defined like this:

    var vars = ['var1', 'var2', ..., 'varN'];

    var scopeEval = (function() {
      eval("var " + vars.join(',') + ';' + EVAL_LITERAL);
    })();

This idea of a scoped eval function is at the heart of ScopeJS.

Let's develop this idea a little bit.

### _Current_ vs _Target_ scopes
Let's call the scope in which our examples are imagined to run
the _current scope_
and the scope in which a scopeEval expression runs, the _target scope_.

For simplicity, in our examples the current scope is the outer scope of
the target scope. In other words,

    // For simplicity:
    // CURRENT SCOPE
    var scopeEval = (function() {
      // TARGET SCOPE
      eval("var " + vars.join(',') + ';' + EVAL_LITERAL);
    })();

In ScopeJS, however, the target scope lives somewhere else and cannot see the
current scope (see the `Scope.eval` class method in the
[Annotated Source](documentation/docs/scope.html))

### Undeclared Variables
In these examples, the variables var1, var2, etc. refer to variables that
are already declared in the target scope. Any assignment to undeclared
variables is a typo and would create global variables.

### Literal Expressions
We can evaluate array, object and function literals inside the target scope.
Those literals have access to local variables. Consider:

    scopeEval('var1 = 3');

    var obj = scopeEval('[var1]');

    log(obj[0]);           // 3

#### Function Literals and Closures
Function literals that are eval'd in a scope close over local variables in that
scope and can manipulate them directly:

    var addToVar = scopeEval('(function(x){var1 += x})');

    addToVar(2);

    log(scopeEval('var1')); // 5

### Extending the Target Scope
As you may have noticed, it's impossible to
declare a new variable in the target scope, once it is created.
This is because the eval happens in the function local scope described
by EVAL_LITERAL.
This means that newly declared variables only exist for the duration of that
function's invocation:

    // Here 'newvar' exists:
    log(scopeEval('var newvar = 3; newvar')); // 3

    // ...but here it doesn't:
    scopeEval('newvar'); // Error: newvar is not defined

Instead, we create an inner scope where new variables are defined.

    var newvars = ['newvar1', 'newvar2', ...];
    var innerEval = scopeEval('var ' + newvars.join(',') + ';' + EVAL_LITERAL);

Because we used `scopeEval` to create our lexical scope, innerEval and scopeEval
share variables var1, var2, etc. In addition, innerEval has it's own variables
newvar1, newvar2, ...

    // var1 is visible in both scopes
    innerEval('var1 = 6');
    log(scopeEval('var1'));      // 6

    // newvar1 is only visible in the inner scope
    innerEval('newvar1 = 'A');
    log(scopeEval('newvar1'));   // Error: newvar1 is not defined.

We might say that `innerEval` _extends_ `scopeEval` because `innerEval` includes
`scopeEval`'s variables and declares new ones.

`scopeEval` is actually the _outer_ lexical scope of `innerEval`, or rather the target
scopes of those two functions. A static view of the nested
scopes we've created so far might look like this:

    // GLOBAL SCOPE
    (function() {
      // scopeEval's target scope
      var var1, var2, ...;
      (function () {
        // innerEval's target scope
        var newvar1, newvar2, ...;
      })()
    })()

~anchor:api
## API
For API documentation, please refer to the [annotated source](documentation/docs/scope.html)

~anchor:coffee
## CoffeeScript support
Every root scope defines all the coffeescript helpers that are currently generated by the CoffeeScript
compiler. This means that recompiling functions that were once compiled from CoffeeScript will
work fine:

    # This is coffeescript
    scope.eval ->
       # invoke CoffeeScript's __bind runtime helper.
       =>

If you have a string of CoffeeScript code that you want to compile or run inside a target scope,
use

    CoffeeScript.compile code, bare:true

to compile `code` first. 'Native' CoffeeScript eval/compile
support could be envisioned for the future.

One important caveat when __setting a variable in the target scope__ inside a CoffeeScript function:
The CoffeeScript compiler does lexical analysis around a function to decide if a variable has been seen
before or if it needs to be declared inside the function.
So, unless you have a local variable with the same name in the current scope, the CoffeeScript compiler will
declare a variable with that name inside your function, even if that variable exists in the target scope.
This function-local variable will mask the one in the target scope:

    scope = Scope.create
      locals:
        foo: 3
      literals:
        # Won't have the desired effect.
        setFoo: (val) -> foo = val

In the above code, calling `scope.setFoo 4` will have no effect since, in the current scope,
 CoffeeScript compiles `(val) -> foo = val` to

    function(val) {
      var foo;
      return foo = val;
    }

A fix which analyzes decompiled CoffeeScript functions is possible.

repository: dbrans/scope

repo_name: dbrans/scope


text:
Metabolic landscape of single cells in the tumor microenvironment
============
Introduction
------------
This pipeline is responsible for analyzing metabolic gene expression profiles in the tumor microenvironment using single-cell RNA sequencing data.

![pipeline](pipeline.png "Schematic representation of single-cell RNA-seq data analysis")

There are 7 main steps involved in data processing which organized in different folders.

Requirements
------------
The required R packages for this pipeline can be installed by:

``` bash
Rscript install_requiredPackages.R
```
Download and read the datasets
-----------------------------
``` bash
cd "1-ReadData"
bash download_dataset.sh
Rscript readData_head_neck.R
Rscript readData_melanoma.R
cd ../
```
The gene expression profiles and the annotations of the cell types will be stored as R objects.

Imputation of the missig values
-------------------------------
``` bash
cd "2-Imputation"
Rscript impute_tpm.R melanoma
Rscript impute_tpm.R head_neck
cd ../
```
This step uses the ["scImpute"](https://github.com/Vivianstats/scImpute) package to impute the missing values in gene expression profiles.

Normalization and evaluation of different normalization methods
---------------------------------------------------------------
``` bash
cd "3-Normalization"
Rscript normalization.R melanoma
Rscript normalization.R head_neck
cd ../
```
Four commonly used data normalization methods are applied on each dataset. The distribution of relative gene expression of each cell type will be ploted to evaluate and select the best normalization method.

Landscape of the metabolic gene expression profile
--------------------------------------------------
``` bash
cd "4-Clustering"
Rscript metabolic_landscape.R melanoma
Rscript metabolic_landscape.R head_neck
Rscript inter_tumor_distance.R melanoma
Rscript inter_tumor_distance.R head_neck
cd ../
```
The t-SNE algorithm will be performed in this step for visualizing metabolic gene expression in millions of cells (The result may be slightly different with the figure in manuscript due to the random initialization). The spearman correlation matrix will aslo be generated to show the inter-tumor heterogeneity using the metabolic genes.

Metabolic pathway activities in different cell types
--------------------------
``` bash
cd 5-PathwayActivity
Rscript scRNA_pathway_activity.R melanoma
Rscript scRNA_pathway_activity.R head_neck
Rscript TCGA_pathway_activity.R
cd ..
```
This step will calculate the metabolic pathway activities for different single cell populations or bulk tumor/normal samples. The scatter plot will show the discrepancy of pathway activities between single malignant cells and bulk tumors. The violin plot will show the distribution of metabolic pathway activities in single cell populations or bulk tumor/normal samples.

*The bulk RNA-seq data was downloaded from TCGA website, please see the instruction of data downloading and preprocessing in Data/TCGA/README.md*

Metabolic pathway heterogeneity
-------------------------------
``` bash
cd 6-PathwayHeterogeneity
Rscript intra_malignant_heterogeneity.R melanoma
Rscript intra_malignant_heterogeneity.R head_neck
Rscript intra_non-malignant_heterogeneity.R melanoma
Rscript intra_non-malignant_heterogeneity.R head_neck
Rscript OXPHOS_Glycolysis_Hypoxia_Correlation_plot.R melanoma
Rscript OXPHOS_Glycolysis_Hypoxia_Correlation_plot.R head_neck
Rscript OXPHOS_Glycolysis_Hypoxia_Correlation_plot-CCLE.R
Rscirpt GeneSignature-of-Low_OXPHOS_Glycolysis_Hypoxia.R melanoma
Rscript GeneSignature-of-Low_OXPHOS_Glycolysis_Hypoxia.R head_neck
cd ..
```
In this step, the PCA and GSEA analysis will be performed to investigate the metabolic pathway heterogeneity across single cells in malignant and non-malignant cell populations. The scatter plots will be performed to compare activities of OXPHOS, glycolysis and response to hypoxia in single malignant cells and cultured cell lines from CCLE database. The gene signatures in single cells with low OXPHOS/glycolysis/hypoxia activity will be identified and stored as the text files, which can be used as the input of GO analysis on the website: http://metascape.org

Metabolic features of nonmalignant cell subtypes
-----------------------------------
``` bash
cd 7-MetabolicFeatures
Rscript non-malignant_subtype.R melanoma
Rscript non-malignant_subtype.R head_neck
cd ..
```
The metabolic features in different T-cell subtypes and fibroblast subtypes will be identified in this step.

Contact
-------
zhengtao.xiao[at]xjtu.edu.cn

gha_language: R


text:
# Gmail Rapid Scrolling Extension
Allows the user to rapidly scroll over their mails by pressing "n" (next) in the keyboard. The window supports auto-scroll by adjusting the view and zooming at colorful text.

### Usage
Clone the repository, go to the extensions tab `chrome://extensions`, press **Load unpacked** and select the cloned directory. The extension will launch automatically once you open Gmail on any tab. For more settings go to *Details* > Extension options.


text:
# **************IMPORTANT NOTICE ************
## As of __30 September__ 2015, the OpenCalais API on which this gem is built, has been discontinued by Thomson-Reuters. A new and significantly changed API is now in use by OpenCalais. You can read about the changes [here](http://www.opencalais.com/upgrade/). Unfortunately this means that DoverToCalais is no longer functional. I don't know -at this stage- if and when I'll upgrade this gem to the new API. Thank you for your time and effort in using DoverToCalais.

# DoverToCalais

DoverToCalais allows the user to send a wide range of data sources (files & URLs)
to [OpenCalais](http://www.opencalais.com/about) and receive asynchronous responses when [OpenCalais](http://www.opencalais.com/about) has finished processing
the inputs. In addition, DoverToCalais enables response filtering in order to find relevant tags and/or tag values.

## What is OpenCalais?
In short -and quoting the [OpenCalais](http://www.opencalais.com/about) creators:
> "*The OpenCalais Web Service automatically creates rich semantic metadata for the content you submit – in well under a second. Using natural language processing (NLP), machine learning and other methods, Calais analyzes your document and finds the entities within it. But, Calais goes well beyond classic entity identification and returns the facts and events hidden within your text as well.*"

In general, OpenCalais Simple XML Format (the one used by DoverToCalais) returns three kinds of tags: [Entitites, Events](http://www.opencalais.com/documentation/calais-web-service-api/api-metadata/entity-index-and-definitions) and [Topics](http://www.opencalais.com/documentation/calais-web-service-api/api-metadata/document-categorization). ***Entities*** are static 'things', like Persons, Places, et al. that are involved in the textual context in some capacity. OpenCalais assigns a *relevance* score to each entity to indicate it's relevance within the context of the data source's general topic.  ***Events*** are facts or actions that pertain to one or more Entities.  ***Topics*** are a characterisation or generic description of the data source's context.

We can use these tags and the information within them to extract relevant information from the data or to draw useful conclusions about it. For example, if the data source tags include an *<Event>* with the value of *'CompanyExpansion'*, I can then look for the <City> or  <Company> tags to find out which company is expanding and if it's near my location (hint: they may be looking for more staff :))  Or, I could pick out all <Company>s involved in a <JointVenture>, or all  <Person>s implicated in an  <Arrest> in my  <City>, etc.

DoverToCalais, from version 0.2.1 onwards also supports the OpenCalais rich [JSON Output format](http://www.opencalais.com/documentation/calais-web-service-api/interpreting-api-response/opencalais-json-output-format). This format returns relationships between entities, as well as the previous tags returned by the Simple XML format, thus allowing a deeper level of data analysis and detection.

## Why use OpenCalais?
There are many reasons, mainly to:

 * incorporate tags into other applications, such as search, news aggregation, blogs, catalogs, etc.
 * enrich search by looking for deeper, contextual meaning instead of merely phrases or keywords.
 * help to discern relationships between semantic entities.
 * facilitate data processing and analysis by allowing easy identification of relevant or important data sources and the discarding of irrelevant ones.

## DoverToCalais Features
1. **Multiple data source support**: Thanks to the power of [Yomu](https://github.com/Erol/yomu), DoverToCalais can process a vast range of files (and, of course, web pages), extract text from them and send
them to OpenCalais for analysis and tag generation.

2. **Asynchronous responses (callbacks)**:
Users can set callbacks to receive the processed meta-data, once the OpenCalais Web Service response has  been received.
Furthermore, a user can set multiple callbacks for the same request (data source), thus enabling cleaner,
more modular code.

3. **Result filtering**: DoverToCalais uses the OpenCalais [Simple XML Format](http://www.opencalais.com/documentation/calais-web-service-api/interpreting-api-response/simple-format) as the preferred response format. The user can work directly with the XML-formatted response, or -if feeling a bit lazy- can take advantage of the DoverToCalais filtering functionality and receive specific entities, optionally  based on specified conditions.

For more details of the features and code samples, see [Usage](#usage).

##Pre-requisites and dependencies

To use the OpenCalais Web Service  and -by extension- DoverToCalais, one needs to possess an OpenCalais API key, which is easily obtainable from the [OpenCalais web site](http://www.opencalais.com/APIkey).

DoverToCalais requires the presence of a working [JRE](http://en.wikipedia.org/wiki/JRE#Execution_environment).

Also, if you're going to use the rich JSON output format, you'll need to have [Redis](http://redis.io/topics/quickstart) running on an accessible node.

## Installation

Add this line to your application's Gemfile:

    gem 'dover_to_calais'

And then execute:

    $ bundle

Or install it yourself as:

    $ gem install dover_to_calais

## Compatibility
DoverToCalais has been developed in Ruby 1.9.3 and should work fine on post-1.9.3 MRI versions too. If anyone is succesfully running it on other Ruby runtimes please let me know.

## Usage
Using DoverToCalais is extremely simple.

### The Basics
As DoverToCalais uses the awesome-ness of [EventMachine](http://rubyeventmachine.com/), code must be placed within an EM *run* block:

```ruby
EM.run do

    # use Control + C to stop the EM
    Signal.trap('INT')  { EventMachine.stop }
    Signal.trap('TERM') { EventMachine.stop }

    # we need an API key to use OpenCalais
    DoverToCalais::API_KEY =  'my-opencalais-api-key'
    # create a new dover
    dover =  DoverToCalais::Dover.new('http://www.bbc.co.uk/news/world-africa-24412315')
    # parse the text and send it to OpenCalais
    dover.analyse_this
    puts 'do some stuff....'
    # set a callback for when we receive a response
    dover.to_calais { |response| puts response.error ? response.error : response }

    puts 'do some more stuff....'

end
```
This will produce the following result:

> do some stuff....

> do some more stuff....

> <?xml version="1.0"?>

> <OpenCalaisSimple>

> ..........

> (the rest of the XML response from OpenCalais)

As can be observed, the callback (#to_calais) is trigerred after the rest of the code has been executed and only when the OpenCalais request has been completed.

Of course, we can analyse more than one sources at a time:

```ruby
EM.run do

  # use Control + C to stop the EM
  Signal.trap('INT')  { EventMachine.stop }
  Signal.trap('TERM') { EventMachine.stop }

  DoverToCalais::API_KEY =  'my-opencalais-api-key'

  d1 =  DoverToCalais::Dover.new('http://www.bbc.co.uk/news/world-africa-24412315')
  d2 =  DoverToCalais::Dover.new('/home/fred/Documents/RailsRecipes.pdf')
  d3 =  DoverToCalais::Dover.new('//network-drive/annual_forecast.doc')

  d1.analyse_this; d2.analyse_this; d3.analyse_this;

  puts 'do some stuff....'

  d1.to_calais { |response| puts response.error ? response.error : response }
  d2.to_calais { |response| puts response.error ? response.error : response }
  d3.to_calais { |response| puts response.error ? response.error : response }

  puts 'do some more stuff....'

end
```

This will output the two *puts* statements followed by the three callbacks (d1, d2, d3) in the order in which they are triggered, i.e. the first callback to receive a response from OpenCalais will fire first.

###Filtering the response
Why parse the response XML ourselves when DoverToCalais can do it for us? We'll just use the *#filter* method on the response object, passing a filtering hash:

 ```ruby
     my_filter = {:entity => 'Entity1', :value => 'Value1', :given => {:entity => 'Entity2', :value => 'Value2'}}
     reponse.filter(my_filter)
 ```

The above tells DoverToCalais to look in the reponse for an entity called 'Entity1' with a value of 'Value1', **only** if the response contains an entity called 'Entity2' which has a value of 'Value2'.

The conditional clause (*:given*) is optional; the filtering hash can be used in pretty much any permutation.  For instance:

```ruby
EM.run do

    DoverToCalais::API_KEY =  'my-opencalais-api-key'

    dover =  DoverToCalais::Dover.new('http://www.bbc.co.uk/news/world-africa-24412315')
    dover.analyse_this

    dover.to_calais do |response|
    if   response.error
      puts  response.error
    else
      puts response.filter({:entity => 'Company'})
    end
    end

end
```

This will pick out all entities tagged 'Company' from the data source. The output will be an Array of ResponseItem objects.

> <struct DoverToCalais::ResponseItem name="Company", value="BBC News", relevance=0.654, count=13, normalized=nil, importance=nil, originalValue=nil>

> <struct DoverToCalais::ResponseItem name="Company", value="TV Radio", relevance=0.565, count=2, normalized="HERALD & WEEKLY-TV,RADIO OPS", importance=nil, originalValue=nil>

> <struct DoverToCalais::ResponseItem name="Company", value="Reuters", relevance=0.255, count=2, normalized="THOMSON REUTERS GROUP LIMITED", importance=nil, originalValue=nil>

> <struct DoverToCalais::ResponseItem name="Company", value="Twitter", relevance=0.395, count=1, normalized="TWITTER, INC.", importance=nil, originalValue=nil>

> <struct DoverToCalais::ResponseItem name="Company", value="Huffington Post UK", relevance=0.136, count=1, normalized=nil, importance=nil, originalValue=nil>

> <struct DoverToCalais::ResponseItem name="Company", value="Ireland Kenya", relevance=0.144, count=1, normalized=nil, importance=nil, originalValue=nil>

> <struct DoverToCalais::ResponseItem name="Company", value="Yahoo! UK", relevance=0.144, count=1, normalized="YAHOO! UK LIMITED", importance=nil, originalValue=nil>

If this output looks a bit cluttered, we can easily tidy it up:

```ruby
EM.run do

  DoverToCalais::API_KEY =  'my-opencalais-api-key'

  dover =  DoverToCalais::Dover.new('http://www.bbc.co.uk/news/world-africa-24412315')
  dover.analyse_this

  dover.to_calais do |response|
    if   response.error
      puts  response.error
    else
      items = response.filter({:entity => 'Company'})
      items.each do |item|
        puts "#{item.name}: #{item.value}, relevance = #{item.relevance}"
      end
    end
  end

end
```

Which will give us:

> Company: BBC News, relevance = 0.656

> Company: TV Radio, relevance = 0.566

> Company: Reuters, relevance = 0.26

> Company: Guardian.co.uk, relevance = 0.143

> Company: Twitter, relevance = 0.399

> Company: Huffington Post UK, relevance = 0.132

> Company: Ireland Kenya, relevance = 0.139

> Company: Yahoo! UK, relevance = 0.139

Let's see if the data source refers to any business partnerships:

```ruby
EM.run do

  DoverToCalais::API_KEY =  'my-opencalais-api-key'

  dover =  DoverToCalais::Dover.new('http://www.bbc.co.uk/news/technology-24380202')
  dover.analyse_this

  dover.to_calais do |response|
    if   response.error
      puts  response.error
    else
      items = response.filter({:entity => 'Event', :value => 'Business Partnership'})
      puts "There are #{items.length} events like that in the source"
    end
  end

end
```

which will produce:

> There are 1 events like that in the source

Now let's find all companies involved in any business partnerships:

```ruby
EM.run do

  DoverToCalais::API_KEY =  'my-opencalais-api-key'

  dover =  DoverToCalais::Dover.new('http://www.bbc.co.uk/news/technology-24380202')
  dover.analyse_this

  dover.to_calais do |response|
    if   response.error
      puts  response.error
    else
      items = response.filter( {:entity => 'Company', :given => {:entity => 'Event',  :value => 'Business Partnership'}} )
      items.each do |item|
        puts "#{item.name}: #{item.value} a.k.a #{item.normalized}, relevance = #{item.relevance}"
      end
    end
  end

end
```

which gives us:

> Company: BBC News a.k.a , relevance = 0.678

> Company: Google a.k.a GOOGLE INC., relevance = 0.508

> Company: Flutter a.k.a FLUTTER COM INC, relevance = 0.531

> Company: TV Radio a.k.a HERALD & WEEKLY-TV,RADIO OPS, relevance = 0.558

> Company: Microsoft a.k.a MICROSOFT CORPORATION, relevance = 0.303

> Company: Adobe a.k.a ADOBE SYSTEMS INCORPORATED, relevance = 0.193

> Company: Netflix a.k.a NETFLIX, INC., relevance = 0.301

> Company: Y Combinator a.k.a Y Combinator, relevance = 0.258

> Company: Nintendo a.k.a Nintendo Co., Ltd., relevance = 0.286

> Company: Samsung a.k.a Samsung C&T Corporation, relevance = 0.285

> Company: Glyndwr University a.k.a , relevance = 0.269

At this point, someone may ask: "But what if we want to get more than one entity for a given condition? The filter hash doesn't allow that!"

No it doesn't. However, given that filtering is done on the *whole* reponse *after* it's been received, we can apply many filters on the same response:

```ruby
EM.run do

  DoverToCalais::API_KEY =  'my-opencalais-api-key'

  dover =  DoverToCalais::Dover.new('http://www.bbc.co.uk/news/technology-24380202')
  dover.analyse_this

  dover.to_calais do |response|
    if   response.error
      puts  response.error
    else
      result1 = response.filter( {:entity => 'Company', :value => 'Google', :given => {:entity => 'Technology',  :value => 'gesture recognition'}} )
      result2 = response.filter( {:entity => 'Product', :given => {:entity => 'Technology',  :value => 'gesture recognition'}} )
      puts result1 | result2
    end
  end

end
```

Which will give us all the gesture-recognition products that Google is associated with according to our data source:

> <struct DoverToCalais::ResponseItem name="Company", value="Google", relevance=0.506, count=7, normalized="GOOGLE INC.", importance=nil, originalValue=nil>

> <struct DoverToCalais::ResponseItem name="Product", value="Xbox Kinect", relevance=0.286, count=1, normalized=nil, importance=nil, originalValue=nil>

> <struct DoverToCalais::ResponseItem name="Product", value="Galaxy S4 smartphone", relevance=0.282, count=1, normalized=nil, importance=nil, originalValue=nil>

> <struct DoverToCalais::ResponseItem name="Product", value="Wii", relevance=0.286, count=1, normalized=nil, importance=nil, originalValue=nil>

> <struct DoverToCalais::ResponseItem name="Product", value="Galaxy S4", relevance=0.282, count=1, normalized=nil, importance=nil, originalValue=nil>

***PS***: If you're not sure about the names or values of the tags you want to filter, you can get a listing with the following Constants:

```ruby
CalaisOntology::CALAIS_ENTITIES
CalaisOntology::CALAIS_EVENTS
CalaisOntology::CALAIS_TOPICS
```

### Rich output format
Since version 0.2.1, DoverToCalais users can request to receive the OpenCalais output in OpenCalais's rich JSON format. This has the advantage of producing relation data, which allows for a much deeper level of analysis and data detection.

The rich format can be requested simply by passing one of the following arguments to the *#analyse_this* method: *:rich, :rich_format, :rich_output*, i.e.

```ruby
dover.analyse_this :rich
```
When DoverToCalais processes the *rich* output, it will create a pseudo-relational data model on Redis. The model can then be be queried and searched using standard Ruby and the [Ohm](https://github.com/soveran/ohm) API.

The only difference in DoverToCalais usage when using the *rich* output is that there's no longer a need to do our response analysis in the callback (*#to_calais* method). The callback now only serves to let us know when the response has been processed. Once the callback returns, we know that we can find all our source data nicely modelled in Redis and we can access it ouside and independently of our EventMachine create->analyze->callback loop.

#### The Data Model
DoverToCalais creates three kinds of key classes on Redis: *DoverToCalais::EntityModel, DoverToCalais::EntityModel::RelationModel, DoverToCalais::EntityModel::EventModel*.

As is suggested by the namespacing, the relational aspects of the model are that an EntityModel *has* a number of RelationModels and a number of EventModels. Knowing this simple fact, it becomes fairly straightforward to discover which entities are inter-connected and how.

*DoverToCalais::EntityModel* has the following attributes

* name - the entity name - String
* type - the entity type, e.g. Person, Location, etc - String
* calais_id - a unique id assigned by OpenCalais - String
* relations - a set of generic relations connected to the entity - Set
* events - a set of events connected to the entity - Set

*DoverToCalais::EntityModel::RelationModel* has the following attributes

* subject - the entity applying the action - Hash
* verb - an action - String
* object - the entity receiving the action - Hash
* detection -  the most accurate string description of the relation - String
* calais_id - a unique id assigned by OpenCalais - String

*DoverToCalais::EntityModel::EventModel* has the following attributes

* calais_id - a unique id assigned by OpenCalais - String
* info_hash - a Hash incorporating the event's attributes and values. As the number of attributes depends on the type of event (e.g. MilitaryAction will have very different attributes to MovieRelease), the info_hash is a good way to dynamically encapsulate an event's attributes.

#### Redis
DoverToCalais relies on [Redis](http://redis.io/topics/quickstart) to store processed responses in the rich JSON format. By default, DoverToCalais will use the local Redis instance <IP_ADDRESS>, on the default port 6379 with database #6 (no password). If any of this proves inconvenient, it can be changed by modifying the constant *DoverToCalais::REDIS*.

#### Rich output usage

```ruby
#probably good idea to clear Redis DB first
DoverToCalais::flushdb
EM.run do

    # use Control + C to stop the EM
    Signal.trap('INT')  { EventMachine.stop }
    Signal.trap('TERM') { EventMachine.stop }

    # we need an API key to use OpenCalais
    DoverToCalais::API_KEY =  'my-opencalais-api-key'
    # create a new dover
    dover =  DoverToCalais::Dover.new('http://www.bbc.co.uk/news/world-africa-24412315')
    # parse the text and send it to OpenCalais
    dover.analyse_this :rich
    puts 'do some stuff....'
    # set a callback for when we receive a response
    dover.to_calais { |response| puts response.error ? response.error : "finished!" }

    puts 'do some more stuff....'

end
```

As you can see, this isn't much different from our bread-and-butter usage with the simple format. The differences are:

* we clear the Redis data store before we begin. Not a necessary step, but we may want to start on a clean slate. DoverToCalais provides its own method for doing that and it's recommended this method is used instead of an external Redis command .
* we pass the *:rich* symbol to *#analyse_this*.
* in our callback, instead of doing something with the response we simply notify when it's done. What this means is that DoverToCalais has processed the response and has created a data model on Redis. We can now go and search our data model.

```ruby
require 'dover_to_calais'

  #make sure we're connected to the DoverToCalais data-store
  Ohm.redis = Redic.new(DoverToCalais::REDIS)

  #let's find out how many entities we have in our store
  puts DoverToCalais::EntityModel.all.to_a.length
  puts DoverToCalais::EntityModel::RelationModel.all.to_a.length
  puts DoverToCalais::EntityModel::EventModel.all.to_a.length

  #find all relations where the subject is Jesse James
  all_relations = DoverToCalais::EntityModel::RelationModel.all.to_a
  selected = all_relations.select {|v| v.subject['name'].eql?("Jesse James")}
  selected.each do |relation|
    puts relation.subject, relation.verb, relation.object, relation.detection
  end

  #get JFK (the Person, not the Airport)
  presidents = DoverToCalais::EntityModel.find(name: "JFK", type: "Person")

  #make sure there's only one
  if presidents.size == 1
    the_president = presidents.first
    #get all JFK-related events
    the_president.events.each do |e|
      e.info_hash.each_pair do |k,v|
        puts "#{k}: #{v}"
      end
    end

  end

```

###Code samples

More examples of using DoverToCalais can be found as GitHub Gists:

[Using DoverToCalais to semantically tag all files in a directory](https://gist.github.com/RedFred7/6961349)
[Use DoverToCalais to find all Persons or Organizations with a relevance score greater than 0.1, if the data source contains an environmental event](https://gist.github.com/RedFred7/6961853)

### Using a Proxy

If you're behind a corporate firewall and the only way to reach outside is through a proxy then you need to set the *DoverToCalais::PROXY* constant:

```ruby
    DoverToCalais::PROXY =
        :proxy => {
           :host => 'www.myproxy.com',
           :port => 8080,
           :authorization => ['username', 'password'] #optional
        }
```

If you're connecting through a SOCKS5 Proxy just set the *:type* key to :socks5.

```ruby
    DoverToCalais::PROXY =
        :proxy => {
           :host => 'www.myproxy.com',
           :port => 8080,
           :type => :socks5
        }
```

## Documentation

Comprehensive documentation can be found at [rubydoc](http://rubydoc.info/gems/dover_to_calais) and also at [omniref](http://www.omniref.com/?utf8=%E2%9C%93&q=dovertocalais&p=0&r=20&commit=Search).

## Testing

A list of Cucumber features and scenarios can be found in the *features* directory. The list is far from exhaustive, so feel free to add your own scenarios and steps.

To run the tests, there is already a rake task set up. Just type:

    rake features API_KEY='my_api_key'

## Contributing

1. Fork it
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Add some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
5. Create new Pull Request

##Changelog

   * **07-Oct-2013** Version: 0.1.0
Initial release
   * **10-Feb-2014** Version: 0.1.1
Improved Response error message
   * **10-Feb-2014** Version: 0.2.0
Added #analyse_this to public interface
   * **24-Mar-2014** Version: 0.2.1
New feature: rich JSON output analysis and Redis data modelling

# **************IMPORTANT NOTICE ************
## As of __30 September__ 2015, the OpenCalais API on which this gem is built, has been discontinued by Thomson-Reuters. A new and significantly changed API is now in use by OpenCalais. You can read about the changes [here](http://www.opencalais.com/upgrade/). Unfortunately this means that DoverToCalais is no longer functional. I don't know -at this stage- if and when I'll upgrade this gem to the new API. Thank you for your time and effort in using DoverToCalais.


text:
# McDinero

![McDinero icon](https://github.com/smile0210/mcdinero/blob/master/proj.ios_mac/ios/Images.xcassets/AppIcon.appiconset/Icon-App-83.5x83.5%402x.png)

McDinero is a mini video game inspired by the viral video https://youtu.be/DKUCGh1mH0c

You need to collect as much money as you can, win the rap fights and avoid the "Error" and the "Monarquía Española"

This game use the open source framework **Cocos2dx V4** https://www.cocos.com/en/docs#2dx

# iOS project

Once you downloaded the repo:
- In the root project directory create a folder called **ios-build** with `mkdir ios-build`
- Go to the new folder `cd ios-build`
- Create the Xcode project with `cmake .. -GXcode -DCMAKE_SYSTEM_NAME=iOS -DCMAKE_OSX_SYSROOT=iphoneos`
- Open the Xcode created at `ios-build`
- At the top left corner select `McDinero` and click play
- Have fun!

# Android project
Once you downloaded the repo:
- Open the project **proj.android** with Android Studio
- Open your simulator or connect your device
- Run the project
- Have fun!

# Requirements & References
- Both platforms require `cmake` in order to compile the source files, install it before compile
  - I recommend install it with `homebrew` but you can also refer to the official documentation https://cmake.org/install/
- For iOS you need to install Xcode in order to compile the game, also, you can refer to this video to setup your development environment https://youtu.be/GKT_40cc-a0
- For Android you need to install Android Studio and the NDK, please refer to this video to setup you development environment https://youtu.be/veBSTU0c2YY


text:
# Structure definitions
This directory contains decoded game structures.\
Decided not to write all decoded structures in js, because current project is crude. Feel free to suggest on how to improve basic functionaility.\
If you want to contribute, then only publish js code, don't publish Reclass.net project file (I will sync the changes when I have free time). It would be good to make Reclass.net project file generation from js, but it seems too much of an effort (from quick REClass.net source code inspection).\
Alternatively you can [use lua inside of Cheat Engine](https://wiki.cheatengine.org/index.php?title=Lua).
```lua
-- Just add this line at the top to clear output every time you execute the script
getLuaEngine().Menu.Items[0][5].doClick(); -- clear output
```
BTW Reclass.net is atrocious piece of garbage. Performance in large project such as this is terrible. Don't even try pressing "Enable hierarchy view".  If you want to know where the class is used, then you are in a hard spot (can't use hierarchy view; alternatively you can use "Generate C++ code"), so I would strongly suggest to always comment at the top of your new structure where it can be found (didn't do it personally for some structures, and I was in a lot of agonizing pain after that).

If you are going to decode memory structures, then you should learn to immediately recognise some common structures.
> Source: ["Auto-Padding" for class recreation (by Public_Void)](https://guidedhacking.com/threads/auto-padding-for-class-recreation.13478/#post-79838)
```h
#define STR_MERGE_IMPL(a, b) a##b
#define STR_MERGE(a, b) STR_MERGE_IMPL(a, b)
#define MAKE_PAD(size) STR_MERGE(_pad, __COUNTER__)[size]
#define DEFINE_MEMBER_N(type, name, offset) struct {unsigned char MAKE_PAD(offset); type name;}
```
### Strings
```h
// @string, @unicode
struct CA_String {
    union {
        DEFINE_MEMBER_N(int32, length, 0x00);
        DEFINE_MEMBER_N(int32, capacity, 0x04);
        DEFINE_MEMBER_N(char*, data, 0x08);
    }
}
```
### Arrays
```h
struct CA_Array<T> {
    union {
        DEFINE_MEMBER_N(int32, capacity, 0x00);
        DEFINE_MEMBER_N(int32, length, 0x04);
        DEFINE_MEMBER_N(T*, data, 0x08);
    }
}
```
### Linked list
This suckers have hash part. CA uses this all over da place.\
You really need to recognise these structures at first glance, otherwise you will waste too much time following pointers.\
`head->prev == nullptr`, `tail->next == CA_List<T>` (be careful, when traversing with scripts).\
If `actual_size == 0`, then `head == tail == CA_List<T>; capacity == size == 2`.\
Size of `hash` part is always power of 2.\
It is easy to recognise if you are inside of `hash` part. You will see continious array of pointers, and some of these pointers will be repeated (consequtively). And, of course, you would need to look whether it's items are of `CA_Member<T>` type.\
If you want to iterate over linked list, you should go from `tail` to the `head`. `head` always has `head->prev == nullptr`.\
If you are inside `CA_Member<T>` and want to find you where the fuck is `CA_List<T>`, follow `next`, it will bring you to `CA_List<T>`.
```c++
CA_Member<T> *cur, *next;
while (true) {
    next = cur->next;
	// first element has `CA_Member<T>.prev=0`
    if (next->prev == 0) { return (CA_List<T>*)(cur - 0x08); }
    cur = next;
}
```
```h
struct CA_Member<T> {
    union {
        DEFINE_MEMBER_N(CA_Member<T>*, prev, 0x00);
        DEFINE_MEMBER_N(CA_Member<T>*, next, 0x08);
        DEFINE_MEMBER_N(T, data, 0x10);
    }
}
// sizeof = 0x30 = 48
struct CA_List<T> {
    union {
        DEFINE_MEMBER_N(int32, actual_size, 0x00);
        DEFINE_MEMBER_N(CA_Member<T>*, prev, 0x08); // tail
        DEFINE_MEMBER_N(CA_Member<T>*, next, 0x10); // head
        DEFINE_MEMBER_N(int32, capacity, 0x18);
        DEFINE_MEMBER_N(int32, size, 0x1C);
        DEFINE_MEMBER_N(CA_Member<T>**, hash, 0x20);
        DEFINE_MEMBER_N(float, some_number, 0x28); // I've only seen value 1.0
    }
}
```

## Static pointers
### Database table list
*-- Only tested in frontend and campaign!*
```js
// Points to the very first db table
// They are not continious, some (DB_DATA) are overwritten with junk
// [[[<Warhammer2.exe>+03695798]+20]+10]
// [[[base + 03695740 + 50 + 08]+20]+10]
const b = require('@basic')
const uic_db = require('@static/uic_db')
let ptr = b.read_pointer(b.base, 0x03695740) // @static/uic_db.base
let uic_db_entry = b.read_instance(undefined, ptr, 0x00, uic_db.static_uic_db)
for (const entry of uic_db_entry.uic_list.data()) { // uic_list: 0x50; prev: 0x08
    // prev(0): idx = 0
    if (entry.uic_path === 'ui\\common ui\\tab_completer') {
        let ptr = entry.uic._pointer // @uic.UIC: 0x20
        ptr = b.read_pointer(ptr, 0x10) // @db.DB_TABLE
        break;
    }
}
```
#### LUA
```lua
-- UICreated @event
-- Could also find where this `tab_completer` hides among root children?
local ptr = mr.base
ptr = read_pointer(ptr, '\152\87\105\3') -- 0x03695798
ptr = read_pointer(ptr, 0x20)
ptr = read_pointer(ptr, 0x10)
```

### Current selected character (+hover)
After **Silence & The Fury** update, UIC data structures changed too much,
and I can't figure them out. So, only lua example.
#### LUA
```lua
-- Does it already exist in UICreated? Does it get recreated at some conditions?
local CA_cip = root:SequentialFind(
    'layout',
    'info_panel_holder',
    'primary_info_panel_holder',
    'info_panel_background',
    'CharacterInfoPopup')
local ptr = ud_topointer(CA_cip)
ptr = read_pointer(ptr, 0x18) -- @uic.UIC
ptr = read_pointer(ptr, 0x50) -- cco_selected
ptr = read_pointer(ptr, 0x08) -- @character.CharacterDetails
ptr = read_pointer(ptr, 0x00) -- @character.Character
local cqi = read_int32(ptr, 0xF0)
```

## Some useful info
### effect_bonus_value
`ebv` is defined as:
```
0x00: uniq_key[ebv_table_row(20 bits), 0(20 bits), bonus_value_id(16 bits), ebv_table_id(8 bits)]
0x08: bonus_value_id(16 bits)
0x0A: ebv_table_id(8 bits)
0x0B: ???(40 bits)
0x10: entry pointer // (union) treated dynamically based on ebv_table_id
// In most cases it is the same value as parent entity `value`.
// (ex: character_trait_level__effects has `value` field, and `ebv.value == value`)
// But I've seen one case where they are not the same. Not sure why
0x18: value(float)
0x1C: value(float)
```
`ebv_table_id` value is mapped like this:
```
[0x1D] = 'ebv_agent_action_record'
[0x00] = 'ebv_agent'
[0x20] = 'ebv_agent_subtype'
[0x1E] = 'ebv_attrition_record'
[0x01] = 'ebv_basic'
[0x17] = 'ebv_battle_context_army_special_ability'
[0x14] = 'ebv_battle_context'
[0x15] = 'ebv_battle_context_unit_ability' (doesn't work in effect_bundles)
[0x16] = 'ebv_battle_context_unit_attribute'
[0x18] = 'ebv_battlefield_deployables' (doesn't work)
[0x02] = 'ebv_building_chain' (doesn't work)
[0x19] = 'ebv_building_set'
[0x09] = 'ebv_faction'
[0x12] = 'ebv_id_action_results_additional_outcomes'
[0x1A] = 'ebv_ids_units_sets'
[0x29] = 'ebv_loyalty_event'
[0x2C] = 'ebv_missile_weapon'
[0x1F] = 'ebv_name_record'
[0x23] = 'ebv_military_force_ability'
[0x2A] = 'ebv_pooled_resource_factor'
[0x2B] = 'ebv_pooled_resource'
[0x03] = 'ebv_population_class' (doesn't exist)
[????] = 'ebv_population_class_and_religion' (doesn't load)
[0x05] = 'ebv_projectile' (works)
[0x06] = 'ebv_projectile_shot_type_enum' (doesn't work)
[0x13] = 'ebv_provincial_initiative_effect_record'
[0x07] = 'ebv_religion'
[0x0A] = 'ebv_resource'
[0x2D] = 'ebv_ritual_category'
[0x25] = 'ebv_ritual_chains' (works)
[0x24] = 'ebv_ritual'
[0x10] = 'ebv_siege_item'
[0x22] = 'ebv_special_ability_phase_record'
[0x08] = 'ebv_subculture'
[0x11] = 'ebv_technology_category' (works)
[0x1B] = 'ebv_technology' (doesn't work)
[0x0B] = 'ebv_unit_abilitiy'
[0x21] = 'ebv_unit_attribute'
[0x0E] = 'ebv_unit_caste' (works)
[????] = 'ebv_unit_caste_stat_modifiers' (doesn't load)
[0x0D] = 'ebv_unit_category' (works)
[????] = 'ebv_unit_category_stat_modifiers' (doesn't load)
[0x0C] = 'ebv_unit_class' (works)
[????] = 'ebv_unit_class_stat_modifiers' (doesn't load)
[0x0F] = 'ebv_unit_record'
[0x26] = 'ebv_unit_set_special_ability_phase'
[0x28] = 'ebv_unit_set_unit_ability'
[0x27] = 'ebv_unit_set_unit_attribute'
```


text:
1 Introduction
==============

This [OGC API Coverages](https://github.com/opengeospatial/ogc_api_coverages) specification establishes how to access coverages as defined by the [Coverage Implementation Schema (CIS) 1.1](http://docs.opengeospatial.org/is/09-146r6/09-146r6.html) through [OpenAPI](https://www.openapis.org/).

Current scope:

* Only gridded coverages are addressed, not MultiPoint/Curve/Surface/SolidCoverages. Reason is that gridded coverages receive most attention today.
* Only GeneralGridCoverage is addressed, other coverage types will follow later. Reason is to have a first version early which shows and allows to evaluate the principles.
* Only coverage extraction functionality is considered, not general processing (as is provided with Web Coverage Service (WCS) extensions such as the Processing Extension). Exceptions from this rule are subsetting including band subsetting, scaling, and CRS conversion and data format encoding, given their practical relevance.
* Subsetting is considered in the query component only for now. As typically all dimensions in a coverage are of same importance subsetting might not fit perfectly in the hierarchical nature of the path component. Further, subsetting may reference any axis and leave out any other, which makes positional parameters unsuitable. Nevertheless subsetting in the path component particularly limited to fixed subsets might be considered in a future version.

As such, the functionality provided below resembles [OGC Web Coverage Service (WCS) 2.1 Interface Standard - Core](http://docs.opengeospatial.org/is/17-089r1/17-089r1.html).

2 Principles
============

[OpenAPI](https://www.openapis.org/) establishes URL-based access patterns, as defined by [RFC 3986 "Uniform Resource Identifier (URI): Generic Syntax"](https://tools.ietf.org/html/rfc3986), [RFC 3987 "Internationalized Resource Identifiers (IRIs)"](https://tools.ietf.org/html/rfc3987), and [RFC 6570 "URI Template"](https://tools.ietf.org/html/rfc6570) following a syntax like
        http://www.acme.com/path/to/resource/{id}{?query,parameters}
whereby

* `/path/to/resource/{id}` defines the local path to the resource to be retrieved on the node identified by the head part, http://www.acme.com.
* the `{?query,parameters}` represent key/value pairs with further parameters passed to the request.

As a general rule,

* accessing a coverage component is done in the path section,
* subsetting is done in the query parameter section
* format encoding is controlled via HTTP headers

As the coverage model normatively is given by the corresponding XML schema (JSON and RDF are built in sync with XML) specification of the [OpenAPI](https://www.openapis.org/) access paths follows this schema. Note, though, that OWS Common, while normatively
referenced in CIS, is not followed by [OpenAPI](https://www.openapis.org/), so here deviations will occur.

For path expressions abbreviations (i.e., aliases) may be defined for convenience.

3 [OGC API Coverages](https://github.com/opengeospatial/ogc_api_coverages) Requests
===================================================================================

This clause defines the basics of accessing OGC coverages via OpenAPI.

3.1 General
-----------

Requirement:
A service offering [OGC API Coverages](https://github.com/opengeospatial/ogc_api_coverages) access shall be accessible via an URL as service endpoint (subsequently referred to as Base URL).

Example:
        http://acme.com/oapi

Requirement:
Coverage access paths, as defined in the next clause, are concatenated to the Base URL via a "/" character, followed (optionally) by a query part.

Example:
        http://acme.com/oapi/collections/{collectionid}/coverages/{coverageid}/rangeset

A request may be denied for server-specific reasons, such as quota.

3.2 Coverage Access Paths
-------------------------

In this clause, the path component of [OGC API Coverages](https://github.com/opengeospatial/ogc_api_coverages) access is established.

Access paths follow the XML Schema of [CIS](http://docs.opengeospatial.org/is/09-146r6/09-146r6.html) in their structure.

To access coverage service constituents, such as formats supported, OGC 14-121 Web Query Service provides guidance, see https://github.com/opengeospatial/ogc_api_coverages/blob/master/CIS%2BWCS-standards/14-121_Web-Query-Service_2016-06-19.pdf .

3.3 Coverage Query Parameters
-----------------------------

In this clause, the query component of [OGC API Coverages](https://github.com/opengeospatial/ogc_api_coverages) access is established.

Subsetting parameters may be mixed in a query part, in no particular order.

3.3.1 Coverage Subsetting
-------------------------

Without any subsetting parameter the whole coverage extent is the target resource addressed. If a subsetting operation is provided then the coverage subset indicated is the target resource addressed.

Coverage subsetting is indicated through the SUBSET parameter name. The value following the "=" symbol is built as follows:

    SubsetSpec:            SUBSET = axisName ( intervalOrPoint )
    axisName:              {NCName}
    intervalOrPoint:       interval | point
    interval:              low :  high
    low:                   point | *
    high:                  point | *
    point:                 {number} | " {text} "    //" = double quote = ASCII code 0x42

where {NCName} is an XML-style identifier not containing ":" (colon) characters, {number} is an integer or floating-point number, and {text} is some general ASCII text (such as a time and date notation in ISO 8601). A coverage access request may have any number of SUBSET parameters, however for each axis of the coverage at most one SUBSET parameter may be provided.

In case of an interval a trim operation is specified, with lower and upper bound. In case of a point a slicing operation is specified. For the detailed semantics of subsetting, trimming, and slicing see [OGC WCS](http://docs.opengeospatial.org/is/17-089r1/17-089r1.html).

3.4 Coverage Encoding
-----------------------

Without any format encoding HTTP header (Accept and Accept-Encoding) a representation of the coverage in its Native Format will be returned. If an encoding HTTP header is provided then a representation of the coverage in the format indicated will be returned.
The syntax for format encoding HTTP headers is defined in [RFC 7231 "Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content"](https://tools.ietf.org/html/rfc7231).

The format chosen must be able to represent the output of the request.

For the detailed semantics of format encoding see [OGC WCS](http://docs.opengeospatial.org/is/17-089r1/17-089r1.html).

4 Examples
==========

This section contains examples for coverage access, subsetting, and encoding. It assumes a service endpoint of http://acme.com/oapi/ .

4.1 Service Metadata
--------------------

The first part is about service metadata. Currently this is wild speculation as it will mainly be defined through [OGC API Common](https://github.com/opengeospatial/oapi_common).

* http://acme.com/oapi/servicedescription/formatssupported  -- returns a list of formats in which coverage representations can be requested

4.2 Coverage Finding
--------------------

* http://acme.com/oapi/collections  -- returns a list including links to all collections
* http://acme.com/oapi/collections/{collectionid}  -- returns the description of a specific collection
* http://acme.com/oapi/collections/{collectionid}/coverages  --  returns a list of all coverages included in a specific collection
* http://acme.com/oapi/collections/{collectionid}/coverages/{coverageid}  --  returns a specific coverage as multipart coverage including DomainSet, RangeType, and RangeSet in the coverage's Native Format
* http://acme.com/oapi/collections/{collectionid}/coverages?bbox=160.6,-55.95,-170,-25.89  -- returns a list of all coverages intersecting in a specific collection that is in the New Zealand economic zone

4.3 Coverage Access
-------------------

The second part is about coverage access, which (as described earlier) is driven by the coverage structure and, hence, given:

* http://acme.com/oapi/collections/{collectionid}/coverages/{coverageid}/domainset  -- returns the coverage's domain set definition
* http://acme.com/oapi/collections/{collectionid}/coverages/{coverageid}/domainset/generalgrid/srsname  -- returns the srsName value, i.e.: the coverage's (spatio-temporal) CRS
* http://acme.com/oapi/collections/{collectionid}/coverages/{coverageid}/domainset/generalgrid/axislabels  -- returns a list o the coverage's axis names
* http://acme.com/oapi/collections/{collectionid}/coverages/{coverageid}/rangetype  -- returns the coverage's range type information (i.e., a description of the data semantics)
* http://acme.com/oapi/collections/{collectionid}/coverages/{coverageid}/metadata  -- returns the coverage's metadata (may be empty)
* http://acme.com/oapi/collections/{collectionid}/coverages/{coverageid}/rangeset  -- returns the coverage's range set, i.e., the actual values in the coverage's Native Format

4.4 Coverage Subsetting
-----------------------

The third part is about query parameters:

* http://acme.com/oapi/collections/{collectionid}/coverages/{coverageid}?SUBSET=Lat(40,50)&SUBSET=Long(10,20)  -- returns a coverage cutout between (40,10) and (50,20), as multipart coverage
* http://acme.com/oapi/collections/{collectionid}/coverages/{coverageid}/rangeset?SUBSET=Lat(40,50)&SUBSET=Long(10,20)  -- returns a coverage cutout between (40,10) and (50,20), in the coverage's Native Format
* http://acme.com/oapi/collections/{collectionid}/coverages/{coverageid}?SUBSET=time("2019-03-27")  -- returns a coverage slice at the timestamp given (in case the coverage is Lat/Long/time the result will be a 2D image)

5 Open Issues
=============

* Establish service parameter access, based on [OGC API Common](https://github.com/opengeospatial/oapi_common)
* What is the output format of items typically returned as XML or JSON, such as DomainSet and RangeType? Should maybe FORMAT be applicable here as well? If so, should it be listed as a possible output format (which might be confusing)?
* [OGC 14-121 Web Query Service](https://github.com/opengeospatial/ogc_api_coverages/blob/master/CIS%2BWCS-standards/14-121_Web-Query-Service_2016-06-19.pdf) provides a definition of path syntax, but adds more functionality (such as selection predicates), all based on the XPath standard. Such extra functionality might come handy.


text:
# element-web-azure-pulumi
Project to provision a vector-im / element-web Server with Pulumi on Azure

Creating a private https://github.com/vector-im/element-web on Azure with Pulumi.

### Docker Image

https://hub.docker.com/r/vectorim/riot-web/ start it with:

```shell
docker run -p 80:80 vectorim/element-web
```

### How to run on Azure

Let's search in https://docs.microsoft.com/en-us/azure/architecture/aws-professional/services#web-applications

--> Try Container Instances: https://azure.microsoft.com/en-us/services/container-instances/

### Azure Container Instances with Pulumi

https://www.pulumi.com/docs/tutorials/azure/container-webserver/

Create a new Pulumi Skelleton:

```shell
pulumi new azure-typescript
```

Let's choose `public` and  `westeurope` which runs in Berlin: https://azure.microsoft.com/de-de/global-infrastructure/geographies/#geographies

```shell
$ pulumi new azure-typescript
This command will walk you through creating a new Pulumi project.

Enter a value or leave blank to accept the (default), and press <ENTER>.
Press ^C at any time to quit.

project name: (testserver) element-web-azure-pulumi
project description: (A minimal Azure TypeScript Pulumi program) Project to provision a vector-im / element-web Server with Pulumi on Azure
Created project 'element-web-azure-pulumi'

Please enter your desired stack name.
To create a stack in an organization, use the format <org-name>/<stack-name> (e.g. `acmecorp/dev`).
stack name: (dev)
Created stack 'dev'

azure:environment: The Azure environment to use (`public`, `usgovernment`, `german`, `china`): (public)
azure:location: The Azure location to use: (WestUS) westeurope
Saved config

Installing dependencies...

added 184 packages, and audited 185 packages in 17s

20 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
Finished installing dependencies

Your new project is ready to go! ✨

To perform an initial deployment, run 'pulumi up'
```

repository: jonashackt/element-web-azure-pulumi

repo_name: jonashackt/element-web-azure-pulumi


text:
# Timing Attack

## The Basics

Timing attacks are possible when an application takes different amounts of time, responding to different scenarios.

For context, a common case is a login operation. Imagine a system that searches in the database for the supplied username, then when found hashes the password supplied and compares it with the hash from the database, checks if the user is locked, if needs to reset password etc. All of these operations will increase the response time, compared to a request where the username doesn't exist in the system.

This is where a timing attack can get into place, to find user accounts available in the system.

In a stable system even the string comparing of the hashes can be susceptible to a timing attack. To prevent it a secure string comparison should be implemented.

## Solving the challenge

To solve this challenge we created a tool called [BeOnTime](https://github.com/DamnVulnerableCryptoApp/BeOnTime) that helps performing Timing Attacks

THe idea is to brute force the usernames, and validate if some usernames take more time to process then others. This may reveal existing users in the application.

Capture a request made by the web application (with Burp for example) to use it with the tool.
BeOnTime requires you to add the placeholder '{{PLACEHOLDER}}' where you want to try multiple values, in this case, the username.

This is an example of the content of the file (lets cal it req.txt):

```plaintext
POST /timing-attack/login HTTP/1.1
Host: localhost:4000
Content-Length: 35
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36
content-type: application/json
Accept: */*
Origin: http://localhost:3000
Sec-Fetch-Site: same-site
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: empty
Referer: http://localhost:3000/
Accept-Encoding: gzip, deflate
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
Connection: close

{"username":"{{PLACEHOLDER}}","password":"asd"}
```

Then you need a username wordlist. You can use one for example one available in the awesome [SecList](https://github.com/danielmiessler/SecLists/blob/master/Usernames/cirt-default-usernames.txt) collection.

With this, running the BeOnTime is quite simple:

```bash
BeOnTime -r req.txt -w dict.txt
```

## Lesson Learned

It's hard to have in mind all possible scenarios where a timing attack can be performed, but be extra careful when dealing with sensitive/personal information and make sure you the requests take in average the same time, with different scenarios.

A common exploit vector is in string comparison, so make sure you use secure methods to do the comparison. When checking characters make sure you go through all, even if already identified that is a different string.

repository: fjsnogueira/DamnVulnerableCryptoApp

repo_name: fjsnogueira/DamnVulnerableCryptoApp


text:
### Requirements
- A Git repository
- A Dotbot configuration file in that repository
- Python 3

### Instructions
Save `pre-commit` and `prepare-commit-msg` to `.git/hooks` in your dotfiles directory. Alternatively, use a Git hook manager like [git-hooks](https://github.com/git-hooks/git-hooks).

Then when a file is added to the index and committed, your Dotbot configuration file will be updated with a new "link" entry:
```
link:
    ~/.filename: filename
```

If your Python executable has a name other than Python (e.g. `python3` or `python3.5`) you could use a wrapper script with the correct executable:
```
#! /usr/bin/env sh
python3.5 pre-commit "$@"
```

### Options
Options are configurable through environment variables:

- `AUTOBOT_DISABLED`: If this is non-empty, these hooks do nothing (default is empty).
- `AUTOBOT_REPO_ROOT`: Set the root directory for the repository (default is `.`).
- `AUTOBOT_DOTBOT_CONF`: Set an alternative name for your Dotbot config file (default is `dotbot.conf.yaml`).
- `AUTOBOT_INCLUDE`: Colon-separated list of files or directories files to exclude (default is the value of `$AUTOBOT_REPO_ROOT`). Basic wildcards (`*`, `**`, `?`, `[abc]`, and `[!abc]`) are allowed. Paths are first expanded using `glob.glob()`, and then matched using `Path.match()`, both in the Python standard library.
- `AUTOBOT_EXCLUDE`: Same notes as above. The exclusion check is applied _after_ the inclusion check.
- `AUTOBOT_DELETE_BACKUP`: If this is non-empty, delete the backup file when finished (not recommended).
- `AUTOBOT_DEBUG`: If this is non-empty, activate extremely verbose debug-level logging.

### Missing features

- Wildcard include/exclude
- Gitignore parsing

repository: nagromc/dotbot-autobot

repo_name: nagromc/dotbot-autobot


text:
---
layout: post
title: Understanding EmberJS and JSONApi 2.0
date: '2015-08-10T21:29:25+01:00'
tags: []
---

### Summary

After working too many hours on a solution that was just wrong I finally figured out how to work with the serializers in Ember 1.1.13+.

The below should work for Ember 2.0 as well.

### Solution

What I do below is let Ember do the magic of converting the payload from the old format to the new,
I just use the new hooks defined in the blog-post EMBER DATA V1.13 RELEASED.

This should help you if you want to:

* Change the payload and do some magic before hitting the store/model
* Generally modify data returned from all endpoints in your API