text:
---
layout: ontology_detail
id: amphx
title: The Amphioxus Development and Anatomy Ontology
build:
  checkout: git clone https://github.com/EBISPOT/amphx_ontology.git
  path: .
  system: git
contact:
  email<EMAIL_ADDRESS>  github: hescriva
  label: Hector Escriva
  orcid: 0000-0001-7577-5028
dependencies:
- id: uberon
description: An ontology for the development and anatomy of Amphioxus (Branchiostoma lanceolatum).
domain: anatomy and development
homepage: https://github.com/EBISPOT/amphx_ontology
license:
  label: CC BY 3.0
  url: http://creativecommons.org/licenses/by/3.0/
preferredPrefix: AMPHX
products:
- id: amphx.owl
- id: amphx.obo
repository: https://github.com/EBISPOT/amphx_ontology
tracker: https://github.com/EBISPOT/amphx_ontology/issues
activity_status: active
---

The Amphioxus Development and Anatomy Ontology (AMPHX) is to describe the anatomy and development of Amphioxus, also known as lancelet, member of the invertebrate subphylum Cephalochordata and the phylum Chordata. This ontology is intended to be used for description of gene expression in amphioxus (e.g. Insitus, RNA-seq). The ontology was created in the context of the European project [CORBEL](https://www.corbel-project.eu/home.html), and used in the database [MARIMBA](http://marimba.obs-vlfr.fr/home).

gha_language: Python

gha_license_id: NOASSERTION


text:
# purj

Purge and Purify JS: This library extends standard JavaScript classes and adds some helper functionality - all in pure, modern, lightweight JavaScript.

Trying to mimic some of the commonly-used (basic) jQuery functionality for iterating through elements, adding event listeners, etc. without all the extra fluff that comes along with it.

jQuery is close to 90kb minified, and even the slim version is still around 72kb. Gzipped or not, that's a lot of blocking in a time where blocking time is precious (ahem, Lighthouse).

purj will be providing only minimal helpers to achieve some of the commonly-used logic from jQuery. It will not (nearly) meet all of them. No AJAX or animations. No CSS helpers. No daisy chaining. No selector parsing. Just simple JavaScript.

purj is only a few days old as of this writing. I wrote what's currently here to meet a requirement of my day job - trying to remove as much superfluous JavaScript as possible.

## Installation

You've got a couple options...

### External Script

You can source the dist script directly in your page:
```html
<script src="purj.min.js"></script>
```

### Yarn/NPM

Or, use yarn or npm to add the library to your project:
```js
yarn add purj --dev

// or

npm i purj --save-dev
```

Then include it wherever you need it:
```js
import purj from 'purj';
```

## Features

As purj is still very young, there are only a few features right now...

* `Array.apply(callback)` - Run callback on every item in array. The callback is passed the iteration index, and `this` is the item.
  ```js
  var arr = ['one', 'two', 'three'];
  arr.apply(function(i) {
    console.log(i + '. ' + this);
  });
  // 0. one
  // 1. two
  // 2. three
  ```
* `Array.first()` - Retrieve first item from array.
* `Array.last()` - Retrieve last item from array.

* `Document.find(selector)` - Shortcut for document.querySelectorAll(). Find all elements in document matching selector.
* `Document.pluck(selector)` - Shortcut for document.querySelector(). Find first element in document matching selector.
* `Document.on(event, selector, callback)` - Add event listener to DOM for dynamic elements. Mimics jQuery's `$(document).on()` in this way.
  For example:
  ```js
  document.on('click', 'a.logo', function(e) {
    e.preventDefault();
    this.classList.add('clicked');
  });
  ```
* `Document.ready(callback)` - Fire callback when document is loaded & ready.

* `Element.find(selector)` - Find all children matching element.
* `Element.pluck(selector)` - Find first child matching selector.
* `Element.findParent(selector)` - Find first parent element matching selector.
* `Element.getData(?name)` - Get attribute data-name from element. If name is omitted, return hash of all data attributes on element.
* `Element.setData(data, ?val)` - Set data data-data attribute on element. If data is an object, val is ignored and every key in data is added as a data attribute to element. Deeply nested objects will work too.
* `Element.on(event, callback, ?options)` - Shortcut for element.addEventListener().

* `NodeList.apply(callback)` - Run callback on every element in list. Same details as `Array.apply`.
* `NodeList.first()` - Retrieve first element from list.
* `NodeList.last()` - Retrieve last element from list.
* `NodeList.on(event, callback, ?options)` - Shortcut for adding an event listener to each element in list.
  ```js
  document.find('a').on('click', function(e) {
    console.log(this, e);
  });
  ```

* `Object.apply(callback)` - Run callback on every item in object. The callback is passed the key, and `this` is the item.

* `String.camelize()` - Convert `this-dashed-string` to `thisDashedString`.

license: MIT

repository: dovestyle/purj

branch_name: refs/heads/master

extension: md

license_type: permissive

repo_name: dovestyle/purj


text:
# Wifi Master

try this for hc-06 config: [here](https://mcuoneclipse.com/2013/06/19/using-the-hc-06-bluetooth-module/).

potentially use both put requests and web sockets. put requests for the fast data and web sockets for the 2-way communication.

Master code for Arduino communicating with Hololens via Bluetooth. Deployed on ESP32: [this](http://a.co/d/21BAwOl) exact board. Utilized led and oled screen included on the board. SD card breakout board used for getting static files (asyncronously). HC-05 board used because platformio does not allow for setting enabling concurrent wifi and bluetooth connections on ESP-32 board. Software serial avoided because it caused too many errors when sending data from Hololens or other connected device to the board. Hardware Serial used instead.

See [this](http://docs.platformio.org/en/latest/platforms/espressif32.html#partition-tables), [this](https://docs.espressif.com/projects/esp-idf/en/latest/api-guides/partition-tables.html), and [this](https://desire.giesecke.tk/index.php/2018/01/30/change-partition-size/) for partition changing for bluetooth.

See [this](https://github.com/esp8266/Arduino/issues/570) for increasing max connections to access point. Ended up changing config settings and it worked anyway (up to 10).

## Trying ESP32-IDF Compiler

### Reasoning

Now I have another problem! There's a setting for enabling bluetooth and wifi at the same time, which doesn't exist for platformio. Thankfully others have had this problem, see [here](https://community.platformio.org/t/menuconfig-options-for-esp32-arduino/3871). I'm going to try [this](https://hackaday.io/project/43374-esp32-idf-deploying-the-development-platform/details) solution and see how it goes, tomorrow. But right now if both bluetooth and wifi are on it doesn't work...

### What I learned from trying this

TLDR: Didn't work. Use PlatformIO if you have a bunch of dependencies and don't try bluetooth and wifi at the same time.

Follow the steps below to recreate the configuration I had. I was trying to use ESP32 IDF compiler so that I could use the menuconfig. Unfortunately using ESP32-IDF is a PAIN and not worth it. Using another bluetooth module indead. The whole point was to get make menuconfig working so that I could toggle the Software Wifi/BT Coexistence option so that wifi and bluetooth could happen at the same time. But this method just doesn't work well. There were too many dependency issues, and integrating with PlatformIO didn't work, and in the end I was left where I started with nothing working correctly.

Menuconfig options for ESP32 Arduino - General Discussion - PlatformIO Community
https://community.platformio.org/t/menuconfig-options-for-esp32-arduino/3871/2

Wi-Fi/BLE Contention : esp32
https://www.reddit.com/r/esp32/comments/9bq22e/wifible_contention/

esp32-devenv-vscode/tutorial.md at master · VirgiliaBeatrice/esp32-devenv-vscode
https://github.com/VirgiliaBeatrice/esp32-devenv-vscode/blob/master/tutorial.md

ESP32-IDF - Deploying the development platform | Details | Hackaday.io
https://hackaday.io/project/43374-esp32-idf-deploying-the-development-platform/details

Remove these directories from the Arduino Library
* WiFiClientSecure
* HTTPClient
* HTTPUpdate
* ESP32/examples/Camera

`git submodule update --init --recursive`
`git clone --single-branch --branch idf-update https://github.com/me-no-dev/AsyncTCP`
`git clone https://github.com/me-no-dev/ESPAsyncWebServer`
`git clone https://github.com/bblanchon/ArduinoJson`
`git clone https://github.com/olikraus/u8g2`
`git clone https://github.com/espressif/arduino-esp32`

repository: iamsile/SpaceSuits

repo_name: iamsile/SpaceSuits


text:
@def title = "Approximate Bayesian Inference - Introduction"

# Introduction to Approximate Bayesian Inference

In *Bayesian Machine Learning* (BML), one is broadly interested in recovering a posterior distribution over the parameters $x$ (with typically $x\in\mathcal X\subseteq \R^d$) of some parametric model of interest for which we have a likelihood $p(\mathcal D|x)$ (where $\mathcal D$ denotes the data) using Bayes' trick:

$$ p(x|\mathcal D) \spe{\propto} p(\mathcal D|x)p(\theta). \label{bayes-ml-1}$$

In equation \eqref{bayes-ml-1}, $p(x)$ denotes our *prior* belief on what the parameters should be and $p(x|\mathcal D)$ is the *posterior distribution* over $x$.
The posterior is then used to make more robust predictions with the model and, in particular, to account for **uncertainty** in the model parameters.

This approach is optimal under two key assumptions:
1. the prior is chosen appropriately,
1. the posterior is tractable (or at least we can sample from it).

The first assumption is usually overlooked by researchers in computational Bayesian statistics who usually assume the prior is given (and, in practice, it typically encodes a regularisation).
This is not without its weaknesses but it would require an entire set of notes to be discussed appropriately. \br
The second assumption is generally far from true especially with complex models and many approaches have been suggested such as MCMC methods; while these can do very well when $\theta$ is low-dimensional, they typically struggle when the dimensionality is high.

In these notes we define *Approximate Bayesian Inference* (ABI) in broad terms as the class of methods attempting to recover a distribution $q$ such that $q$ is a good *proxy* for the posterior distribution of interest $p$.
A number of such methods are commonly used when the dimensionality of the problem makes MCMC-like methods computationally too expensive.

## Variational problem

In what follows we simplify the context a little bit by considering methods which find good proxy $q$ to some target distribution $p$.
We write $\mathcal F\mathcal \subset \mathcal P(\mathcal X)$ the space of candidate proxy distributions where $\mathcal P(\mathcal X)$ designates all probability distributions on $\mathcal X$.

Let $D:\mathcal P(\mathcal X)\times \mathcal P(\mathcal X) \mapsto \R_+$ denote a discrepancy measure between two probability distribution functions.
Then, the generic variational problem considered in ABI is the minimisation of this $D$ between the candidate $q\in \mathcal F$ and the target $p$:

$$ q^\star \spe{\in} \arg\min_{q\in\mathcal F} D(q, p). \label{eq:basic-abi}$$

Techniques attempting to solve such problems rely upon exploiting at least one of the following three main characteristics:

1. the definition of the discrepancy measure $D$ (aka. divergence),
1. the definition of the restricted set of probability distributions $\mathcal F$,
1. the structure of the target distribution $p$.

Many discrepancy measures can be considered such as the total variation distance, Wasserstein distances or $f$-divergences.
However, most of these choices can lead to the corresponding problem \eqref{eq:basic-abi} being computationally (very) expensive to solve in general, especially if the space $\mathcal X$ is continuous.

In these notes we will focus on the Kullback-Leibler (KL) divergence which, as we'll see, can lead to optimisation problems for which efficient computational methods exist.
As a reminder, the KL divergence between two distributions $p$ and $q$ is defined as

$$ \KL(p, q) \speq \E_p[\log p(X) - \log q(X)]. $$

We will look at both $\KL(p, q)$ and $\KL(q, p)$ and, while in both case \eqref{eq:basic-abi} is intractable, in both cases proxy problems can be obtained and lead to two well known algorithms (along with related variants):

* the (Mean-Field) Variational Bayes algorithm ((MF)VB), which we will discuss [here](\abi{vb.html}),
* the Expectation Propagation algorithm (EP), which we will discuss [here](\abi{ep.html}).

## A warning note

At the risk of stating the obvious, it must stressed that ABI algorithms do in general *not lead to the correct posterior* but rather to a proxy which may be a very poor approximation to the target posterior.
Therefore, no matter what ABI algorithm you use, you should in general be very careful about how you interpret the recovered proxy $q$ and should ideally refrain from making strong statements about the uncertainty in the model parameters based upon $q$ unless you've tested it somehow.

While there are some cases where ABI algorithms can give good proxy (in particular if $p$ is known to be log-concave and thin-tailed), in many circumstances, and especially in the context of complex/high-dimensional models such as neural networks, ABI algorithms lead to proxy distributions which may

* typically recover reasonable location (e.g. the most likely $\theta$ according to the proxy $q$ is a good parameter for the model in terms of generalisation error)
* typically recover appalling uncertainty (often catastrophically underestimated).
ABI is a nice theoretical field with interesting mathematical tools but that should not prevent you from checking that the results obtained are sensible.
If you are not in a position to check the validity of the obtained proxy, you should probably revert to simply considering the maximum a posteriori (MAP) and avoid fooling yourself into thinking that you have a workable model of the uncertainty.


text:
Crystal Melting
===============

Molecular liquids crystallise seven orders of magnitude slower than metals[^1], with no real
understanding of this difference. This is a project to understand the slow crystallisation and
melting behaviour of molecular crystals, using an asymmetric two dimensional particle as a simple
model of a molecular system.

Dependencies
------------

All the dependencies for the project are managed using conda. Once conda is installed ([link to
guide][conda installation]), all dependencies can be installed with the command

```shell
conda env update
```

This creates an *environment*---a self contained collection of software which only has local scope;
that is the software is accessible only when the environment is active and software outside the
environment is unaffected. This environment can be activated by running

```shell
conda activate crystal
```

Data
----

### Subset of Data

A subset of the dataset is available for download from [Zenodo][crystal melting dataset]. This dataset demonstrates the melting
behaviour of the crystals studied in this project across the range of crystal structures and from
temperatures slightly below to slightly above the melting point.

### Raw Data

The entire dataset associated with this project is too large to store in any version control
repository, currently standing at many terabytes. The raw datafiles can be regenerated by re-running
the simulations in the `data/simulations` directory. Using the melting simulation as an example this
can be done with

```shell
experi -f data/simulations/melting/experiment.yml
```

It is recommended to run this command on a HPC which is running a PBS scheduler like Torque or
PBSPro, identifiable by requiring `qsub` command to submit jobs. This command will submit a series
of batch jobs containing all the combinations of variables.

### Processed Data

I don't currently have a good method of making the processed data available. It's dynamic nature
makes publishing at this point in time not a particularly useful option.

[^1]: Orava, J., & Greer, A. L. (2014). Fast and slow crystal growth kinetics in glass-forming melts. Journal of Chemical Physics, 140(21), 214504. https://doi.org/10.1063/1.4880959

[conda installation]: https://conda.io/docs/user-guide/install/index.html
[crystal melting dataset]: http://doi.org/10.5281/zenodo.1315097

gha_license_id: MIT


text:
# Bat Diet via DNA Metabarcoding -- Illumina MiSeq and Ion Torrent PGM
Timothy J Divoll
27 February, 2017

#Install and Setup

####This tutorial provides guidance on how to transform raw Ion Torrent **or** Illumina MiSeq amplicon data into OTUs for taxonomic assignment. The workflow includes UNIX commands to call QIIME (http://qiime.org/) and FASTX Toolkit (http://hannonlab.cshl.edu/fastx_toolkit/download.html) scripts to quality filter and cluster sequences before using Python scripts and R packages to filter and clean data for taxonomic assignment.

####We recommend first running through the entire process outlined in this document with the provided test data sets to make sure all software is installed properly under reduced processing loads (i.e., only a couple samples). Materials are available here: https://github.com/tdivoll/Bat-Diet-Metabarcoding

####The first section must be performed in UNIX (http://opengroup.org/unix), Macqiime (http://www.wernerlab.org/software/macqiime), or through a Virtual Machine running UNIX, as outlined here: (http://qiime.org/install/virtual_box.html). We used the recommended QIIME Virtual Box install on a Windows machine with 16GB RAM and Oracle Virtual Box Version 5.0.8 r103449 (https://www.virtualbox.org/wiki/Downloads).

####In the QIIME VB System Settings, we changed default settings to allow 4 cores and 11 GB of virtual memory.

###*Virtual Box Settings*
![](D:\\Google Drive\\IONvsMISEQ\\BatDietWorkflow_files\\VBMain.jpg)

###*Virtual Box Processor Cores*
![](D:\\Google Drive\\IONvsMISEQ\\BatDietWorkflow_files\\VBCores.jpg)

###*Virtual Box Memory*
![](D:\\Google Drive\\IONvsMISEQ\\BatDietWorkflow_files\\VBMem.jpg)

#Illumina MiSeq Workflow -- ___Run commands in UNIX___

#### If you only have IonTorrent data, skip the [Illumina MiSeq] section and go to the [Ion Torrent](#Ion Torrent) section.

####All commands in this section were run inside the QIIME Virtual Box created above in the [Install](#Install) section. Open the QIIME machine and start a new Terminal window. It should provide a command line prompt similar to this: qiime@qiime-190-virtual-box:

####If you will be using a similar workflow more than once, it helps to use the **`time`** UNIX command in front of other commands. This command returns the processing time, in minutes and seconds, which simply helps plan future data analyses. For example:
![](D:\\Google Drive\\IONvsMISEQ\\BatDietWorkflow_files\\timeEx.jpg)

### **1)** Organize R1 and R2 Files
####First load all compressed R1 and R2 *.fastq.gz* paired files into a new folder, such as: **`2014MISEQ`**

####Drag and drop from Windows to Virtual Box or use **`mv`** if working directly in UNIX or Macqiime

```bash
mkdir 2014MISEQ
```

#### QIIME will not accept underscores or hyphens in Sample IDs used in the QIIME process, thus, you may have to rename folder names so there are no underscores before L001; change underscores and hyphens to periods, except after L001. Everything before L001 will become the Sample ID. An example of bash code to do this follows:

```bash
#!/bin/bash   #save this as a script called fixfilenames.sh and save in your Project folder
cd $1;
for file in *L001*; do
  fp = "${file%L001*}";
  lp = "${file#*L001}";
  new = "${fp//_/.}L001$lp";
  mv "$file" "$new";
done
```
#### Now change permissions and run the script.

```bash
cd 2014MISEQ

chmod u+x fixfilenames.sh

./fixfilenames.sh
```

### **2)** Join Paired Ends
####Next, we join corresponding paired reads **_(R1 and R2)_** for each sample to increase read quality and propagate that quality on through to taxonomic assignment. This command will accept zipped files so there is no need to extract first.

```bash
multiple_join_paired_ends.py -i /home/qiime/2014MISEQ -o /home/qiime/2014MISEQ/joined_seqs
#took 10 secs with test set
```

#### We output the joined sequence files, 1 per bat sample, into a new directory called: **`joined_seqs`**

####Now we take out the **unjoined sequences** so they do not get mixed up in downstream analyses. You may prefer to save these somewhere if they could be useful to your particular application.

```bash
cd joined_seqs     #change directory to the newly created joined_seqs directory

find . -name "*.un1.fastq" -type f -delete     #do this exactly as written to avoid deleting other files

find . -name "*.un2.fastq" -type f -delete
```

### **3)** Quality Filter and Assign Sample IDs to Sequences
####Illumina MiSeq platform trims Nextera indices and adaptors prior to writing sequence data to BaseSpace. However, we still use QIIME's split_libraries script to assign Sample IDs to each individual sequence using folder names and concatenate all joined sequences into one file for downstream analyses.

####Default quality filtering parameters are applied here so that base calls less than Q25 and sequences less than 200 bp are removed. These paramters can be changed with the **`-s (min_qual_score)`** and **`-l (min_seq_length)`** options. Our expected length at this step is 211 bp.

```bash
cd ..     #change directory back to the main working directory

multiple_split_libraries_fastq.py -i /home/qiime/2014MISEQ/joined_seqs -o /home/qiime/2014MISEQ/split_out --demultiplexing_method sampleid_by_file --include_input_dir_path --remove_filepath_in_name  #took 16 secs with test set
```
#### We output the resulting FASTA file (QIIME uses the file extension .fna) and log files into a new directory called: **`split_out`**

#####**This step can take a while to run, so one can make sure QIIME is not hung up by occasionally going into the newly created directory and right clicking on the output file to check file size and make sure it grows between checks. The process is done when the cursor prompt returns to the terminal window.**

### **4)** Trim Primers
#### Now we need to remove Zeale primers from the 5' **and** 3' ends of each sequence. It is easiest to remove the reverse primer first from the 3' end, then reverse complement the sequences, remove the forward primer which is now in a reverse orientation on the 3' end, then reverse complement the sequences again so they are back in the 5' --> 3' direction with both primers removed.

####First we validate the mapping file which tells QIIME what the primer sequences should be.
#### If QIIME flags any errors, make changes as necessary; help can be found here: http://qiime.org/scripts/validate_mapping_file.html

```bash
validate_mapping_file.py -m SampleMapFile.txt  #good practice to make sure all columns are in correct format

truncate_reverse_primer.py -f ./split_out/seqs.fna -m ./SampleMapFile.txt -z truncate_remove -M 1 -o ./Rtruncated
```

#### The .fna file is our sequence file, and the SampleMapFile.txt is the mapping file with forward and reverse primer sequence info. In this case, we allow 1 mismatch in the primer sequence. The mapping file must have unique barcodes for each sample. Becasue the barcodes are already removed by Illumina, we simply concatenated forward and reverse indexes used in the sequencing process so each sample has a unique 'barcode' to satisfy the `validate_mapping_file.py`:
![](D:\\Google Drive\\IONvsMISEQ\\BatDietWorkflow_files\\MapEx.jpg)

####We reverse complement the sequences using the FASTX Toolkit, which must be installed in our QIIME Virtual Box or directly in Macqiime or Linux in the main working directory, then again remove reverse primers, and flip sequences back. We used a local copy of FASTX Toolkit 0.0.12. Reverse complementing can also be done with the QIIME **`adjust_seq_orientation.py`** script. We used the FASTX command because we will use another FASTX tool later in this section.

#####**Note that a separate mapping file is needed with primer sequences also reverse complemented: `SampleMapFile_reverse.txt`.**

```bash
fastx_reverse_complement -i ./Rtruncated/seqs_rev_primer_truncated.fna -o ./Rcomp

truncate_reverse_primer.py -f ./Rcomp -m ./SampleMapFile_reverse.txt -z truncate_remove -M 1 -o ./Rcomp_forward_removed

fastx_reverse_complement -i ./Rcomp_forward_removed/Rcomp_rev_primer_truncated.fna -o ./trimmed_seqs.fna
```
#### We now have a file called **`trimmed_seqs.fna`** with all primers removed.

### **5)** Assign Similar Sequences to OTUs
####Next we cluster similar sequences using the local alignmnet 'swarm' method and allowing only a 2 bp mismatch. Note that this sequence clustering is different than clustering of amplicons on the flow cell during Illumina sequencing.

######Mahé F, Rognes T, Quince C, de Vargas C, Dunthorn M. (2014) Swarm: robust and fast clustering method for amplicon-based studies. PeerJ 2:e593 <http://dx.doi.org/10.7717/peerj.593>

```bash
pick_otus.py -i ./trimmed_seqs.fna -m swarm --swarm_resolution 2 -o ./swarmOTUs    #took 18 sec with test set
```
#### Now we have a new directory with the clusters of sequences: **`swarmOTUs`**

#####**This step can take a while to run, and no files are produced until the process is complete. One can make sure QIIME is not hung up by using the** **`top`** **UNIX command in a new terminal window. The pick-otus QIIME command should normally be at the top of the list and using the greatest %CPU.**

### **6)** Build and Filter OTU Occurrence Table (BIOM)
####We now create our BIOM table to keep track of which OTUs were present in each sample. We use the new swarm output file **`trimmed_seqs_otus.txt`** as input.

```bash
make_otu_table.py -i ./swarmOTUs/trimmed_seqs_otus.txt -o ./swarmOTUs/seqs.biom
```

We filter out low abundance OTU clusters (< 10 copies in at least 1 sample of the entire data set) before writing data to the table using a script that calls the 'pandas' python package. We used Python 2.7.3 that should come as a dependency with the QIIME install. The provided 'pandas_filter10.py' script has to be in the **`swarmOTUs`** directory.

```bash
cd ./swarmOTUs     #change to the new directory

cp ../pandas_filter10.py .

python pandas_filter10.py     #This script will find the seqs.biom file created as output in the last step
```
####This script creates a new file called **`exclude.txt`** that we can use in the next step to remove those low abundance OTU clusters.

####After removing the unwanted low abundance OTU clusters, we write the desired OTUs to a BIOM table in human-readable tabular format. To label the denovo OTUs, we use the **`pick_rep_set.py`** script to pick one representative sequence from each cluster. Then we pull those desired representative sequences from our original .fna file and write a new .fna. From here on out, we use the representative sequences from each OTU cluster to reduce processing demand.

```bash
cd ..     #change back to the main working directory

filter_otus_from_otu_table.py -i ./swarmOTUs/seqs.biom -o ./swarmOTUs/otu_table_no_singletons.biom -e ./swarmOTUs/exclude.txt

biom convert -i ./swarmOTUs/otu_table_no_singletons.biom -o ./swarmOTUs/2014_biom_no_singletons.txt --table-type="OTU table" --to-tsv

pick_rep_set.py -i ./swarmOTUs/trimmed_seqs_otus.txt -f ./trimmed_seqs.fna -m most_abundant -o ./2014_final.fna -l ./2014final_log

filter_fasta.py -f ./2014_final.fna -b ./swarmOTUs/otu_table_no_singletons.biom -o ./2014_final_no_singletons.fna
```
####The last command in UNIX will convert our final FASTA file to a tab-delimited file we can use to assign taxonomy in the BOLD database. This file will be useful when resolving taxonomic discrepancies in the last section: [Manual Vetting of Results]. The **`fasta_formatter`** command is part of the FASTX Toolkit.  Ion does not use the paired end read system; thus, we keep forward and reverse sequences separate and *do not*  try to join sequences.

####All commands in this section were run inside the QIIME Virtual Box created above in the [Install](#Install) section. Open the QIIME machine and start a new Terminal window. It should provide a command line prompt similar to this: qiime@qiime-190-virtual-box:

####If you will be using a similar workflow more than once, it helps to use the **`time`** UNIX command in front of other commands. This command returns the processing time, in minutes and seconds, which simply helps plan future data analyses. For example:
![](D:\\Google Drive\\IONvsMISEQ\\BatDietWorkflow_files\\timeEx.jpg)

### **1)** Organize Forward and Reverse Files
#### Amplicons are read in either forward **_or_**  reverse direction, depending on which end attached to the Ion chip. Thus, we expect 1 FASTQ file for each unique barcode used. In our case, we double barcoded samples (n = 50; 46 bat samples + 4 blanks) with 10 unique forward primers and 5 unique reverse primers, so we expect 15 fastq files. We know the files 1--10 were sequenced in the forward direction and files 11--15 were read in the reverse direction. The raw output folder should look like this:

| **File Name**   | **Direction** |
|:---------------:|:-------------:|
| 2015-03-20.IonXpress_001.fastq | forward  |
| 2015-03-20.IonXpress_002.fastq | forward  |
| 2015-03-20.IonXpress_003.fastq | forward  |
| . . . . . . . . .              | . . .    |
| 2015-03-20.IonXpress_014.fastq | reverse  |
| 2015-03-20.IonXpress_015.fastq | reverse  |

#### Create a new folder and move the expected files to that directory (i.e., IonXpress_001 to IonXpress_015). There will be other files in the directory containing reads not mapped back to a sample by forward/reverse barcode combination (i.e., IonXpress_075). Our new directory is called **`2014ION`**

#### Ion Torrent results are not demultiplexed so first we created 2 sub-folders, **`forward`** and **`reverse`**, and put the appropriate IonXpress fastq files into each one. Next, we perform several steps to prepare all the forward FASTQ files, then repeat the process with the reverse FASTQ files, then put all files into 1 FASTA before picking OTUs.

```bash
cd 2014ION

mkdir forward

mkdir reverse

mv *001* *002* *003* *004* *005* *006* *007* *008* *009* *010*  ./forward

mv *011* *012* *013* *014* *015*  ./reverse
```

### **2)** Prepare Barcodes for each File
####To simplify the demultiplexing process, we moved the 10 bp reverse barcodes to the front, just after the forward barcode, to create a new 20 bp unique 'barcode' for each sequence. For example:

```bash
cd forward

#Note that the 10bp forward barcode changes in each command correspond to each file
#The command searches the fastq file and moves the last 10bp to just after the forward barcode for each read

sed '2~4s/\(.*\)\(.\{10\}\)$/GTTACCTTAG\2\1/;' < 2015-03-20.IonXpress_001.fastq > F01_barcodes.fastq

sed '2~4s/\(.*\)\(.\{10\}\)$/GTTCTCCTTA\2\1/;' < 2015-03-20.IonXpress_002.fastq > F02_barcodes.fastq

sed '2~4s/\(.*\)\(.\{10\}\)$/GAATCCTCTT\2\1/;' < 2015-03-20.IonXpress_003.fastq > F03_barcodes.fastq

sed '2~4s/\(.*\)\(.\{10\}\)$/GATCTTGGTA\2\1/;' < 2015-03-20.IonXpress_004.fastq > F04_barcodes.fastq

sed '2~4s/\(.*\)\(.\{10\}\)$/GTTCCTTCTG\2\1/;' < 2015-03-20.IonXpress_005.fastq > F05_barcodes.fastq

sed '2~4s/\(.*\)\(.\{10\}\)$/GAACTTGCAG\2\1/;' < 2015-03-20.IonXpress_006.fastq > F06_barcodes.fastq

sed '2~4s/\(.*\)\(.\{10\}\)$/GAATCACGAA\2\1/;' < 2015-03-20.IonXpress_007.fastq > F07_barcodes.fastq

sed '2~4s/\(.*\)\(.\{10\}\)$/GTTATCGGAA\2\1/;' < 2015-03-20.IonXpress_008.fastq > F08_barcodes.fastq

sed '2~4s/\(.*\)\(.\{10\}\)$/GTTCCGCTCA\2\1/;' < 2015-03-20.IonXpress_009.fastq > F09_barcodes.fastq

sed '2~4s/\(.*\)\(.\{10\}\)$/GTTCGGTCAG\2\1/;' < 2015-03-20.IonXpress_010.fastq > F10_barcodes.fastq
```

### **3)** Concatenate Forward FASTQ files
#### All sequences still have forward and reverse barcodes attached, in a new unique 20 bp forward barcode, so there is no chance of mixing up sequences by host sample.

```bash
cat *_barcodes.fastq > forwardseqs.fastq
```

### **4)** Convert FASTQ to FASTA
#### We need separate FASTA and QUAL files for demultiplexing and quality filtering because downstream QIIME scripts require FASTA format as input.

```bash
convert_fastaqual_fastq.py -f forwardseqs.fastq -c fastq_to_fastaqual  #took ~25 seconds with test set
```
#### This will created 2 new files in the **`forward`** directory: **`forwardseqs.fna`** and **`forwardseqs.qual`**. These are the new FASTA file (.fna) and the associated quality file (QUAL).

### **5)** Validate Mapping File
#### We validate the mapping file that tells QIIME which which barcodes and primers belong to each sample so that we can maintain the path from sequences back to host samples. Make sure the **`SampleMapF.txt`** file is in the **`forward`** directory.

```bash
cd ..                   #change back to 2014ION dir

validate_mapping_file.py -m SampleMapF.txt

head SampleMapF.txt
```
#### If QIIME flags any errors, make changes as necessary; help can be found here: http://qiime.org/scripts/validate_mapping_file.html

####The mapping file should look like this. Notice the new 20 bp barcode created in step **2)** and the Ion-specific **GAT** adapter on the reverse primer:
![](D:\\Google Drive\\IONvsMISEQ\\BatDietWorkflow_files\\SampleMapF.jpg)

### **6)** Demultiplex and Quality Filter
#### During this step, we quality filter base calls less than Q20 and sequences less than 200 bp, assign QIIME labels to each sequence, corresponding to host sample, and then remove the new 20 bp barcode and both primers. These parameters can be changed with the **`-s (min_qual_score)`** and **`-l (min_seq_length)`** options. We also remove any sequences with a homopolymer run of 10 or more bp; there is a natural 8 bp conserved region in our target area. The expected length at this point is 234 bp.

```bash
split_libraries.py -m SampleMapF.txt -f ./forward/forwardseqs.fna -q ./forward/forwardseqs.qual -o demultiplexedF -b 20 -H 10 -M 1 -s 20 -z truncate_remove   #took 1 min, 7 sec with test set
```
#####**This step can take a while to run, so one can make sure QIIME is not hung up by occasionally going into the newly created directory and right clicking on the output file to check file size and make sure it grows between checks. The process is done when the cursor prompt returns to the terminal window.**

### **7)** Repeat steps 2--6 for Reverse FASTQ files
#####**Note that a separate mapping file is needed with primer sequences also reverse complemented: `SampleMapR.txt`.**

```bash
cd ./reverse

sed '2~4s/\(.*\)\(.\{10\}\)$/GATTCGAGGA\2\1/;' < 2015-03-20.IonXpress_011.fastq > R11_barcodes.fastq

sed '2~4s/\(.*\)\(.\{10\}\)$/GAACCACCTA\2\1/;' < 2015-03-20.IonXpress_012.fastq > R12_barcodes.fastq

sed '2~4s/\(.*\)\(.\{10\}\)$/GTCCGTTAGA\2\1/;' < 2015-03-20.IonXpress_013.fastq > R13_barcodes.fastq

sed '2~4s/\(.*\)\(.\{10\}\)$/GACACTCCAA\2\1/;' < 2015-03-20.IonXpress_014.fastq > R14_barcodes.fastq

sed '2~4s/\(.*\)\(.\{10\}\)$/GACCTCTAGA\2\1/;' < 2015-03-20.IonXpress_015.fastq > R15_barcodes.fastq

cat *_barcodes.fastq > reverseseqs.fastq

convert_fastaqual_fastq.py -f reverseseqs.fastq -c fastq_to_fastaqual
#creates 2 new files: reverseseqs.fna and reverseseqs.qual
cd ..

validate_mapping_file.py -m SampleMapR.txt

split_libraries.py -m SampleMapR.txt -f ./reverse/reverseseqs.fna -q ./reverse/reverseseqs.qual -o demultiplexedR -b 20 -H 10 -M 1 -s 20 -z truncate_remove                 #if you get a high number of mismatches and sequences are discarded, double check that you have the right mapping file
```
#####**This step can take a while to run, so one can make sure QIIME is not hung up by occasionally going into the newly created directory and right clicking on the output file to check file size and make sure it grows between checks. The process is done when the cursor prompt returns to the terminal window.**

### **8)** Concatenate Forward and Reverse to 1 File
#### First we rename the forward FASTA file (containing all the forward sequences now) and move it to our main project folder. Repeat for the reverse FASTA file, reverse complement, and then concatenate all sequences into a single file with everything in the 5' --> 3' orientation.

####We reverse complement the sequences using the FASTX Toolkit, which must be installed in our QIIME Virtual Box or directly in Macqiime or Linux in the main working directory. We used a local copy of FASTX Toolkit 0.0.12. Reverse complementing can also be done with the QIIME **`adjust_seq_orientation.py`** script. We used the FASTX command because we will use another FASTX tool later in this section.

```bash
cd demultiplexedF

mv seqs.fna Fseqs.fna

mv Fseqs.fna ..

cd ..

cd ./demultiplexedR

mv seqs.fna Rseqs.fna

mv Rseqs.fna ..

cd ..

fastx_reverse_complement -i ./Rseqs.fna  -o ./Rseqs_flipped.fna

cat Fseqs.fna Rseqs_flipped.fna > 2014ionseqs.fna
```

### **9)** Assign Similar Sequences to OTUs
####Next we cluster similar sequences using the local alignment 'swarm' method and allowing only a 2 bp mismatch. Note that this sequence clustering is different than clustering of amplicons on the flow cell during Illumina sequencing.

######Mahé F, Rognes T, Quince C, de Vargas C, Dunthorn M. (2014) Swarm: robust and fast clustering method for amplicon-based studies. PeerJ 2:e593 <http://dx.doi.org/10.7717/peerj.593>

```bash
pick_otus.py -i ./2014ionseqs.fna -m swarm --swarm_resolution 2 -o ./swarmOTUs   #took 1.8 sec with test set
```
#### Now we have a new directory with the clusters of sequences: **`swarmOTUs`**

#####**This step can take a while to run, and no files are produced until the process is complete. So one can make sure QIIME is not hung up by using the** **`top`** **UNIX command in a new terminal window. The pick-otus QIIME command should be at the top of the list and using the greatest %CPU.**

### **10)** Build and Filter OTU Occurrence Table (BIOM)
####We now create our BIOM table to keep track of which OTUs were present in each sample. We use the swarm output file `2014ionseqs_otus.txt` as input.

```bash
make_otu_table.py -i ./swarmOTUs/2014ionseqs_otus.txt -o ./swarmOTUs/seqs.biom
```
####We filtered out low abundance OTU clusters (< 10 copies in at least 1 sample of the entire data set) before writing data to the table using a script that calls the 'pandas' python package. We used Python 2.7.3 that should come as a dependency with the QIIME install. The provided 'pandas_filter10.py' script has to be in the **`swarmOTUs`** directory.

```bash
cd ./swarmOTUs     #change to the new directory

cp ../pandas_filter10.py .

python pandas_filter10.py  #This script will find the seqs.biom file created as output in the last step
```
####This scipt creates a new file called **`exclude.txt`** that we can use in the next step to remove those low abundance OTU clusters.

####After removing the unwanted low abundance OTU clusters, we write the desired OTUs to a BIOM table in human-readable tabular format. To label the denovo OTUs, we use the **`pick_rep_set.py`** script to pick one representative sequence from each cluster. Then we pull those desired representative sequences from our original .fna file and write a new .fna. From here on out, we use the representative sequences from each OTU cluster to reduce processing demand.

```bash

cd ..     #change back to the main working directory

filter_otus_from_otu_table.py -i ./swarmOTUs/seqs.biom -o ./swarmOTUs/otu_table_no_singletons.biom -e ./swarmOTUs/exclude.txt

biom convert -i ./swarmOTUs/otu_table_no_singletons.biom -o ./swarmOTUs/2014ion_biom_no_singletons.txt --table-type="OTU table" --to-tsv

pick_rep_set.py -i ./swarmOTUs/2014ionseqs_otus.txt -f ./2014ionseqs.fna -m most_abundant -o ./2014ion_final.fna -l ./2014ion_final_log

filter_fasta.py -f ./2014ion_final.fna -b ./swarmOTUs/otu_table_no_singletons.biom -o ./2014ion_final_no_singletons.fna
```
####The last command in UNIX will convert our final FASTA file to a tab-delimited file we can use to assign taxonomy in the BOLD database. This file will be useful when resolving taxonomic discrepancies in the last section: [Manual Vetting](#Manual Vetting). The **`fasta_formatter`** command is part of the FASTX Toolkit. We run the command and then manually add column headers to the text file called: **seqID** and **seqs**.

```bash
fasta_formatter -i ./2014ion_final_no_singletons.fna -o ./2014iontabseqs.txt -t
```

#Assign Taxonomy and Filter Results -- ___Run commands in R___

####After transferring the **`2014tabseqs.txt`** and the **`2014_biom_no_singletons.txt`** files to a working directory outside of the QIIME system, we used several R packages to assign taxonomy and filter out unwanted returns, such as low similarity matches and those outside a reasonable geographic area.

####The following sections use the output files from the [Illumina Miseq](#Illumina MiSeq) section above. If you only have output from the [Ion Torrent](#Ion Torrent) section, use **`2014iontabseqs.txt`** and **`2014_ion_biom_no_singletons.txt`**

### **1)** RStudio and R Notebooks
#### To edit source code or to run each step and follow the tutorial, the easiest method is to open the provided .Rmd (R Markdown) file in RStudio: https://www.rstudio.com/products/rstudio/download/.
####We used RStudio Version 1.0.136 -- © 2009-2016 RStudio, Inc. Program R must first be installed on the same machine: https://www.r-project.org/. We used R version 3.3.2 (2016-10-31) -- "Sincere Pumpkin Patch". All remaining data processing steps in this section should be run in RStudio.

```r
install.packages('rmarkdown')
library ('rmarkdown')
```

### **2)** Set Working Directory and Read in Data
#### Change directory path as needed.

```r
setwd("D:\\Google Drive\\IONvsMISEQ")

mydata <- read.table("2014tabseqs.txt", header = TRUE, sep = "\t", dec = ".", stringsAsFactors = FALSE)

head(mydata)
```

```
##          denovo0.A4900.S9.L001_R1_001_0
## 1  denovo1 A4593.S13.L001_R1_001_116544
## 2    denovo10 A4900.S9.L001_R1_001_3230
## 3     denovo11 A4900.S9.L001_R1_001_483
## 4 denovo12 A4593.S13.L001_R1_001_117167
## 5 denovo13 A4593.S13.L001_R1_001_116755
## 6    denovo14 A4900.S9.L001_R1_001_2085
##    AGATATTGGAACTTTATATTTTATTTTTGGAATTTGAGCAGGAATAGTAGGAACATCTTTAAGACTTTTAATTCGAGCTGAATTAGGTAATCCAGGATCTCTAATTGGAGATGACCAAATTTATAATACCATTGTAACAGCTCATGCTTTTATTATAATTTTTTTCATAGTTATACCTATTATAATTGGAGGATTTGGAAATTGATTAGTA
## 1  AGATATTGGAACTTTATATTTTATTTTTGGTATTTGATCTGGTATAGTAGGAACTTCATTAAGATTATTAATTCGGGCAGAATTAGGAAATCCTGGATCATTAATTGGTGATGACCAAATTTATAATACTATTGTTACAGCCCATGCTTTTATTATAATTTTTTTTATGGTAATACCCATTATAATTGGAGGATTTGGAAATTGATTAGTA
## 2  AGATATTGGAACTTTATATTTTATTTTTGGGGCATGATCAGGTATAATTGGTACATCTATAAGATTAATAATTCGTACTGAATTAGGAAATTATGGATCTTTAATTGGTGATGATCAAATTTATAATGTTATTGTTACAGCCCATGCTTTTATTATAATTTTCTTTATAGTTATACCTATTATAATTGGAGGATTTGGTAATTGATTAGTA
## 3  AGATATTGGAACTTTATATTTTATTTTTGGAGCTTGGGCAGGAATGGTTGGAACTTCGTTGAGAATTTTAATTCGGGCAGAACTTGGACATCCAGGGGCATTAATTGGTGATGATCAAATTTATAACGTAATTGTAACAGCTCATGCTTTTATCATAATTTTCTTTATAGTAATACCTATTATAATTGGAGGATTTGGAAATTGATTAGTA
## 4  AGATATTGGAACTTTATATTTTATTTTTGGGGCTTGGGCAGGAATAGTTGGAACTTCATTAAGAATTTTAATTCGAGCAGAACTTGGTCATCCGGGGGCACTAATTGGTGATGATCAAATTTATAATGTTATTGTAACAGCTCATGCATTTATTATAATTTTTTTTATAGTAATACCCATTATAATTGGAGGATTTGGAAATTGATTAGTA
## 5  AGATATTGGAACTTTATATTTTATTTTTGGTGCATGATCAGGTATAGTAGGAACATCTTTAAGTATACTAATTCGAGCTGAATTAAATCAACCAGGATCTTTAATTGGAGATGATCAAATCTATAATGTAATTGTAACAGCCCATGCATTTATTATAATTTTTTTCATAGTTATACCAATTTTAATTGGAGGATTTGGAAATTGATTAGTA
## 6 AGATATTGGAACTTTATATTTTATTTTTGGAGCATGATCAGGTGTAATTGGTACATCTATAAGATTAATAATTCGTACTGAATTAGGAAATTCTGGGCCTTTAATTGGTGATGACCAAAATTTATAATGTTATTGTTACAGCCCATGCTTTTATTATATTTTTCTAGAGTTATATACCTATTATAATTGGAGGATTTGGAAATTGATTAGTA
```

```r
mydata2 <- as.list(setNames(mydata$seqs, mydata$seqID))  #make sure headers are not capitalized
```

### **3)** Query BOLD for OTU Matches
####Use the **`bold_identify`** function to get sequences from the BOLD API. This will return all the matches in BOLD, which should also include sequences mined from GenBank (https://www.ncbi.nlm.nih.gov/genbank/).

```r
install.packages('bold')

library('bold')

output <- bold_identify(sequences = mydata2, db = "COX1", response=FALSE) #This can take several hours to run
```

### **4)** Trim Output by User-specific Number of Matches
#### We trimmed our output to the top 40 matches for each OTU; this number can vary depending on project objectives. In many cases there will be up to 100 matches for each OTU. Results mined from GenBank are only returned at the Order level and require further investigation in the [Manual Vetting of Results] section. For our data, trimming to the 40 top matches left enough sequences to resolve discrepancies while also removing extranneous results, ultimately making it easier to parse through each OTU and assign taxonomy.

```r
outtax40 <- lapply(output, head, n=40)

outtaxframe <- do.call("rbind", lapply(outtax40, data.frame))
```
####The **`outtaxframe`** returns the top 40 matches, by similarity, from the BOLD results.

### **5)** Filter out Seqs < 98% and Unlikely by Geography
####This step not only filters out low percent similarity matches, but also helps with discrepancy resolution when several specimens from BOLD match one OTU at the same similarity.

####We only kept matches that were > 98.4% but other studies have used lower % similarities. Without this filtering step it takes much longer to assign taxonomy with manual vetting of matches by geographic area. For example, even at > 98.4% similarity, it is possible for one OTU to match a moth specimen from the USA **and** a butterfly from Papua New Guinea.

```r
#We used a custom header style for the final xlsx file

install.packages('openxlsx')

library('openxlsx')     #Must have Rtools installed and check box to edit PATH or afterwards do: Sys.setenv("R_ZIPCMD" = "C:/Rtools/bin/zip.exe") ##path to zip.exe

HS <- createStyle(fontSize=13, fontColour='navy', numFmt='GENERAL', halign='center', valign='center', textDecoration='bold', wrapText=TRUE)
```

```r
install.packages('dplyr')

library('dplyr')
library('tibble')

outtaxframe %<>%     #This only keeps rows that we want and updates the dataframe
	rownames_to_column("seqID") %>%
	filter(specimen_country %in% c("United States", "Canada"), similarity >= 0.98)

write.xlsx(outtaxframe, file='2014outtaxframe40.xlsx', asTable=FALSE, colNames=TRUE, rowNames=TRUE, headerStyle=HS)

     #Might get an error if Rtools is not installed - follow error message suggestions to get Rtools
```

####The final filtered file should look like this, with up to 40 records for each denovo OTU:

#Manual Vetting of Results -- ___Work in a Web Browser___

####The 3 files of interest for assigning final taxonomy are: **`2014outtaxframe40.xlsx`**, **`2014tabseqs.txt`**, and **`2014final_no_singletons.txt`**.

####First, we open the **`2014biom_no_singletons.txt`** file and convert any within-sample occurrences < 10 to 0 to remove low abundance OTU occurrences with simple find and replace commands in MS Excel. This step is to remove potential sequencing errors; the threshold of 10 is arbitrary and can be changed depending on project objectives.

####Next, we open **`2014outtaxframe40.xlsx`** and , and look for any discrepancies at the same similarity in our output from **`bold_identify`**, and then assign taxonomy in new columns in the **`2014final_no_singletons.txt`** file.

###*Updating the BIOM table with Taxonomy*
![](D:\\Google Drive\\IONvsMISEQ\\BatDietWorkflow_files\\firstLook.png)

####Sometimes there are discrepancies in the results returned from **`bold_identify`** that we need to manually investigate.

####In this example, results for OTU denovo118 match 6 different species in the same genus at 100%. We know that all of these matching specimen records could occur within our region because we filtered the **`2014outtaxframe40.xlsx`** based on similarity *and* geography. Nonetheless, we chose a simple example to demonstrate the process of how we learned to convert results into assigned taxonomy.

###*Discrepancy Example*
![](D:\\Google Drive\\IONvsMISEQ\\BatDietWorkflow_files\\discrepancyex.jpg)

####Next, we go to our **`2014tabseqs.txt`** file to find the sequence for denovo118

###*Find the Sequence in Question*
![](D:\\Google Drive\\IONvsMISEQ\\BatDietWorkflow_files\\findtheseq.jpg)

####We cut and paste the sequence into the BOLD Identification browser in FASTA format (http://www.boldsystems.org/index.php/IDS_OpenIdEngine)

###*Paste Sequence into BOLD*
![](D:\\Google Drive\\IONvsMISEQ\\BatDietWorkflow_files\\pasteSeq.jpg)

####We get many matches at 100%, including some out of our area. There are several *Catocala spp.*, as expected. We don't expect any Nymphalidae butterflies in bat feces so we disregard those. That leaves *Piletocera sodalis* as another potential prey.

###*Re-examine the Matches*
![](D:\\Google Drive\\IONvsMISEQ\\BatDietWorkflow_files\\idRequest.jpg)

####When we investigate *Piletocera sodalis* in the BOLD Taxonomy browser, we learn that it is only found in Japan, China, and Korea, so we disregard that potential prey. Because we still have 6 *Catocala spp.* possible, we can only assign taxonomy to OTU denovo118 as *Catocala spp.*

###*Cross-reference with Expectations*
![](D:\\Google Drive\\IONvsMISEQ\\BatDietWorkflow_files\\mothOutofArea.jpg)

repository: tdivoll/Bat-Diet-Metabarcoding

repo_name: tdivoll/Bat-Diet-Metabarcoding


text:
---
layout: blog
name: "City Data Hack at Urban Innovation Centre, London"
description: "The hack focused on using data to solve urban challenges like social isolation, unemployment and transport difficulties. There were eleven teams with 60 hackers from a range of backgrounds include data, UX, software engineering and service design. Many had a special interest in urban planning and there were quite a few students present."
category: Blog
display_date: March 2018
img: /img/blog/citydatahack/team-sm.jpeg
---

City Data Hack at Urban Innovation Centre, London

The hack focused on using data to solve urban challenges like social isolation, unemployment and transport difficulties. There were eleven teams with 60 hackers from a range of backgrounds include data, UX, software engineering and service design. Many had a special interest in urban planning and there were quite a few students present.

Future Cities Catapult and Space Syntax were the hosts of this hackathon by Tombolo, held at the Urban Innovation Centre in London from 16–18 March 2018.

Tombolo is a Future Cities Catapult and Space Syntax R&D project funded by Innovate UK. They have created the Digital Connector which was used by teams during the hack. They explored how city authorities could improve their use of data in solving urban challenges.

Three challenges were provided by the challenge partners:

Skills and Education – Greater London Authority

Social Isolation – London Borough of Barking and Dagenham

Active Travel Plans – Transport for London

Multi-disciplinary teams had 48 hours to hack the data, try out some new tools, including the Digital Connector and a data visualisation platform, analyse data to uncover new and relevant insights, then propose how each city authority could redesign their services to tackle their local challenge.

The hackathon included mentorship from experienced data scientists, digital designers, service designers and open source experts who work in the urban planning space.

Here are the teams who participated in the City Data Hack:

Projects / Teams Intros

Below are the competing teams project intros.

Familiar Ground

They tackled the social isolation challenge where people feel excluded from their community. They created an app that gives users ‘small wins’ and connects them with others based on their interests.

Alternative Treatment

    This team also tackled social isolation and being excluded from the community. Their solution was targeted at elderly people who are feeling lonely. They created a bot as a form of AI that can propose an event based on your location that helps neighbours communicate with each other. They called it ‘spirit of the place’ since it personifies a building shared by neighbours such as a block of flats or a neighbourhood.

Me2We

    This team also did social isolation and being excluded from the community, which affects 15% of the population – both young and old people in particular. Their app focused on our ‘internal weather’, with the weather as a metaphor for how we’re feeling. They wanted it to be both personal and local, and their strapline is: “Bringing sunshine to Barking and Dagenham”.

Yes

    This team was solving the problem of young people and unemployment, and how we can get young people to explore different career options to the traditional university pathway. Their platform is called SEEB and it matched skills and aspirations with vocational training, courses and jobs.

Keep Going

    Another team that went for social isolation and being excluded from the community. They created a web-based application to help people help themselves by focusing on a sense of accomplishment. It’s aimed at being a source of encouragement for people who are feeling lonely.

O-One

    This team focused on active transit, so they came up with a portal to help TFL (Transport for London) to communicate with its property developers, allowing them to set up targets that would help to monitor project progress.

Find my future

    This team addressed employment and skills challenges, disrupting the standard technology which is typically quizzes and surveys. Their web-based app uses data to help people visualize their interests and career priorities on a graph. It then links on to further vocational and training options.

City Dimensions

    Public transport challenge with the Digital Connector.

Bee Hive

    Bee Hive developed a travel planning tool that helps city developers gain access to urban data earlier in the development process. This means they can make urban planning more accessible for residents and provide desperately needed amenities. This will improve infrastructure and reduce the cost of travel.

Hackathon Presentations

All presentations

Final Results

The team O-One won the hack with their TfL portal to solve the city planning challenge and monitor project progress. They won both the travel challenge category and the hackathon overall, and were recognised for the multiple ways their solution can be developed in the future. Find My Future won the employment and skills challenge.  Visualising Exclusion won the social isolation category.

Team Keep Going won the best photo of the weekend, and for collaborating the best out of all the teams.

The hackathon is now followed by an optional month-long competition for teams to develop on their proposals.

Results

Resources

Playlist for the videos on Youtube and the unofficial GitHub organisation for the game projects.

Youtube playlist

CityDataHack GitHub

{% include catherine-heath.html %}
{% include eddie-jaoude.html %}


text:
# ABE-Proxy
The ABE Proxy is the new architectural component responsible to perform the heavy ABE encryption operations and the integration of CP-ABE and AES encryption techniques, so to speed up the encryption/decryption processing and reduce resource needs on the client side.

It is a web based application that uses novel attribute-based encryption techniques in order to protect access to information, which is encrypted according to user’s provided access policies and the generation of decryption keys tied to the user’s profiles.

## Installation guide
In order to install the application, it is necessary to deploy the war obtained after the build process in the Tomcat environment, and to copy the Public Key generated by CPABE Service in the folder ABE-Proxy\WebContent\WEB-INF\cpabe_keys\pub_10 specified.

Another important configurable parameter is the expiration time that indicates the life-time of the associated symmetric key. When this life-time expires, it is strongly recommended to renegotiate a new symmetric key between data source and ABE proxy in order to guarantee that the communication continues reliable and secure, and to achieve the well-known Perfect Forward Secrecy. This parameter is set in the ABE Proxy’s *web.xm*l file, and is in millisecond format. Following an example:

	<init-param>
		<param-name>expiration_time</param-name>
		<param-value>5000</param-value>
	</init-param>

The ABE Proxy provides two public endpoint through two different sockets:

 - *ABE-Proxy/generate_shared_secret*: performs the agreement of a symmetric key between the data source and the ABE-Proxy using *Elliptic Curve Diffie-Hellman Ephemeral Static*, combined with the *Concat Key-Derivation Function*.
 - *ABE-Proxy/cpabe_information*: the data source delegates the CP-ABE encryption operations to the ABE-Proxy.

## Software dependencies
 The ABE-Proxy component has two important dependencies:

 - *CPABE library* (see [here](https://github.com/FINCONS-IBD/MQTT-SeDEM/tree/master/CPABE)) that provides the CP-ABE encryption and decryption algorithms and entities.
 - *Key Storage Service* component: in which the ABE-Proxy saves the CP-ABE encrypted AES symmetric keys. It is a fully configurable component (specified in the *web.xml* configuration file); in the project the NOSQL OrientDB database has been identified to fulfill these needs.

For more details about the building procedures required to set-up the single components please refer to the single projects descriptions.

repository: FINCONS-IBD/MQTT-SeDEM

repo_name: FINCONS-IBD/MQTT-SeDEM


text:
How to contribute to the plugins for jstart team members.

Contributing the the Object Storage Plugin:
Note: There are two repos you might be looking for.
1. [wp-bluemix-objectstorage](https://github.com/ibmjstart/wp-bluemix-objectstorage) - This is the working version
for the newest IBM Object storage and WordPress 4.4.
  * You should only edit this if a new bug has been found and you need to
    make a quick fix.
1. [wp-openstack-objectstorage](https://github.com/ibmjstart/wp-openstack-objectstorage) - This is the new repo
for the newest features from the WP Offload S3 plugin as well as IBM Object Storage and Wordpress 4.4.
  * You should begin work here so that our wordpress on bluemix app can have the most up to date features
from the WP Offload S3 plugin.
  * Im not entirely sure what you have to change so that the app knows to use the wp-openstack-objectstorage repo instead
of the wp-bluemix-objectstorage repo, but i think you have to edit the composer.json in the repo itself, the composer.json
in the wordpress-on-bluemix app, and the ibmjstart/wp-bluemix-config repo.

Both of these projects employ github releases and git tags. That means, if you want to develop in the code,
there are a couple changes you need to make in order to see your changes.
Note: A \* in the require section (like this `"ibmjstart/wp-bluemix-objectstorage" : "*"`) tells composer to use any
version it finds, and the statement `prefer-stable:true` at the end of the composer.json tells it to only
use the latest release.

Option 1 - Work in a new branch:
1. Create a new branch on the git hub repo
1. Go to the composer.json in your wordpress-on-bluemix project
1. Edit the require section to point to your branch.
```
  "require": {
        "ibmjstart/wp-bluemix-objectstorage" : "dev-<Your Branch>",
  },
    ```
1. Merge your branch with the master.
1. In github (or through git), create a new release with a tag; this will document your changes and mark them
as latest so they will be used. Look [here](https://github.com/ibmjstart/wp-bluemix-objectstorage/releases) for
an example.

Option 2 - Work in the master branch:
* I dont necessarily recommend this option, becuase the master should be kept clean and in the worst case something
could go wrong and mess stuff up.
* But, theoretically, using a version/tag system, you could make your changes in master, then
once complete, add a new release/tag so that your changes are seen as latest.

* Similar to before, you just have to make this change to the require block so you can see your changes during development.
```
  "require": {
        "ibmjstart/wp-bluemix-objectstorage" : "dev-master",
  }, ```

repository: Colonel-Colt/bluemix-wordpress

repo_name: Colonel-Colt/bluemix-wordpress


text:
---
title: Configure SSL/TLS
---
# Configure SSL/TLS

> **NOTE**: This article describes MagicOnion v3 configuration.

## SSL/TLS

As [official gRPC doc](https://grpc.io/docs/guides/auth/) notes gRPC supports SSL/TLS, and MagicOnion also support SSL/TLS.

> gRPC has SSL/TLS integration and promotes the use of SSL/TLS to authenticate the server, and to encrypt all the data exchanged between the client and the server. Optional mechanisms are available for clients to provide certificates for mutual authentication

Let's use [samples/ChatApp/ChatApp.Server](https://github.com/Cysharp/MagicOnion/tree/master/samples/ChatApp/ChatApp.Server) for server project, and [samples/ChatApp/ChatApp.Unity](https://github.com/Cysharp/MagicOnion/tree/master/samples/ChatApp/ChatApp.Unity) for client project.

## HTTP/2 on Amazon ElasticLoadBalancer

This section explains how to setup "SSL/TLS MagicOnion on AWS" with following 5 steps.

* [Generate self-signed certificate](#generate-self-signed-certificate)
* [Server configuration](#server-configuration)
* [Kubernetes deployments](#kubernetes-deployments)
* [Enable ALPN on NLB](#enable-alpn-on-nlb)
* [Client configuration](#client-configuration)

AWS LoadBalancer supports ALPN with ALB (Application Load Balancer) and NLB (Network Load Balancer), however ALB doesn't support HTTP/2 to the backend group, you must use NLB.

**Connection flow**

To establish HTTP/2 between Client and Server with NLB, gRPC connection should follow full `HTTP/2 over TLS`.
You must establish TLS connection for both "NLB (Listener) & Client" and "NLB (TargetGroup) & Server".

```
Server <-- TLS (Self-signed) --> NLB <-- TLS (ACM) --> Client
```

* NLB (Listener) & Client: NLB Listener work with ACM. Client use Gprpc.Core.dll embedded [roots.pem](https://github.com/grpc/grpc/blob/master/etc/roots.pem) by default.
* NLB (TargetGroup) & Server: You need set TargetGroup as TLS and listen gRPC Server as TLS. You can use self-signed cert, Let's Encrypt and others. (This README use self-signed cert.)

## Generate self-signed certificate

Following command will create 3 files `server.csr`, `server.key` and `server.crt`.
gRPC/MagicOnion Server requires server.crt and server.key.

```shell
# move to your server project
$ cd samples/ChatApp/ChatApp.Server

# generate certificates
# NOTE: CN=xxxx should match domain name to magic onion server pointing domain name
$ openssl genrsa 2048 > server.key
$ openssl req -new -sha256 -key server.key -out server.csr -subj "/C=JP/ST=Tokyo/L=Tokyo/O=MagicOnion Demo/OU=Dev/CN=*.example.com"
$ openssl x509 -req -in server.csr -signkey server.key -out server.crt -days 7300 -extensions server
```

Please modify `/C=JP/ST=Tokyo/L=Tokyo/O=MagicOnion Demo/OU=Dev/CN=*.example.com` to your domain.
Make sure `CN=xxxx` should match to domain that your MagicOnion Server will recieve request from your client.

> ATTENTION: **server.key** is very sensitive file, while **server.crt** can be public. DO NOT COPY server.key to your client.

## Server configuration

> NOTE: Server will use **server.crt** and **server.key**, if you didn't copy OpenSSL generated `server.crt` and `server.key`, please back to [generate certificate](#generate-self-signed-certificate) section and copy them.

Open `samples/ChatApp/ChatApp.Server/ChatApp.Server.csproj` and add following lines before `</Project>`.

```xml
  <ItemGroup>
    <Folder Include="LinkFromUnity\" />
  </ItemGroup>

  <!-- FOR SSL/TLS SUPPORT -->
  <ItemGroup>
    <None Update="server.crt">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
    <None Update="server.key">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
  </ItemGroup>

</Project>
```

MagicOnion.Hosting supports specifing certificate with appsettings.json.

```json
{
  "MagicOnion": {
    "ServerPorts": [
      {
        "Host": "<IP_ADDRESS>",
        "Port": 12345,
        "UseInsecureConnection": false,
        "ServerCredentials": [
          {
            "CertificatePath": "./server.crt",
            "KeyPath": "./server.key"
          }
        ]
      }
    ]
  }
}
```

Now `dotnet publish` output dll and certificates, generate your docker image and push it.

```shell
cd samples/ChatApp
docker build -t chatapp_magiconion:latest -f ChatApp.Server/Dockerfile .
docker tag chatapp_magiconion:latest your-image:latest
docker push your-image:latest
```

## Kubernetes deployments

Apply yaml to configure EKS deployments.

```shell
# replace with your ACM Arn and others.
ACM=arn:aws:acm:ap-northeast-1:123456781234:certificate/12345678-abcd-1234-efgh-1234abcd5678
DOMAIN=your-domain.com
IMAGE=your-image:latest

cat ./samples/ChatApp/k8s/nlb_acm.yaml \
    | sed -e "s|arn:aws:acm:ap-northeast-1:123456781234:certificate/12345678-abcd-1234-efgh-1234abcd5678|${ACM}|g" \
    | sed -e "s|nlb.example.com|${DOMAIN}|g" \
    | sed -e "s|cysharp/magiconion_sample_chatapp:3.0.13-chatapp|${IMAGE}|g" \
    | kubectl apply -f -
```

## Enable ALPN on NLB

Current Kubernetes Service AWS integration not support annotations for ALPN.
Configure NLB Listener's ALPN policy to `HTTP2Preferred` by following command, or via AWS Console.

```shell
$ dns=$(kubectl get svc chatapp-svc -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}')
$ nlb_arn=$(aws elbv2 describe-load-balancers | jq -r ".LoadBalancers[] | select(.DNSName == \"${dns}\") | .LoadBalancerArn")
$ listeners=$(aws elbv2 describe-listeners --load-balancer-arn "${nlb_arn}" | jq -r ".Listeners[].ListenerArn")
$ for listner in "${listeners[@]}"; do aws elbv2 modify-listener --listener-arn "${listner}" --alpn-policy HTTP2Preferred; done
```

## Client configuration

> NOTE: Client will use Grpc.Core.dll embedded **roots.pem** when not specify cert file with `SslCredentials` instance.

Open `samples/ChatApp/ChatApp.Unity/Assets/ChatComponent.cs`, channel creation is defined as `ChannelCredentials.Insecure` in `InitializeClient()`.
What you need tois change this line to use `SslCredentials`.

```csharp
this.channel = new Channel("localhost", 12345, ChannelCredentials.Insecure);
```

Replace this line to following.

```csharp
this.channel = new Channel("NLB.YOUR_DOMAIN.com", 12345, new SslCredentials());
```

Play on Unity Editor and confirm Unity MagicOnion Client can connect to MagicOnion Server.

![image](https://user-images.githubusercontent.com/3856350/95848171-a632ea00-0d88-11eb-8fb4-d3b322be5fb6.png)

> NOTE: If there are any trouble establish SSL/TLS connection, Unity Client will show `disconnected server.` log.

## TLS on LocalHost

This section explains how to setup "SSL/TLS MagicOnion on localhost" with following 4 steps.

* [Generate self-signed certificate](#generate-self-signed-certificate)
* [Simulate dummy domain via hosts](#simulate-dummy-domain-via-hosts)
* [Server configuration](#server-configuration)
* [Client configuration](#client-configuration)

**Connection flow**

```
Server <-- TLS (Self-signed) --> Client
```

## Generate self-signed certificate

Certificates are required to establish SSL/TLS with Server/Client channel connection.
Let's use [OpenSSL](https://github.com/openssl/openssl) to create required certificates.

Following command will create 3 files `server.csr`, `server.key` and `server.crt`.
gRPC/MagicOnion Server requires server.crt and server.key, and Client require server.crt.

```shell
# move to your server project
$ cd MagicOnion/samples/ChatApp/ChatApp.Server

# generate certificates
# NOTE: CN=xxxx should match domain name to magic onion server pointing domain name
$ openssl genrsa 2048 > server.key
$ openssl req -new -sha256 -key server.key -out server.csr -subj "/C=JP/ST=Tokyo/L=Tokyo/O=MagicOnion Demo/OU=Dev/CN=*.example.com"
$ openssl x509 -req -in server.csr -signkey server.key -out server.crt -days 7300 -extensions server

# server will use server.crt and server.key, leave generated certificates.

# client will use server.crt, copy certificate to StreamingAssets folder.
$ mkdir ../ChatApp.Unity/Assets/StreamingAssets
$ cp server.crt ../ChatApp.Unity/Assets/StreamingAssets/server.crt
```

Please modify `/C=JP/ST=Tokyo/L=Tokyo/O=MagicOnion Demo/OU=Dev/CN=*.example.com` your domain.
Make sure `CN=xxxx` should match to domain that your MagicOnion Server will recieve request from your client.

> ATTENTION: **server.key** is very sensitive file, while **server.crt** can be public. DO NOT COPY server.key to your client.

## Simulate dummy domain via hosts

Editting `hosts` file is the simple way to redirect dummy domain request to your localhost.

Let's set your CN to you hosts, example is `dummy.example.com`.
Open hosts file and add your entry.

```shell
# NOTE: edit hosts to test on localhost
# Windows: (use scoop to install sudo, or open elevated cmd or notepad.)
PS> sudo notepad c:\windows\system32\drivers\etc\hosts
# macos:
$ sudo vim /private/etc/hosts
# Linux:
$ sudo vim /etc/hosts
```

Entry format would be similar to this, please follow to your platform hosts rule.

```shell
<IP_ADDRESS>	dummy.example.com
```

After modifying hosts, `ping` to your dummy domain and confirm localhost is responding.

```shell
$ ping dummy.example.com

pinging to dummy.example.com [<IP_ADDRESS>] 32 bytes data:
<IP_ADDRESS> response: bytecount =32 time <1ms TTL=128
```

## Server configuration

> NOTE: Server will use **server.crt** and **server.key**, if you didn't copy OpenSSL generated `server.crt` and `server.key`, please back to [generate certificate](#generate-certificate) section and copy them.

Open `samples/ChatApp/ChatApp.Server/ChatApp.Server.csproj` and add following lines before `</Project>`.

```xml
  <ItemGroup>
    <Folder Include="LinkFromUnity\" />
  </ItemGroup>

  <!-- FOR SSL/TLS SUPPORT -->
  <ItemGroup>
    <None Update="server.crt">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
    <None Update="server.key">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
  </ItemGroup>

</Project>
```

MagicOnion.Hosting supports specifing certificate with appsettings.json.

```json
{
  "MagicOnion": {
    "ServerPorts": [
      {
        "Host": "<IP_ADDRESS>",
        "Port": 12345,
        "UseInsecureConnection": false,
        "ServerCredentials": [
          {
            "CertificatePath": "./server.crt",
            "KeyPath": "./server.key"
          }
        ]
      }
    ]
  }
}
```

Debug run server on Visual Studio, any IDE or docker.

> NOTE: Set `DOTNET_ENVIRONMENT=Production` to use tls configured appsettings.json

```shell
D0729 11:08:21.767387 Grpc.Core.Internal.NativeExtension gRPC native library loaded successfully.
Application started. Press Ctrl+C to shut down.
Hosting environment: Production
```

## Client configuration

> NOTE: Client will use **server.crt**, if you didn't copy OpenSSL generated `server.crt` and `server.key`, please back to [generate certificate](#generate-certificate) section and copy it.

Open `samples/ChatApp/ChatApp.Unity/Assets/ChatComponent.cs`, channel creation is defined as `ChannelCredentials.Insecure` in `InitializeClient()`.
What you need tois change this line to use `SslCredentials`.

```csharp
this.channel = new Channel("localhost", 12345, ChannelCredentials.Insecure);
```

Replace this line to following.

```csharp
var serverCred = new SslCredentials(File.ReadAllText(Path.Combine(Application.streamingAssetsPath, "server.crt")));
this.channel = new Channel("dummy.example.com", 5001, serverCred);
```

Play on Unity Editor and confirm Unity MagicOnion Client can connect to MagicOnion Server.

![image](https://user-images.githubusercontent.com/3856350/62017554-1be97f00-b1f2-11e9-9769-70464fe6d425.png)

> NOTE: If there are any trouble establish SSL/TLS connection, Unity Client will show `disconnected server.` log.

repository: Cysharp/MagicOnion

branch_name: refs/heads/main

gha_language: C#

repo_name: Cysharp/MagicOnion


text:
Bootstrap Jumbotron App
==========
[![Build Status](https://travis-ci.org/tmoreira2020/bootstrap-jumbotron-app.svg?branch=master)](https://travis-ci.org/tmoreira2020/bootstrap-jumbotron-app)
[![Maven Central](https://maven-badges.herokuapp.com/maven-central/br.com.thiagomoreira.liferay.plugins.bootstrap-jumbotron-app/bootstrap-jumbotron-app/badge.svg)](https://maven-badges.herokuapp.com/maven-central/br.com.thiagomoreira.liferay.plugins.bootstrap-jumbotron-app/bootstrap-jumbotron-app)

Bootstrap Jumbotron theme is a theme based on the Jumbotron example from http://getbootstrap.com/2.3.2/examples/hero.html and the _styled theme. By default it prints the site's name and description into the jumbotron area. It also enables add contact information to the footer through theme settings in the Site Administration area.

### License

Bootstrap Jumbotron App is licensed under [Apache 2](http://www.apache.org/licenses/LICENSE-2.0) license.

### Maven/Gradle

Bootstrap Jumbotron App is available on Maven central, the artifact is as follows:

Maven:

```xml
<dependency>
    <groupId>br.com.thiagomoreira.liferay.plugins.bootstrap-jumbotron-app</groupId>
    <artifactId>bootstrap-jumbotron-app</artifactId>
    <version>1.0.1</version>
    <type>lpkg</type>
</dependency>
```
Gradle:

```groovy
dependencies {
    compile(group: "br.com.thiagomoreira.liferay.plugins.bootstrap-jumbotron-app", name: "bootstrap-jumbotron-portlet", version: "1.0.1", type: "lpkg");
}
```
### Support
Bootstrap Jumbotron Portlet tracks [bugs and feature requests](https://github.com/tmoreira2020/bootstrap-jumbotron-app/issues) with Github's issue system. Feel free to open your [new ticket](https://github.com/tmoreira2020/bootstrap-jumbotron-app/issues/new)!

### Contributing

Bootstrap Jumbotron Portlet is a project based on Maven to improve it you just need to fork the repository, clone it and from the command line invoke

```shell
mvn package
```
After complete your work you can send a pull request to incorporate the modifications.

Enjoy!


text:
---
title: "Basic usage"
draft: false
type: docs
layout: single

menu:
  docs:
    weight: 10
---

# Basic usage

For the basic usage introduction we will be installing `pendulum`, a datetime library.
If you have not yet installed Poetry, refer to the [Introduction]({{< relref "docs" >}} "Introduction") chapter.

## Project setup

First, let's create our new project, let's call it `poetry-demo`:

```bash
poetry new poetry-demo
```

This will create the `poetry-demo` directory with the following content:

```text
poetry-demo
├── pyproject.toml
├── README.md
├── poetry_demo
│   └── __init__.py
└── tests
    └── __init__.py
```

The `pyproject.toml` file is what is the most important here. This will orchestrate
your project and its dependencies. For now, it looks like this:

```toml
[tool.poetry]
name = "poetry-demo"
version = "0.1.0"
description = ""
authors = ["Sébastien Eustace<EMAIL_ADDRESS>readme = "README.md"
packages = [{include = "poetry_demo"}]

[tool.poetry.dependencies]
python = "^3.7"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
```

Poetry assumes your package contains a package with the same name as `tool.poetry.name` located in the root of your
project. If this is not the case, populate [`tool.poetry.packages`]({{< relref "pyproject#packages" >}}) to specify
your packages and their locations.

Similarly, the traditional `MANIFEST.in` file is replaced by the `tool.poetry.readme`, `tool.poetry.include`, and
`tool.poetry.exclude` sections. `tool.poetry.exclude` is additionally implicitly populated by your `.gitignore`. For
full documentation on the project format, see the [pyproject section]({{< relref "pyproject" >}}) of the documentation.

Poetry will require you to explicitly specify what versions of Python you intend to support, and its universal locking
will guarantee that your project is installable (and all dependencies claim support for) all supported Python versions.

### Initialising a pre-existing project

Instead of creating a new project, Poetry can be used to 'initialise' a pre-populated
directory. To interactively create a `pyproject.toml` file in directory `pre-existing-project`:

```bash
cd pre-existing-project
poetry init
```

### Specifying dependencies

If you want to add dependencies to your project, you can specify them in the `tool.poetry.dependencies` section.

```toml
[tool.poetry.dependencies]
pendulum = "^2.1"
```

As you can see, it takes a mapping of **package names** and **version constraints**.

Poetry uses this information to search for the right set of files in package "repositories" that you register
in the `tool.poetry.source` section, or on [PyPI](https://pypi.org) by default.

Also, instead of modifying the `pyproject.toml` file by hand, you can use the `add` command.

```bash
$ poetry add pendulum
```

It will automatically find a suitable version constraint **and install** the package and sub-dependencies.

Poetry supports a rich [dependency specification]({{< relref "dependency-specification" >}}) syntax, including caret,
tilde, wildcard, inequality and
[multiple constraints]({{< relref "dependency-specification#multiple-constraints-dependencies" >}}) requirements.

## Using your virtual environment

By default, Poetry creates a virtual environment in `{cache-dir}/virtualenvs`.
You can change the [`cache-dir`]({{< relref "configuration#cache-dir" >}} "cache-dir configuration documentation") value
by editing the Poetry configuration.
Additionally, you can use the
[`virtualenvs.in-project`]({{< relref "configuration#virtualenvsin-project" >}}) configuration variable to create
virtual environments within your project directory.

There are several ways to run commands within this virtual environment.

{{% note %}}
**External virtual environment management**

Poetry will detect and respect an existing virtual environment that has been externally activated. This is a powerful
mechanism that is intended to be an alternative to Poetry's built-in, simplified environment management.

To take advantage of this, simply activate a virtual environment using your preferred method or tooling, before running
any Poetry commands that expect to manipulate an environment.
{{% /note %}}

### Using `poetry run`

To run your script simply use `poetry run python your_script.py`.
Likewise if you have command line tools such as `pytest` or `black` you can run them using `poetry run pytest`.

### Activating the virtual environment

The easiest way to activate the virtual environment is to create a nested shell with `poetry shell`.

To deactivate the virtual environment and exit this new shell type `exit`.
To deactivate the virtual environment without leaving the shell use `deactivate`.

{{% note %}}
**Why a nested shell?**

Child processes inherit their environment from their parents, but do not share
them. As such, any modifications made by a child process is not persisted after
the child process exits. A Python application (Poetry), being a child process,
cannot modify the environment of the shell that it has been called from such
that an activated virtual environment remains active after the Poetry command
has completed execution.

Therefore, Poetry has to create a sub-shell with the virtual environment activated
in order for the subsequent commands to run from within the virtual environment.
{{% /note %}}

If you'd like to prevent `poetry shell` from modifying your shell prompt on virtual environment activation, you should
set `VIRTUAL_ENV_DISABLE_PROMPT=1` as an environment variable before running the command.

Alternatively, to avoid creating a new shell, you can manually activate the
virtual environment by running `source {path_to_venv}/bin/activate` (`{path_to_venv}\Scripts\activate.ps1` in PowerShell).
To get the path to your virtual environment run `poetry env info --path`.
You can also combine these into a one-liner, such as `source $(poetry env info --path)/bin/activate`
(`& ((poetry env info --path) + "\Scripts\activate.ps1")` in Powershell).

To deactivate this virtual environment simply use `deactivate`.

|                   | POSIX Shell                                     | Windows (PowerShell)                                     | Exit/Deactivate |
|-------------------| ----------------------------------------------- |----------------------------------------------------------| --------------- |
| Sub-shell         | `poetry shell`                                  | `poetry shell`                                           | `exit`          |
| Manual Activation | `source {path_to_venv}/bin/activate`            | `{path_to_venv}\Scripts\activate.ps1`                    | `deactivate`    |
| One-liner         | `source $(poetry env info --path)/bin/activate` | `& ((poetry env info --path) + "\Scripts\activate.ps1")` | `deactivate`    |

## Version constraints

In our example, we are requesting the `pendulum` package with the version constraint `^2.1`.
This means any version greater or equal to 2.1.0 and less than 3.0.0 (`>=2.1.0 <3.0.0`).

Please read [Dependency specification]({{< relref "dependency-specification" >}} "Dependency specification documentation")
for more in-depth information on versions, how versions relate to each other, and on the different ways you can specify
dependencies.

{{% note %}}
**How does Poetry download the right files?**

When you specify a dependency in `pyproject.toml`, Poetry first takes the name of the package
that you have requested and searches for it in any repository you have registered using the `repositories` key.
If you have not registered any extra repositories, or it does not find a package with that name in the
repositories you have specified, it falls back to PyPI.

When Poetry finds the right package, it then attempts to find the best match for the version constraint you have
specified.
{{% /note %}}

## Installing dependencies

To install the defined dependencies for your project, just run the [`install`]({{< relref "cli#install" >}}) command.

```bash
poetry install
```

When you run this command, one of two things may happen:

### Installing without `poetry.lock`

If you have never run the command before and there is also no `poetry.lock` file present,
Poetry simply resolves all dependencies listed in your `pyproject.toml` file and downloads the latest version of their files.

When Poetry has finished installing, it writes all the packages and their exact versions that it downloaded to the `poetry.lock` file,
locking the project to those specific versions.
You should commit the `poetry.lock` file to your project repo so that all people working on the project are locked to the same versions of dependencies (more below).

### Installing with `poetry.lock`

This brings us to the second scenario. If there is already a `poetry.lock` file as well as a `pyproject.toml` file
when you run `poetry install`, it means either you ran the `install` command before,
or someone else on the project ran the `install` command and committed the `poetry.lock` file to the project (which is good).

Either way, running `install` when a `poetry.lock` file is present resolves and installs all dependencies that you listed in `pyproject.toml`,
but Poetry uses the exact versions listed in `poetry.lock` to ensure that the package versions are consistent for everyone working on your project.
As a result you will have all dependencies requested by your `pyproject.toml` file,
but they may not all be at the very latest available versions
(some dependencies listed in the `poetry.lock` file may have released newer versions since the file was created).
This is by design, it ensures that your project does not break because of unexpected changes in dependencies.

### Committing your `poetry.lock` file to version control

#### As an application developer

Application developers commit `poetry.lock` to get more reproducible builds.

Committing this file to VC is important because it will cause anyone who sets up the project
to use the exact same versions of the dependencies that you are using.
Your CI server, production machines, other developers in your team,
everything and everyone runs on the same dependencies,
which mitigates the potential for bugs affecting only some parts of the deployments.
Even if you develop alone, in six months when reinstalling the project you can feel confident
the dependencies installed are still working even if your dependencies released many new versions since then.
(See note below about using the update command.)

{{% warning %}} If you have added the recommended [`[build-system]`]({{< relref "pyproject#poetry-and-pep-517" >}}) section to your project's pyproject.toml then you _can_ successfully install your project and its dependencies into a virtual environment using a command like `pip install -e .`. However, pip will not use the lock file to determine dependency versions as the poetry-core build system is intended for library developers (see next section).
{{% /warning %}}

#### As a library developer

Library developers have more to consider.

The application ignores your library's lock file. It can use whatever dependency version meets the constraints in your `pyproject.toml`. The application will probably use the latest compatible dependency version. If your library's `poetry.lock` falls behind some new dependency version that breaks things for your users, you're likely to be the last to find out about it.

A simple way to avoid such a scenario is to omit the `poetry.lock` file. However, by doing so, you sacrifice reproducibility and performance to a certain extent. Without a lockfile, it can be difficult to find the reason for failing tests, because in addition to obvious code changes an unnoticed library update might be the culprit. Further, Poetry will have to lock before installing a dependency if `poetry.lock` has been omitted. Depending on the number of dependencies, locking may take a significant amount of time.

If you do not want to give up the reproducibility and performance benefits, consider a regular refresh of `poetry.lock` to stay up-to-date and reduce the risk of sudden breakage for users.

### Installing dependencies only

The current project is installed in [editable](https://pip.pypa.io/en/stable/topics/local-project-installs/) mode by default.

If you want to install the dependencies only, run the `install` command with the `--no-root` flag:

```bash
poetry install --no-root
```

## Updating dependencies to their latest versions

As mentioned above, the `poetry.lock` file prevents you from automatically getting the latest versions
of your dependencies.
To update to the latest versions, use the `update` command.
This will fetch the latest matching versions (according to your `pyproject.toml` file)
and update the lock file with the new versions.
(This is equivalent to deleting the `poetry.lock` file and running `install` again.)

{{% note %}}
Poetry will display a **Warning** when executing an install command if `poetry.lock` and `pyproject.toml`
are not synchronized.
{{% /note %}}

repository: python-poetry/poetry

repo_name: python-poetry/poetry


text:
---
layout: post
title: Creating your machine on the cloud with Digital Ocean
date: 2021-04-21T19:48:25.000-03:00
image: "https://res.cloudinary.com/jesstemporal/image/upload/v1640360836/covers/tutorial_gfgm5n.png"
comments: true
description: A step-by-step guide to create an instance (Droplet) at Digital Ocean
type: post
lang: en
tags:
- droplet
- ssh
- digital ocean
- cloud
- english
- tutorial
author_note_link: "https://jtemporal.com/criando-a-sua-maquina-na-nuvem-com-digital-ocean"
author_note: "Você pode ler este tutorial em Português"
translator: false
translated: "/criando-a-sua-maquina-na-nuvem-com-digital-ocean"

---
This post is a step-by-step on how to create a cloud instance on Digital Ocean.

## Get acquainted with Digital Ocean

So Digital Ocean (DO) provides a range of solutions in cloud computing. My favorite one is called [Droplets](https://www.digitalocean.com/products/droplets/). Droplets are a virtual machine that you can use for pretty much anything, and I love to use them as a powerful computer when I have limited power available. Still, it is also a pretty neat choice for those who want to code from anywhere, even from mobile devices.

There is a wide range of operating systems to chose from, and it is pretty straightforward to begin with, so whenever I need some cloud computing, I think of Digital Ocean as the first choice.

## Create the droplet

The first thing you’ll need to do is create your account on DO. To do this, you must [access their website](https://digitalocean.com) and sign up using the form already available on the landing page:

![Welcome page from Digital Ocean](https://i.imgur.com/fMY3yXz.jpg)

After that, you’ll be taken to your dashboard, where you’ll see all the solutions available, and you should click on the `Droplets` option on the menu on the left-hand side, and this will take you to the page where you can see a list of your current droplets:

![droplets list on dashboard](https://i.imgur.com/fKpV2pK.jpg)

As you can see in the above image, I already have a droplet there called `jtemporal` and I can also create a new one by clicking the `Create droplet` button on the top left corner, and this will take you to a page where you can choose the configuration you want for your droplet.

The first thing you need to choose, is the operational system:

![OS options](https://i.imgur.com/NsgOBze.jpg)

Then you have to pick a payment plan:

![payment plans for droplets](https://i.imgur.com/rgDRnA3.jpg)

Choosing the right one for either of these choices is essential. Currently, I mostly use my droplet for things like writing and creating examples for blog posts. The basic plan with a shared CPU fits the bill perfectly, but maybe a dedicated CPU might be your thing if you need something more heavy-duty. So choose with care. My current instance is from the basic plan with a 2GB CPU and 50GB disk, which will cost me ten dollars a month.

After picking the best configuration for your use case, you have to choose whether you want to “Add block storage”, imagine this is the external HD that you can take with you whenever you travel. Block storage saves the data and shares it across instances or maybe takes it from one instance to another. But note that block storages cost extra, so if you are on a budget. Keep that in mind. 😉

![add block storage](https://i.imgur.com/T9ajYfH.jpg)

After deciding which plan to use and whether or not you want to have block storage, you must choose a region for your instance; This means you have to decide where your machine's data center is physically located. The closer they are to you geographically, the less delay you'll have to deal with. I chose the New York region since it is the closest to me.

![datacenter region](https://i.imgur.com/FMYxQYU.jpg)

Each region comes with a VPC network configured, so you can seamlessly transfer data and information from one instance to another in the same region. You can add custom VPC networks too, but for now, the default will be enough.

![aditional options](https://i.imgur.com/ouc22ui.jpg)

You can also add some extra add-ons to your droplets, like monitoring and IPv6 connections availability. I didn't choose any add-ons since I don't need them right now but chose what best fits your needs.

Now that you have all the basic configuration for your instance set up, you'll need to define a way to access it. This is where the SSH keys come in. I have my SSH keys already set up in my account, and you can see by the picture below that I have a couple, so for this instance, I picked the one I'm currently using that is named `textastic`.

![SSH keys configuration](https://i.imgur.com/n4FyN4T.jpg)

To finalize, you need to give a name to your instance. If you don't want to choose a name Digital Ocean generates a random name, as you can see below:

![hostname](https://i.imgur.com/gsAWQ0E.jpg)

For the instance I'm going to use daily, I gave it a shorter name - jtemporal - the name I usually use for my handles on social media 🤣.

There are 3 more things you'll need to decide on: you can add tags to your instance; this can be a helpful organizational tool if you have many projects going on and you want to group instances based on topics. You can also create different projects and select the project you want here, and finally, you can enable backups. I didn't do any of this, but I truly recommend it. Now you are almost there. You only need to click the big green button that says `Create Droplet`:

![create droplet](https://i.imgur.com/i3F6S0A.jpg)

and your instance will be created and will start to boot up:

![instance booting up](https://i.imgur.com/LKMzsga.jpg)

Once you click the "Create Droplet" button, you'll see your new droplet on the list of droplets linked to your account, and you might need to wait a couple minutes while it finishes booting up. Once the droplet is ready, it will have a green dot by the name like my instance  `jtemporal` has in the above image, and then you can access it.

Congratulations! You did it! You created your droplet successfully!

---

[In this next post I talk about how to configure the Droplet so you can actually use it with SSH and all](https://jtemporal.com/configuring-and-accessing-your-droplet-via-ssh/), check it out 😉

gha_language: HTML


text:
---
title: IAfcApi.afc_file_read method
---

Attempts to the read the given number of bytes from the given file.

```csharp
public AfcError afc_file_read(AfcClientHandle client, ulong handle, byte[] data, uint length,
    ref uint bytesRead)
```

| parameter | description |
| --- | --- |
| client | The relevant AFC client |
| handle | File handle of a previously opened file |
| data | The pointer to the memory region to store the read data |
| length | The number of bytes to read |
| bytes_read | The number of bytes actually read. |

## Return Value

AFC_E_SUCCESS on success or an AFC_E_* error value.

## See Also

* enum [AfcError](../AfcError.md)
* class [AfcClientHandle](../AfcClientHandle.md)
* interface [IAfcApi](../IAfcApi.md)
* namespace [iMobileDevice.Afc](../../iMobileDevice-net.md)

<!-- DO NOT EDIT: generated by xmldocmd for iMobileDevice-net.dll -->

repository: qmfrederik/docsy-perf

repo_name: qmfrederik/docsy-perf


text:
### [CVE-2014-0984](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-0984)
![](https://img.shields.io/static/v1?label=Product&message=n%2Fa&color=blue)
![](https://img.shields.io/static/v1?label=Version&message=n%2Fa&color=blue)
![](https://img.shields.io/static/v1?label=Vulnerability&message=n%2Fa&color=brighgreen)

### Description

The passwordCheck function in SAP Router 721 patch 117, 720 patch 411, 710 patch 029, and earlier terminates validation of a Route Permission Table entry password upon encountering the first incorrect character, which allows remote attackers to obtain passwords via a brute-force attack that relies on timing differences in responses to incorrect password guesses, aka a timing side-channel attack.

### POC

#### Reference
- http://www.coresecurity.com/advisories/sap-router-password-timing-attack
- http://www.exploit-db.com/exploits/32919

#### Github
- https://github.com/martingalloar/martingalloar

repository: trickest/cve

repo_name: trickest/cve


text:
# backup
## Usage ##
backup output_directory directories_to_backup...

### Example ###
backup /mnt/external_drive /home/user/projects /etc

## Disc layout ##
Script creates new directory for every backup with name $(date -Iseconds).
It alsa maintains symlink for recent backup with name last_backup

### Result ###
```
# ll /run/backup
total 28
drwxr-xr-x 3 user users  4096 cze 24 15:40 2017-06-24T13:44:09+02:00
drwxr-xr-x 3 user users  4096 cze 24 15:40 2017-06-24T14:45:32+02:00
drwxr-xr-x 4 user users  4096 cze 24 15:39 2017-06-24T15:39:37+02:00
lrwxrwxrwx 1 user users    25 cze 24 15:39 last_backup -> 2017-06-24T15:39:37+02:00
```

## How it works ##
It's just a simple wrapper over rsync --link-dir=$LAST_BACKUP

repository: canthar/backup

repo_name: canthar/backup