text:
SVAFotate
=========================

Structural Variant Allele Frequency annotate or SVAFotate is a tool for
annotating structural variant VCFs with population level allele frequency
information and other related metrics.

Overview
=========================
Being able to distinguish whether a mutation or variant is common or
rare in the general population is crucial in many genomic analyses, including
structural variant (SV) analysis. More and more large population SV cohorts,
with population level allele frequencies (AFs), are becoming available, but
each consist of primarily rare variants that are unique to the individual
dataset. SVAFotate provides the means to aggregate and combine AF and related
data from multiple population datasets into simple annotations that are
then added to a SV VCF. These annotations are then readily available and
can be used for different filtering schemes or other analyses. SVAFotate
is a command-line tool and provides a variety of annotation options related
to AF metrics.

## Quick Links
[**Installation**](https://github.com/fakedrtom/SVAFotate#installation)

[**Usage**](https://github.com/fakedrtom/SVAFotate#usage)

  * [**annotate**](https://github.com/fakedrtom/SVAFotate#annotate)

   [Minimum Overlap Fraction](https://github.com/fakedrtom/SVAFotate#minimum-overlap-fraction)

   [Sources to Annotate](https://github.com/fakedrtom/SVAFotate#sources-to-annotate)

   [Extra Annotations](https://github.com/fakedrtom/SVAFotate#extra-annotations)

   [Observed SV Coverage](https://github.com/fakedrtom/SVAFotate#observed-sv-coverage)

   [Unique SV Regions](https://github.com/fakedrtom/SVAFotate#unique-sv-regions)

   [SV Size Limit](https://github.com/fakedrtom/SVAFotate#sv-size-limit)

   [Targets BED File](https://github.com/fakedrtom/SVAFotate#targets-bed-file)

   [Use CI boundaries](https://github.com/fakedrtom/SVAFotate#use-ci-boundaries)

   [Change SV Size](https://github.com/fakedrtom/SVAFotate#change-sv-size)

   [CPU Count](https://github.com/fakedrtom/SVAFotate#cpu-count)

  * [**pickle-source**](https://github.com/fakedrtom/SVAFotate#pickle-source)

  * [**custom-annotation**](https://github.com/fakedrtom/SVAFotate#custom-annotation)

[**Acknowledgements**](https://github.com/fakedrtom/SVAFotate#acknowledgements)

Installation
========================
0) Installing Miniconda

- If Miniconda is not installed on your system, install it from [miniconda](https://conda.io/en/latest/miniconda.html)

1) Set up new conda environment

```
$ conda create --name svafotate-env python=3
```

```
$ conda activate svafotate-env
```

2) Install package requirements

```
$ conda install --file https://raw.githubusercontent.com/fakedrtom/SVAFotate/master/requirements.txt
```

3) Install SVAFotate

```
$ pip install git+https://github.com/fakedrtom/SVAFotate.git
```

4) Check that SVAFotate installed Correctly

```
$ svafotate --version

svafotate 0.0.1
```

Usage
========================
There are currently three SVAFotate subcommands: `annotate`,
`pickle-source`, and `custom-annotation`.

```
$ svafotate -h
usage: svafotate [-h] [-v] {annotate,pickle-source,custom-annotation} ...

SVAFotate: Structural Variant Allele Frequency annotator ==================================================

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         Installed version

[sub-commands]:
  {annotate,pickle-source,custom-annotation}
    annotate            Annotate SV VCF File
    pickle-source       Pickle Source Bed
    custom-annotation   Add custom annotation(s) to source annotation file
```

annotate
------------------------
This is the main functionality of SVAFotate and adds AF related metrics to
your SV VCF based on overlapping matches between SVs in an input VCF and SVs
in a provided BED file.

```
Required Arguments:
  -v INPUT VCF, --vcf INPUT VCF
                        Path and/or name of the VCF file to annotate.
  -o OUTPUT VCF, --out OUTPUT VCF
                        Path and/or name of the output VCF file.
			sources [SOURCES TO ANNOTATE [SOURCES TO ANNOTATE ...]]

Requires Only One Argument:
  -b SOURCE BED, --bed SOURCE BED
                        Path and/or name of the combined sources AF bed file (--pickled-source can be used instead of --bed) (NOTE: --bed takes higher priority than --pickled-source).
  -p Pickled Source Data, --pickled-source Pickled Source Data
                        Path and/or name of the pickled source data to use (--bed can be used instead of --pickled-source) (NOTE: if --bed is provided, --pickled-source is ignored).
```

As stated, this requires an input VCF (`-v`) and output VCF name (`-o`). SVAFotate was
developed using SV VCFs derived from the Lumpy SV caller, however it has
been tested on VCFs created by other SV callers. As long as SVTYPE and END
(preferably SVLEN as well) are included in the INFO fields, any SV VCF that
follows expected VCF conventions should be usable with SVAFotate.

SVAFotate also requires a BED file corresponding to population SV data (`-b`) that you
wish to compare the SVs in the input VCF against. A BED file with currently available
large population SV data is provided along with SVAFotate, called:

```
SVAFotate_core_SV_popAFs.GRCh38.bed.gz
```

This file can be found in the [supporting_data](https://github.com/fakedrtom/SVAFotate/tree/master/supporting_data)
folder. The BED file was compiled using information gathered from the publicly available
CCDG, gnomAD, and 1000G SV callsets. AF related data (including HOM_REF, HET,
and HOM_ALT genotype counts, where available), were parsed from these into a single BED
file. CCDG and 1000G data were provided with GRCh38 alignments, but gnomAD data
was generated using GRCh37 alignments. These have been converted to GRCh38 using
[UCSC's command-line liftover tool](https://genome.ucsc.edu/cgi-bin/hgLiftOver). A
different BED file could be used or customized, user-specific data could be added to the
provided BED file, but please note that SVAFotate expects specific columns, their header
names, and their order placement to be present in the BED file. If a different BED
file is used or additional data is added to the provided BED, please ensure that
it follows the same ordering and column information as found in `SVAFotate_core_SV_popAFs.GRCh38.bed.gz`.
As a minimum, it is necessary that any other BED file used or other data added to this provided
BED file include: CHROM, START, END, SVLEN, SVTYPE, SOURCE, SV_ID, and AF.
Please note that all other columns do *not* need to be populated with actual data and where
data is unavailable an 'NA' is recommended.

Thanks to Adam English and the Sedlazeck lab, SV calls from TOPMed have been added to
SVAFotate. These calls can be found the BED file named:

```
TOPMed.GRCH38.bed.gz
```

A serialized pickle object of the BED file can be used in place of the BED file (`-p`).
This may result in faster performance when using the SVAFotate `annotate` subcommand.
Included with SVAFotate is the subcommand [pickle-source](https://github.com/fakedrtom/SVAFotate#pickle-source)
which enables a pickle object of a BED file to be generated and then repeatedly used
with the `annotate` subcommand.

The bare minimum to run the `annotate` subcommand is:

```
svafotate annotate -v in.vcf.gz -o out.vcf -b SVAFotate_core_SV_popAFs.GRCh38.bed.gz
```

The `annotate` subcommand considers each SV in the VCF input individually and identifies
all matching SVs from the BED file. **Matching SVs are defined as SVs from the input VCF
that share overlapping genomic coordinates with SVs provided in the BED file and that are
described as having the same SVTYPE**. Under default settings even a single basepair overlap
can qualify as a matching SV which may result in many matches being rather imprecise (and also
an overabundance of matches). There are options to require more stringent overlap requirements
to help avoid this (please see [Minimum Overlap Fraction](https://github.com/fakedrtom/SVAFotate#minimum-overlap-fraction)
). Once SVAFotate collects all matching SVs, it assesses individual metrics from each matching
SV and creates new population AF related annotations for each SV from the input VCF. By
default, SVAFotate will add the following annotations to the INFO field of the input VCF:

```
Max_AF		 The maximum AF from all matching SVs across all specified data sources found in the provided BED file
Max_Het	      	 The maximum Het count from all matching SVs across all specified data sources found in the provided BED file
Max_HomAlt    	 The maximum HomAlt count from all matching SVs across all specified data sources found in the provided BED file
Max_PopMax_AF	 The maximum PopMax_AF from all matching SVs across all specified data sources found in the provided BED file
```

For each of these annotations the maximum values from all matching SVs is returned.
If there are no matching SVs, each of these annotations will be added with a 0 or 0.0 value.
The following figure provides example input SVs compared to SVs of the same SVTYPE derived
from the CCDG, gnomAD, and 1000G datasets, which are included with the input BED file.
Consider the example SV on the far right which has matches with 4 different SVs of the same
SVTYPE from CCDG, gnomAD, and 1000G in the BED file, and these have reported AFs of
0.01, 0.03, 0.63, and 0.09. In this case the `Max_AF` annotation that is added to the
input SV will be 0.63.

![max_af](https://github.com/fakedrtom/SVAFotate/blob/master/images/max_AF_example_fig.png)

Please note that each of these annotations is determined independently of one another.
That is to say, just because the SV with an AF of 0.63 is returned for the `Max_AF`
annotation, the HET or HOM_ALT genotype counts corresponding to the SV with the AF
of 0.63 are not necessarily going to be returned for the `Max_Het` and `Max_HomAlt`
annotations unless they are also the maximum values from all matching SVs (in this
case they likely would be added given the wide disparity between AFs in all matching SVs).
Also by default an annotation reflecting the number of matches per data source in the
BED file is also added. For example, if CCDG is one of the data sources in the input
BED file, then `CCDG_Count` is added and lists the number of CCDG matching SVs.

#### Matching more complex SVs
As previously mentioned, matching SVs are defined as an SV from the input VCF that shares
an overlap of genomic coordinates with an SV in the BED file, provided that these SVs share
the same SVTYPE. This is fairly straightforward for many SVTYPES, such as, DELs, DUPs,
and INVs. This matching scheme can be more complicted for other SVTYPEs. Insertions
(INSs), for example, are often reported as a single basepair (or 2bp) genomic coordinate
with an accompanying SVLEN that reflects the size of the insertion. INSs are still matched
based on overlapping coordinates, which generally means that INSs from the input VCF only match
when their coordinates are (nearly) the same as those in the BED file (even if SVLENs differ).
Reported SVLENs for INSs are then used if certain options (such as `-f` or `-a best`) are
requested to better refine the matching INSs. Other even more complex SVTYPES may require more
specialized attention. In some of these cases, it may be helpful to include the `-a mis`
parameter which would add annotations regarding overlapping SVs that have different SVTYPEs.
For more information please see the [Extra Annotations](https://github.com/fakedrtom/SVAFotate#extra-annotations)
section.

There are many additional options beyond the defaults of SVAFotate that may result
in improved or more detailed annotations. Some of these are highly recommended for
most uses of SVAFotate. These options are listed here along with explanations:

#### Minimum Overlap Fraction
```
  -f [MINIMUM OVERLAP FRACTION [MINIMUM OVERLAP FRACTION ...]], --minf [MINIMUM OVERLAP FRACTION [MINIMUM OVERLAP FRACTION ...]]
                        A space seperated list of minimum reciprocal overlap fractions required between SVs for each source listed with the `-s` option. If `-s` is not used, only the first minf will be
                        used and will be applied to all sources. minf values must be between 0.0 and 1.0 (Default = 0.001).
```

By default, SVAFotate will consider any overlap of genomic coordinates between SVs from the
input VCF and SVs in the supplied BED file, even those of a single base pair, as matching SVs.
While this represents all possible overlaps, it could result in imprecise matching of SVs
that do not actually reflect similar SV alleles. **It is highly recommended to use the `-f` option
to increase the required amount of reciprocal overlap between putative matching SVs**. The
higher the value of `-f`, the more exact the overlapping match between SVs must be (and
likely the more precise and limited the number of overlapping matches will be). For example,
a `-f` value of 0.9 would require that 90% of the genomic region belonging to an SV from
the input VCF must overlap with at least 90% of the genomic region belonging to an SV in
the BED file in order to be considered a matching SV (provided they have the same SVTYPE).
If this requirement is not met, the SV from the BED file will not be added as a matching SV.
Given the inherent variance of SV breakpoints, it is difficult to presume what the best value
for `-f` is, but a minimum of 0.5 should be a decent starting value to consider.

#### Sources to Annotate
```
  -s [SOURCES TO ANNOTATE [SOURCES TO ANNOTATE ...]], --sources [SOURCES TO ANNOTATE [SOURCES TO ANNOTATE ...]]
                        Space seperated list of data sources to use for annotation. If '-s' is not used, all sources available in the source bed file will be used (Example: ' -s CCDG gnomAD ' ).
```

The required BED file for running the `annotate` subcommand may contain SV data from multiple
datasets. The `SOURCE` column in this BED file refers to the data source for the given SVs. For
example, the supplied `SVAFotate_core_SV_popAFs.GRCh38.bed.gz` file contains SVs from the CCDG,
gnomAD, and 1000G dataset sources. By default SVAFotate will consider all SVs in the BED file for
determining potential matches (and can report metrics specifc to each source). If annotations
based on comparisons with select sources only is desired, the `-s` parameter will reduce the
considered SVs from the BED file to only those belonging to the requested sources. A single
source could be selected or any combination of included sources from the BED file.

#### Extra Annotations
```
  -a [EXTRA ANNOTATIONS [EXTRA ANNOTATIONS ...]], --ann [EXTRA ANNOTATIONS [EXTRA ANNOTATIONS ...]]
                        By default, only the Max_AF, Max_Hets and Max_HomAlt counts, and Max_PopMax_AF are annotated in the output VCF file. `-a` can be used to add additional annotations, with each
                        anntotation seperated by a space (Example ' -a mf best pops ' ). Choices = [all, mf, best, pops, AFR, AMR, EAS, EUR, OTH, SAS, full, mis]
```

A number of optional annotations are available and can be included using the `-a` option. This
option requires one or more of the following choices which each add more annotations as described
here:

***all***

Adds all of the annotations described by each of the other choices available with `-a`.

***mf***

Adds male and female annotations including:

```
Max_Male_AF	    The maximum Male AF from all matching SVs across all specified data sources
Max_Male_Het	    The maximum Male Het count from all matching SVs across all specified data sources
Max_Male_HomAlt	    The maximum Male HomAlt count from all matching SVs across all specified data sources
Max_Female_AF	    The maximum Female AF from all matching SVs across all specified data sources
Max_Female_Het	    The maximum Female Het count from all matching SVs across all specified data sources
Max_Female_HomAlt   The maximum Female HomAlt count from all matching SVs across all specified data sources
```

If used alongside `best`, `pops` or any of the individual population choices, those annotations
will also include male and female specific annotations regarding AF and genotype counts.

***best***

Adds the "best" match for each data source included in the BED file (or as specified)
and creates the following annotations:

```
Best_[data_source]_ID		The SV_ID of the best matching SV for that data source
Best_[data_source]_OFP		The OFP of the best matching SV for that data source
Best_[data_source]_AF	    	The AF from the best SV_ID for that data source
Best_[data_source]_Het    	The Het count from the best SV_ID for that data source
Best_[data_source]_HomAlt 	The HomAlt count from the best SV_ID for that data source
Best_[data_source]_PopMax_AF	The PopMax_AF from the best SV_ID for that data source
```

The "best" match here is defined by the matching SVs with the highest overlap fraction
product (OFP) and considers all overlaps even those that may be filtered out using `-f`
option. Calculating the OFP is straightforward and relies on the genomic size of
overlapping SVs and the amount of overlap that is shared between them.

![calculating_ofp](https://github.com/fakedrtom/SVAFotate/blob/master/images/calculating_ofp.png)

From this example, an overlap fraction is calculated for each SV by dividing the amount
of overlap by the size of each SV, respectively. Then these fractions are multiplied to
create the OFP which will range between 0.0 and 1.0. As illustrated below, high OFP scores
reflect matching SVs that are more identical in terms of both their genomic sizes and the
amount of overlap they share while low OFP scores suggest a disparity in genomic sizes
between matching SVs or a low amount of shared overlap between them (or both a discrepancy
in sizes and low overlap).

![ofp](https://github.com/fakedrtom/SVAFotate/blob/master/images/ofp_range.png)

If used alongside `mf`, `pops` or any of the individual population choices, "best" annotations
will also be included for those annotations.

***pops***

The including BED file `SVAFotate_core_SV_popAFs.GRCh38.bed.gz` contains data from gnomAD and
1000G which both include population specific metrics belonging to the following populations:
AFR, AMR, EAS, EUR, OTH, and SAS. The `pops` choice will add the following annotations for all
populations:

```
Max_[population]_AF	    The maximum population AF from all matching SVs across all specified data sources
Max_[population]_Het	    The maximum population Het count from all matching SVs across all specified data sources
Max_[population]_HomAlt	    The maximum population HomAlt count from all matching SVs across all specified data sources
```

If preferred, individual population designations can be selected instead to add annotations
pertaining to that population only. If used alongside `mf` or `best`, male and female or "best"
annotations will also be added for the designated populations.

***full***

Adds all information available for all matches. This will add the following data source specific
annotations:

```
[data_source]_Matches	Comma-separated list of each SV match with the data source
```

Please note that this annotation will include all information available in the BED file
for all matching SVs. This is the "kitchen sink" annotation for matches and will potentially
add very lengthy annotations to the INFO field. For example, if an SV from the input
VCF is found to have 4 matches with SVs from the gnomAD dataset, all information from
the BED file for all 4 of those matches will be listed.

***mis***

While matching SVs are defined as having the same SVTYPE it is possible to see if any SVs with a
different SVTYPE also overlap with an SV from the input VCF (a mismatched SV) with the `mis`
option. This will add the following annotations:

```
[data_source]_Mismatches		Comma-separated list of the SV_IDs of overlapping SVs from the data source with different SVTYPEs
[data_source]_Mismatches_Count		The number of overlapping SVs from the data source with different SVTYPEs
[data_source]_Mismatch_SVTYPEs		Comma-separated list of the other overlapping SVTYPEs from the data source
```

The same requirements for traditional matching SVs will be applied here. For example, if the `-f`
option is used, mismatched SVs will be required to meet those same thresholds. Additionally, the
following annotations will also be added reflecting the "best" mismatch based on OFP scores:

```
Best_[data_source]_Mismatch_ID		The SV_ID of the best overlapping SV with different SVTYPE from the data source
Best_[data_source]_Mismatch_OFP		The OFP of the best overlapping SV with different SVTYPE from the data source
Best_[data_source]_Mismatch_SVTYPE	The SVTYPE of the best mismatch SV_ID from the data source
Best_[data_source]_Mismatch_AF		The AF of the best mismatch SV_ID from the data source
Best_[data_source]_Mismatch_Het		The Het count of the best mismatch SV_ID from the data source
Best_[data_source]_Mismatch_HomAlt	The HomAlt count of the best mismatch SV_ID from the data source
```

When using the `-a` option, any combination of the above (`mf`, `best`, `pops`, `full`, and
`mis`) choices can be selected, but if `all` is included than all of these annotations will
be added to the VCF.

#### Observed SV Coverage
```
  -c OBSERVED SV COVERAGE, --cov OBSERVED SV COVERAGE
                        Add an annotation reflecting how much of the queried SV genomic space has been previously observed with the same SVTYPE. Uses the data sources listed with -s as the previously
                        observed SVs. Please provide minimum AF to exclude all SVs from data sources with a total AF below that value (must be between 0 and 1.0).
```

To see how much of a given SV in the input VCF has been observed with the same SVTYPE from SVs
in the BED file, the `-c` option is provided. This will add the `SV_Cov` annotation
which reflects the fraction of the genomic coordinates for an SV in the input VCF that
are found to overlap with SVs of the same SVTYPE in the BED file. The following figure provides example
scenarios with all SVs being the same SVTYPE and the resulting `SV_Cov` annotations that would
be added to the given SVs.

![max_af](https://github.com/fakedrtom/SVAFotate/blob/master/images/SV_Cov_example_fig.png)

All overlaps from all given sources in the BED file are considered when calculating `SV_Cov`,
provided that the SVs share the same SVTYPE, but source specific `SV_Cov` annotations are also
added. For example, if `CCDG` is one of the sources in the input BED file (and has not been excluded
using the `-s` parameter), `CCDG_SV_Cov` will be added and reflects the `SV_Cov` calculation
using only SVs from the input BED file that come from `CCDG`. All `SV_Cov` annotations measure
*all* overlaps and not just those restricted by the `-f` option. This parameter does expect
a minimum AF value to be included which restricts SVs from the BED file to only those meeting
that value. In this manner, SVs from the BED file, perhaps very rare ones, can be exlcuded from
the calculation of `SV_Cov`.

#### Unique SV Regions
```
  -u UNIQUE SV REGIONS, --uniq UNIQUE SV REGIONS
                        Generate a file of unique SV regions called 'unique.bed'. These regions reflect genomic space within the queried SV region that have not been previously observed with the same
                        SVTYPE. This will also add an annotation regarding the number of unique regions within a given SV. Please provide minimum AF to exclude all SVs from data sources with a total AF
                        below that value (must be between 0 and 1.0).
```

The opposite of the `-c` option is availble with the `-u` option. This will identify
"unique" regions within a given SV from the input VCF that are defined as sharing no overlap
with any of the SVs of the same SVTYPE in the BED file. This option will create an additional
output file named `unique.bed` which will list the coordinates of these unique regions, their
corresponding SVTYPE, the SV_ID of the SV from which they are found, as well the sample names that
are genotyped as heterozygous (0/1) and homozygous (1/1) for the alternate allele. The following
figure illustrates how these regions are determined in a variety of matching SV scenarios.

![max_af](https://github.com/fakedrtom/SVAFotate/blob/master/images/SV_Uniq_example_fig.png)

Additionally, the `-u` option will also add the `SV_Uniq` annotation to the VCF which represents
the number of unique regions found within a given SV from the input VCF. Similar to the `-c`
option, unique regions do not consider restrictions from the `-f` option and this parameter
also expects a minimum AF value to be provided that can restict the inclusion of SVs from the BED
file.

#### SV Size Limit
```
  -l SV SIZE LIMIT, --lim SV SIZE LIMIT
                        Only include previously observed SVs from data sources with a size less than or equal to this value (only available when using --cov or --uniq).
```

If any SVs in the BED file are exceedingly large, they may have big effects on the determination
of observed SV coverage and unique SV regions. For example, CCDG reports an exceptionally rare
deletion that is over 61Mb in size. This event is likely to overlap with many putative deletions
that likely represent distinct SV events. Considering this may obscure meaningful coverage and
unique region annotations and interpretations, the SV size limit `-l` is available. Without
including a size limit with the `-l` option all of these example events overlapping with the
very large and rare deletion from CCDG would show observed SV coverage of 1.0 and no
unique SV regions regardless of other more precise and potentially more meaningful comparisons.
By including an SV size limit alongside the `-c` or `-u` options, such scenarios can be avoided.

#### Targets BED File
```
  -t TARGETS BED FILE, --target TARGETS BED FILE
                        Path to target regions BED file. Expected format is a tab delimited file listing CHROM START END ID where ID is a genomic region identifier that will be listed as an annotation
                        if an overlap exists between a given SV and the target regions.
```

There may be particular genomic regions where an overlap with any reported SV event would be of
interest. Using the `-t` option and supplying a simple BED file consisting of CHROM, START, END,
and a region identifier will create a `Target_Overlaps` annotation that lists the supplied
region identifier for all overlaps between the SV and the regions in this targets BED file. Regions
of interest may include genes, specific exons, promoters, enhancers, etc. If the `-t` option is
used when the `-u` option has also been invoked, then the resulting `unique.bed` file from the
`-u` option will also include a column corresponding to any overlaps found between the unique
regions and the regions provided by the `-t` option. This column will be populated with the
region identifier found in the BED file used with the `-t` option.

#### Use CI Boundaries
```
  -ci USE CI BOUNDARIES, --ci USE CI BOUNDARIES
                        Expects CIPOS and CIEND to be included in the INFO field of the input VCF (--vcf). If argument is selected, use 'inner' or 'outer' confidence intervals (CIPOS, CIEND) for SV
                        boundaries. Choices = [in, out]
  -ci95 USE CI BOUNDARIES, --ci95 USE CI BOUNDARIES
                        Expects CIPOS95 and CIEND95 to be included in the INFO field of the input VCF (--vcf). If argument is selected, use 'inner' or 'outer' confidence intervals (CIPOS95, CIEND95) for
                        SV boundaries. Choices = [in, out]
```

Lumpy and other SV callers may generate confidence interval metrics for predicted SV breakpoints. To
use these confidence intervals instead of the given START and END coordinates, the `-ci` or `-ci95`
options may be used. There are two choices for each of these parameters: `in` and `out`. The `in` choice
will use the CIPOS and CIEND (or CIPOS95 and CIEND95) that will result in a smaller SV while the `out`
choice will use the intervals resulting in a larger SV. If CIPOS and CIEND or CIPOS95 and CIEND95 are
not present in the INFO field for a given SV in the input VCF these options will not work. Please be
sure to consider the adjustments to SV genomic coordinates when using the `-f` option for requiring
reciprocal overlaps.

#### Change SV Size
```
  -e EMBIGGEN THE SV SIZE, --emb EMBIGGEN THE SV SIZE
                        Increase the size of the SV coordinates in the input VCF (--vcf) by a single integer; Subtract that value from the start and add it to the end of each set of coordinates.
  -r REDUCE THE SV SIZE, --red REDUCE THE SV SIZE
                        Reduce the size of the SV coordinates in the input VCF (--vcf) by a single integer; Add that value to the start and subtract it from the end of each set of coordinates.
```

It may be helpful in some cases to adjust the reported SV breakpoints for the SVs in the
input VCF. The genomic space corresponding to a reported SV may be enlarged using the
`-e` option or reduced using the `-r` option. Please be sure to consider the adjustments to SV
genomic coordinates when using the `-f` option for requiring reciprocal overlaps.

#### CPU Count
```
  --cpu CPU Count       The number of cpus to use for multi-threading (Default = 1).
```

SVAFotate may use additional cpus for multi-threading purposes which can be designated using
the `--cpu` option.

pickle-source
------------------------
Since SVAFotate may be used repeatedly on different SV VCFs with the same BED file, it
may be advantageous to create a pickle object of the BED file to improve SVAFotate's
performance when running the `annotate` subcommand. This is optional. If SVAFotate or
the BED file used with the `annotate` subcommand is seldom needed, creating a pickle object
of that BED file is probably not necessary. Nevertheless, creating a pickle object out
of a BED file is straightforward and can be accomplished with the following command:

```
svafotate pickle-source --bed SVAFotate_core_SV_popAFs.GRCh38.bed.gz --out SVAFotate_core_SV_popAFs.GRCh38.pickle
```

custom-annotation
-----------------------

This is not currently used, but is a placeholder for potential
future developments.

Acknowledgements
========================
Thank you to Michael Cormier for code review, improvements, and contributions.

repository: fakedrtom/SVAFotate

gha_language: Python

repo_name: fakedrtom/SVAFotate


text:
---
layout: post
title: "Position dependent rotation"
category: CTF, Nebula
---

>         **[Nebula 14](http://exploit-exercises.com/nebula/level14) Agenda**: "It encrypts input and writes it to standard output. An encrypted token file is also in that home directory, decrypt it :)"

During the investigation under the hood, it's was easy to recognize position dependent [ROT](https://en.wikipedia.org/wiki/ROT13) (means, not 13 =). In current case, each char's shift is directly proportional to its position in the sequence):
{% highlight sh linenos %}
level14@nebula:/home/flag14$ ./flag14 -e
111111111
123456789
level14@nebula:/home/flag14$ ./flag14 -e
aaaaaaaaa
abcdefghi
{% endhighlight %}
Thus it was easy to write appropriate decoder:
{% highlight python linenos %}
#!/usr/bin/python
import sys

def decode(string):
    if string is not None:
        result = ""
        for i in range(0, len(string)):
            result += chr(ord(string[i]) - i)
        return result

def readContentOf(filename):
    try:
        with open (filename, "r") as tokenFile:
            return tokenFile.read().replace('\n', '')
    except IOError:
        sys.stderr.write('Problem reading:' + filename)

def main():
    if (len(sys.argv) >= 2) :
        text = readContentOf(sys.argv[1])
        print "Your token is: ", decode(text)
    else :
        print "Please specify the toke_file"

if __name__ == "__main__":
     main()
{% endhighlight %}
And obtain the token:
{% highlight sh linenos %}
level14@nebula:~$ ./decoder.py /home/flag14/token
Your token is:  8457c118-887c-4e40-a5a6-33a25353165
{% endhighlight %}

Resources:

 * [https://developers.google.com/edu/python/](https://developers.google.com/edu/python/)

branch_name: refs/heads/master

extension: md

license_type: permissive


text:
[![Build Status](https://circleci.com/gh/StackStorm-Exchange/stackstorm-cloudflare.svg?style=shield&circle-token=:circle-token)](https://circleci.com/gh/StackStorm-Exchange/stackstorm-cloudflare) [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)

# Cloudflare Pack

A pack for interacting with the [Cloudflare API](https://api.cloudflare.com).

## Configuration

Copy the example configuration in [cloudflare.yaml.example](./cloudflare.yaml.example)
to `/opt/stackstorm/configs/cloudflare.yaml` and edit as required.

``` shell
cp /opt/stackstorm/packs/cloudflare/cloudflare.yaml.example /opt/stackstorm/configs/cloudflare.yaml
```

It should contain:

* ``api_key`` - Cloudflare API key, details [here](https://support.cloudflare.com/hc/en-us/articles/200167836-Where-do-I-find-my-Cloudflare-API-key-)
* ``api_email`` - Cloudflare API email address (username)

You can also use dynamic values from the datastore. See the
[docs](https://docs.stackstorm.com/reference/pack_configs.html) for more info.

**Note** : When modifying the configuration in `/opt/stackstorm/configs/` please
           remember to tell StackStorm to load these new values by running
           `st2ctl reload --register-configs`

## Actions

### get_ips

Returns the current IPs for the Cloudflare service, does not require an API key.

### get_zones

Returns a list of the current DNS Zones in your Cloudflare account. Requires an API key.

### get_zone_dns_records

Returns a list of the DNS Records in a given Zone. Requires an API key.


text:
# CircleCI Reality Check
This repository is used to check for a proper installation of CircleCI Server. It implements a sample app that validates some basic CircleCI features in parallel workflows.

![](misc/reality_check.png)

---

## Installation
To install and run reality check on your CircleCI Server installation, follow these steps.

1. Fork this repository to the GitHub environment tied to your server installation. [See here for more information](https://support.circleci.com/hc/en-us/articles/360011235534).

2. Set up your repository project in your CircleCI Server installation.

3. Generate a **personal access token - not a project token**, and add it as an environment variable in your project with the name `CIRCLE_TOKEN`.

4. Determine the base URL of your install, including protocol, and remove the trailing slash and "app" subdomain if present. Add it as an environment variable in your project with the name `CIRCLE_HOSTNAME`.

    > *Note*: As an example, if your dashboard is accessible at `https://app.server.example.com/dashboard/` your `CIRCLE_HOSTNAME` is `https://server.example.com`

5. Set the environment variable in your project with the name `CIRCLE_CLOUD_PROVIDER` as either `gcp`, `aws`, or `other`. depending on your installation.

6. You can __optionally__ set `CIRCLE_WINDOWS_EXECUTOR` to `true` in order to run a set of verification jobs for the windows execution environment. This requires a supported CircleCI Windows image, see instructions in [CircleCI-Public/circleci-server-windows-image-builder](https://github.com/CircleCI-Public/circleci-server-windows-image-builder).

7. Configure the following contexts and keys (their values can be anything). Docs on how to set up contexts [can be found here](https://circleci.com/docs/2.0/contexts/).

Context Name       | Environment Variable Key Name  | Value
-------------------|------------------------------- |-----------------------------
`org-global`       | `CONTEXT_END_TO_END_TEST_VAR`  | `1`
`individual-local` | `MULTI_CONTEXT_END_TO_END_VAR` | `1`

### Example AWS Server Installation
```bash
# Project Environment Variables
CIRCLE_TOKEN=123456789-personal-access-token
CIRCLE_HOSTNAME=https://aws-server-install.example.com
CIRCLE_CLOUD_PROVIDER=aws
CIRCLE_WINDOWS_EXECUTOR=true # Optional, only if using a Windowws AMI

# org-global context environment variables
CONTEXT_END_TO_END_TEST_VAR=1

# individual-local context environment variables
MULTI_CONTEXT_END_TO_END_VAR=1
```

### Example GCP Server Installation
```bash
# Project Environment Variables
CIRCLE_TOKEN=123456789-personal-access-token
CIRCLE_HOSTNAME=https://gcp-server-install.example.com
CIRCLE_CLOUD_PROVIDER=gcp
CIRCLE_WINDOWS_EXECUTOR=true # Optional, only if using a Windows AMI

# org-global context environment variables
CONTEXT_END_TO_END_TEST_VAR=1

# individual-local context environment variables
MULTI_CONTEXT_END_TO_END_VAR=1
```

---

## Running Reality Check
Once installed, to run reality check, make any commit to the default branch of your forked repository. If installed correctly, your CircleCI Server Installation will pick up the commit and build it in parallel workflows.

```bash
git commit --allow-empty -m "Trigger Build, Empty commit" && git push
```

---

## Reality Check Workflows
Descriptions of the three workflows follow.

## `resource_class` workflow

Tests all known `resource_class` options—queries the CircleCI API to verify that jobs ran with the requested resources.

### Prerequisites
- Your Nomad client instances must be large enough to accommodate these resource classes — see our [Configuration Reference](https://circleci.com/docs/2.0/configuration-reference/#resource_class) for details on the available resource classes and [our server docs](https://circleci.com/docs/2.0/nomad/#auto-scaling-policy-best-practices) about sizing your Nomad client instances.
- The base URL of your CircleCI installation (e.g. https://circleci.com) must be specified via a `CIRCLE_HOSTNAME` project environment variable
- A personal API token (see `CIRCLE_HOSTNAME/account/api` URL endpoint) must be stored as a `CIRCLE_TOKEN` project environment variable

## VM service workflow
Tests the functionality  of the [`machine` executor](https://circleci.com/docs/2.0/executor-types/#using-machine), [Remote Docker Environment](https://circleci.com/docs/2.0/building-docker-images), and [Docker Layer Caching](https://circleci.com/docs/2.0/docker-layer-caching).

## Features workflow
- Tests ability to save and restore [caches](circleci.com/docs/2.0/caching)
- Tests writing to and reading from [workspaces](https://circleci.com/docs/2.0/workflows/#using-workspaces-to-share-data-among-jobs)
- Tests the default `org-global` [context](https://circleci.com/docs/2.0/contexts) (*NOTE:* needs a key called `CONTEXT_END_TO_END_TEST_VAR` to exist in a context called `org-global`)
- Tests multiple contexts (*NOTE:* needs a key called `MULTI_CONTEXT_END_TO_END_VAR` to exist in a context called `individual-local`)
- Tests upload/storage of [artifacts](https://circleci.com/docs/2.0/artifacts) and [test results](https://circleci.com/docs/2.0/collect-test-data)

## GCP Jobs Workflow
Tests android machine images on Google Cloud Platform. First checks if the environment is configured for GCP with the CIRCLE_CLOUD_PROVIDER environment variable.

### Prerequisites
You will need to have your CIRCLE_TOKEN environment variable set to a personal access token, and will need your CIRCLE_CLOUD_PROVIDER set to `gcp` in lower-case. You must also have nomad clients configured, and your vm-service configured.

## AWS Jobs Workflow
Tests ARM based images on AWS. First checks if the environment is configured for AWS with the CIRCLE_CLOUD_PROVIDER environment variable.

### Prerequisites
You will need to have your CIRCLE_TOKEN environment variable set to a personal access token, and will need your CIRCLE_CLOUD_PROVIDER set to `aws` in lower-case. You must also have nomad clients configured, and your vm-service configured.

---

## Contributing
If you have more ideas for things that should tested, please submit a PR against the open-source repository on GitHub where this project is maintained: <https://github.com/circleci/realitycheck>.
See the current CI status of the main repo at <https://circleci.com/gh/circleci/workflows/realitycheck>.
View the LICENSE file in this repository for licensing information.

license: MIT

repository: circleci/realitycheck

gha_license_id: MIT

repo_name: circleci/realitycheck


text:
# Open Source Spell Checker
This is a free client-side spell checking library written in JavaScript.

- It is... Simple, Lightweight, Customizable, Fast, Client-side, Free

- It is **not**... Perfect, Extremely Accurate, Neural Network-based

# How does a Spell Checker work?
A spell checker must be configured with a valid word list.  Given an input string, the spell checker recommends words from the valid word list.

Our spell checker also returns a score between zero and one with each recommended word.  This score is intended to measure the similarity in spelling between the input string and the recommended word.  A higher score means that the input string and the recommended word have a similar spelling while a lower score means that they do not have a similar spelling.

# Getting Started
**Step 1:** Define your valid word list.  This list should contain all (if possible) of the valid words or strings for your application.

In our example, we define a valid word list with 30,000 English language words.  We define our list in a separate file because it is large.  We define our list in a somewhat strange looking way to obtain a performance benefit.

**Step 2:** Include the spellchecker.js file in the body of your html document.  To do this, you can add the following line of code: <script src="spellchecker.js"></script>

**Step 3:** Set the valid word list using the set_valid_word_list method.  To do this, you can use the following line of JavaScript code: set_valid_word_list(word_list);

**Step 4:** Call the find_similar method on your input string.  To do this, you can add the following line of JavaScript code: find_similar(input_string, threshold);

The threshold is a minimum score requirement.  You can set this to any value between 0 and 1.  I would recommend setting it to 0.7 for English language words.

The find_similar method returns a pair consisting of a list of words and a list of associated scores between 0 and 1.

# Demo Website
Link: http://michaelwehar.com/spellchecker/

# License
- MIT

repository: JustForkin/Open-Source-Spell-Checker

repo_name: JustForkin/Open-Source-Spell-Checker


text:
[![Build Status](https://travis-ci.org/cuplv/droidel.svg?branch=master)](https://travis-ci.org/cuplv/droidel)

Droidel
=======

Droidel is a model of the Android framework that simplifies static analysis of Android apps. It works by examining app code and using it to explicate tricky uses of reflection in the Android framework. The result is a transformed app with a single entrypoint ready to be analyzed by any Java program analysis framework ([WALA](http://wala.sourceforge.net/wiki/index.php/Main_Page), [Soot](http://www.sable.mcgill.ca/soot/), [Chord](http://pag.gatech.edu/chord), etc.). See our SOAP 2015 [paper](http://www.cs.colorado.edu/~bec/papers/droidel-soap15.pdf) for a full explanation of the Droidel approach.

What Android features does Droidel handle?
------------------------------------------
(1) Framework-created types.
Android is an event-driven system whose primary mechanism for control flow is the invocation of application-overridden callback methods by the framework. The framework code that invokes callbacks is complicated and uses reflection heavily. Droidel understands what application types the framework will instantiate reflectively. It uses this information to inject stubs into the framework that explicate what types are allocated via reflection.

(2) Harness generation.
Droidel generates a harness that can be used as a single entrypoint for an Android app. The harness is a slightly modified version of the framework's `ActivityThread.main` method. The important changes are that the modified `main` calls Droidel's stubs for framework-created types to de-obfuscate reflection, and it adds all possible externally generated events to the main `Looper` of the app.

(3) Layout inflation.
Most Android applications define their GUI structure in specialized layout XML files. At runtime, the Android framework *inflates* the GUI by parsing the XML and instantiating each XML-declared element via reflection. Applications can then look up and manipulate the GUI elements via methods such as [`findViewById`](http://developer.android.com/reference/android/app/Activity.html#findViewById(int)). The inflation process and the correspond look up methods are also opaque to most static analyzers. Droidel parses the layout XML to determine the element types that may be allocated during inflation and their id's. Droidel then generates Java bytecodes stubs that explicate the allocations performed during layout inflation and injects them into the framework code.

(4) XML-registered callback invocation.
When callbacks are registered in the layout XML via the `onClick` attribute, Droidel sees this and will generate stubs to make sure the method referred to by the `onClick` attribute gets invoked.

Installation
------------
Droidel has been tested on JDK 7. There are known issues with JDK 8.

Installing Droidel requires Scala 2.10.2 or later, sbt, maven, and ant. Droidel depends on [WALA](https://github.com/wala/WALA) and it also uses the [JPhantom](http://cgi.di.uoa.gr/~smaragd/jphantom-oopsla13.pdf) [tool](https://github.com/gbalats/jphantom). To install Droidel, run the following from Droidel's top-level directory:

(1) Download and build unmanaged dependencies using the provided script:

    cd lib
    ./install_deps.sh

(2) Build Droidel (from the droidel/ directory, do):

    sbt compile

Setting up an Android framework JAR
-----------------------------------
In order to run Droidel, you first need to generate a JAR file for the Android framework that has been injected with Droidel's stub interfaces. Do this by moving the Android JAR you want to use into the `stubs` directory and running `compile_stubs.sh <android_jar>` in the `stubs` directory. The resulting injected JAR will be places in `stubs/out/droidel_<android_jar>`.

Droidel has been tested using the [Android 4.4.2](http://repository.grepcode.com/java/ext/com/google/android/android/4.4.2_r1/android-4.4.2_r1.jar) JAR. Other versions of Android may require slight adjustments to the stubs in order to compile.

    cd stubs
    wget http://repository.grepcode.com/java/ext/com/google/android/android/4.4.2_r1/android-4.4.2_r1.jar
    ./compile_stubs.sh android-4.4.2_r1.jar

Converting Directory Structure of Android Application
---------------------------------------------
The script lib/gradle-to-droidel.sh is used to convert the directory structure from an Android Gradle project to what Droidel expects.  After the script is built with a command such as:

    ./gradlew assembleDebug

The script can be run as

    sh lib/gradle-to-droidel.sh [path to project] [build (eg debug)] [output dir]

The output directory can then be used as a normal project would in droidel.

Running Droidel
---------------

Droidel takes two inputs: APP, a path to an APK file or the top level directory of an Android application, and ANDROID_JAR, a path to a JAR containing the desired version of the Android library. Given an application directory as input, Droidel expects the directory for the application to be organized as follows:

<pre>
APP
+-- AndroidManifest.XML
+-- bin/
|   +-- classes/
|       +-- bytecodes
+-- res/
|   +-- layout/
|       +-- layout XML files
+-- libs/
|   +-- library JARs, if any
</pre>

If Droidel is given an APK, it decodes the application manifest and resources using [apktool](http://code.google.com/p/android-apktool/) and decompiles the DEX bytecodes to Java bytecodes using [dex2jar](http://code.google.com/p/dex2jar/). We plan to add support for decompiling using [Dare](http://siis.cse.psu.edu/dare/) in the near future. Droidel also organizes the decoded APK into the directory format outlined above.

Run Droidel using:

	./droidel.sh -app <APP> -android_jar <JAR> [OPTIONS]

or

	sbt "run -app <APP> -android_jar <JAR>"

The Android JAR passed here should be the one you generated using the process described in the "Setting up an Android framework JAR" section. Droidel will run JPhantom on the app and generate a harness and specialized stubs. The original bytecodes of the app will remain unchanged; Droidel's output is placed in `<APP>/bin/droidel_classes`. This directory will also contain the bytecodes for the Android JAR specified on the command line and the bytecodes of any libraries in the `<APP>/libs` directory.

The most important output is Droidel's harness class; this is placed in `<APP>/bin/droidel_classes/generatedHarness/GeneratedAndroidHarness.class`. To use Droidel in a static analysis framework such as WALA or Soot, simply load all of the code in `<APP>/bin/droidel_classes` into the analyzer and use `GeneratedAndroidHarness.androidMain()` as the entrypoint for analysis. Because `droidel_classes` contains all of the Android library bytecodes and bytecodes of all other libraries the app depends on, there should be no need to load any additional code other than the core Java libraries. For an example of how to build a call graph using Droidel's output in WALA, see how [`AndroidCGBuilder`](https://github.com/cuplv/droidel/blob/master/src/main/scala/edu/colorado/droidel/driver/AndroidCGBuilder.scala) is used in the regression [tests](https://github.com/cuplv/droidel/blob/master/src/test/scala/Regression.scala).

Droidel generates its harness and stubs at the Java source level and then compiles them against the application, its libraries, and the Android framework. For convenience, it preserves these artifacts so that they be be manually inspected and modified/recompiled if necessary. The harness source code is located at `<APP>/bin/droidel_classes/generatedharness/GeneratedAndroidHarness.java`, and the stub source code is located in `<APP>/bin/droidel_classes/generatedstubs`.

Known limitations
-----------------
Creating a reasonable framework model for Android is difficult, and Droidel is not perfect. A few known limitations of Droidel are:

(1) Currently, Droidel's harness is designed for flow-insensitive analysis. It does not faithfully model the invocation order of callbacks in the Android framework, and thus would not be suitable for direct use by a flow-sensitive analysis. We hope to add support for flow-sensitive harness generation in the future.

(2) There are other Android framework methods that use reflection under the hood that we also do not (yet) handle.

Troubleshooting
---------------
Problem: Droidel fails with "Couldn't compile stub file" or "Couldn't compile harness" message.
Solution: Make sure that you are using the appropriate version of the Android JAR for your target application. Check the android:minSdkVersion and/or android:targetSdkVersion in AndroidManifest.xml to see what version of the framework is expected. Also, make sure that you have followed the instructions in the "Setting up an Android framework JAR" section. If all else fails, try manually fixing the compiler errors and re-compiling the stubs and/or harness.

Problem: Droidel fails with "com.ibm.wala.shrikeCT.InvalidClassFileException" error message.
Solution: Droidel uses WALA and Shrike, which cannot currently parse bytecodes produced by the Java 8 compiler. Try switching your default Java version to Java 7 or earlier.

Contributions
-------------
We welcome contributions to Droidel -- please do not hesitate to report an issue, send a pull request, or ask to be added as a contributor. Send questions and comments about Droidel to<EMAIL_ADDRESS>

Acknowledgments
----------------
We are indebted to the creators of [FlowDroid](http://sseblog.ec-spride.de/tools/flowdroid/) for both their list of callback classes in the Android framework and their list of lifecycle methods for the core Android lifecycle types (Activity, Service, etc.).

This material is based on research sponsored in part by DARPA under agreement number FA8750-14-2-0039, the National Science Foundation under CAREER grant number CCF-1055066, and a Facebook Graduate Fellowship. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon.

repository: ftc/droidel

repo_name: ftc/droidel


text:
---
name: Known Issues
sort: 2
---

# Known Issues

This sheet shows the known issues with latest **release(v0.3.1)** version, be aware of them will reduce exceptions and improve your user experience. Some of them might be fixed in up-to-date code. See [Change Log](change_log.md) for getting up-to-date information about what issue has been fixed since last release.

Anything that is related to some version(e.g. `v0.4.0`) is not a promise but a early stage plan.

#### Issue

We're going to add a new issue-user relation table for recording statistic information of an issue and a user in a repository in `v0.4.0`, so the current statistic system of issue will be replaced, which program still works with data that are created before `v0.4.0` but has wrong statistic information.

### Repository

#### Tag

- Panic when Tag name contains `/` [#101](https://github.com/gogits/gogs/issues/101).

### Git

#### `git push` with error `cannot find parent commit of XXX`

As we've testes so far, this error only occurs when your repository was cloned by something like `git clone --depth=XX` which specified the depth of clone.

### UI

#### Repository viewer(URL: `/:username/:reponame`)

- SSH/HTTPS switch and one-click copy button doesn't work under Firefox

repository: jazzthinking/docs

repo_name: jazzthinking/docs


text:
# Zones
A simple program for fast tilting electron-beam sensitive crystals to zone axes.

To start with, users should download and run the latest 'Zones.exe' in a windows system.

'example_experiment.rar' contains necessary data files of the example shown in the manual. The ‘LTA_inputfile.txt' is an important hkl file, and it contains structure information of the sample.

'experiment_data.txt' records the double-tilt holder position of each pattern.
'image4.tif' is the SAED pattern taken at the beginning of the experiment, and shall be determined as [113] zone by the program.
'image7.tif' is the pattern of tilting to the [001] zone based on the calculation of the program.
'image8.tif' is a slightly tilted version of image7 by the Laue circle refinement function of the program.

For more information, please download and read the 'manual for Zones'.


text:
---
id: fast-components-msft.badgeappearance
title: BadgeAppearance type
hide_title: true
---
<!-- Do not edit this file. It is automatically generated by API Documenter. -->

[@microsoft/fast-components-msft](./fast-components-msft.md) > [BadgeAppearance](./fast-components-msft.badgeappearance.md)

## BadgeAppearance type

<b>Signature:</b>

```typescript
export declare type BadgeAppearance = "accent" | "lightweight" | "neutral" | string;
```

repository: edbzn/fast-dna

repo_name: edbzn/fast-dna


text:
# PDXTechHub - Front-End Project for Code Fellows 301

Search for the most recent web developer jobs in Portland and see them displayed on a map. Uses a simple express server, the Indeed and Google Map APIs to get data, localStorage for data persistence, and Handlebars templating to render views.  The code is organized according to the MVC architecture.

## Getting Started

Fork the PDXTechHub repo.

Clone your forked repo.

Set your remote upstream to the original repo.

## Built With

* [Express](https://expressjs.com) - Node.js web application framework.
* [express-request-proxy](https://www.npmjs.com/package/express-request-proxy) - HTTP request proxy server for Express.
* [Handlebars](handlebarsjs.com) - Templating engine used to render views.
* [Page.js](https://visionmedia.github.io/page.js/) - Micro client-side router inspired by the Express router.

## Versioning

express - 4.13.3

express-request-proxy - 2.0.0

## Authors

[Rita Alfonso](https://github.com/alfonsotech)

[Mark Greenwood](https://github.com/markgreenwood)

[mizutombo](https://github.com/mizutombo)

See also the list of [contributors](https://github.com/markgreenwood/PDXTechHub/graphs/contributors) who participated in this project.

## License

This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details

repository: alfonsotech/PDXTechHub

repo_name: alfonsotech/PDXTechHub


text:
---
type: rule
archivedreason:
title: Control Choice - Do you use checkboxes instead of multi-select dropdowns?
guid: acf80273-edd5-462a-a629-b47f8a00a5aa
uri: control-choice-do-you-use-checked-list-boxes-instead-of-multi-select-list-boxes
created: 2012-11-27T08:53:22.0000000Z
authors:
- title: Adam Cogan
  url: https://ssw.com.au/people/adam-cogan
related: []
redirects: []

---

For multiple items selection, **multi-select dropdown listboxes** have a number of behavioral quirks that make it difficult for users to get used to them:

* They require users to know that you select more than one entry by holding down the **Ctrl** key
* They lose all selections if you click in the wrong place
* You can't tell if a Listbox is single-select or multi-select at first glance

<!--endintro-->

::: bad
![Figure: Bad Example - ListBoxes are impractical for multiple choices](multiple-select-dropdown-bad.jpg)
:::

**Note:** It is OK and recommended to use dropdown lists for single item selection, when there is no more than 10 options.

**Checked Listboxes** are the ideal alternative for multiple items selections. They're much more pleasant to use and are a good deal more intuitive - compare to the list above. Checked Listboxes tell users immediately that they have the ability choose multiple options.

* In ASP.NET, use **System.Web.UI.WebControls.CheckBoxList**. If you're having problems with there being too many items in the list, use a **scrolling div**
* In Windows Forms, use **System.Windows.Forms.CheckedListBox**

::: good
![Figure: Good Example - The beauty of the CheckListBox for multiple choices](multiple-select-checkbox-good.jpg)
:::

We have a program called  [SSW Code Auditor](https://www.ssw.com.au/ssw/codeauditor/) to check for this rule.

branch_name: refs/heads/main

gha_language: TSQL


text:
#generator-sds-angular

>Yeoman Generator for Smart Data Systems

This generator follows the [Angular Best Practice Guidelines for Project Structure](https://github.com/johnpapa/angularjs-styleguide).

Getting Started
-------------

Prerequisites: Node, Grunt, Yeoman, and Bower.  Once Node is installed, do:

    npm install -g grunt-cli yo bower

Next, install this generator:

    npm install -g generator-sds-angular

To create a project:

    mkdir MyNewAwesomeApp
    cd MyNewAwesomeApp
    yo sds-angular

Grunt Tasks
-------------

Now that the project is created, you have 3 simple Grunt commands available:

    grunt serve   #This will run a development server with watch & livereload enabled.
    grunt test    #Run local unit tests.
    grunt build   #Places a fully optimized (minified, concatenated, and more) in /dist

When `grunt serve` is running, any changed javascript files will be linted using JSHint as well as have their appropriate unit tests executed.  Only the unit tests that correspond to the changed file will be run.  This allows for an efficient test driven workflow.

Yeoman Subgenerators
-------------

There are a set of subgenerators to initialize empty Angular components.  Each of these generators will:

* Create one or more skeleton files (javascript, LESS, html, spec etc) for the component type.
* Update index.html and add the necessary `script` tags.
* Update app.less and add the @import as needed.
* For partials, update the app.js, adding the necessary route call if a route was entered in the generator prompts.

There are generators for `directive`,`partial`,`service`, `filter`, `module`, and `modal`.

Running a generator:

    yo sds-angular:directive my-awesome-directive
    yo sds-angular:partial my-partial
    yo sds-angular:service my-service
    yo sds-angular:filter my-filter
    yo sds-angular:module my-module
    yo sds-angular:modal my-modal

The name paramater passed (i.e. 'my-awesome-directive') will be used as the file names.  The generators will derive appropriate class names from this parameter (ex. 'my-awesome-directive' will convert to a class name of 'MyAwesomeDirective').  Each sub-generator will ask for the folder in which to create the new skeleton files.  You may override the default folder for each sub-generator in the `.yo-rc.json` file.

The modal subgenerator is a convenient shortcut to create partials that work as modals for Bootstrap v3.1 and Angular-UI-Bootstrap v0.10 (both come preconfigured with this generator).  If you choose not to use either of these libraries, simply don't use the modal subgenerator.

Subgenerators are also customizable.  Please read [CUSTOMIZING.md](CUSTOMIZING.md) for details.

Submodules
-------------

Submodules allow you to more explicitly separate parts of your application.  Use the `yo sds-angular:module my-module` command and specify a new subdirectory to place the module into.  Once you've created a submodule, running other subgenerators will now prompt you to select the module in which to place the new component.

Preconfigured Libraries
-------------

The new app will have a handful of preconfigured libraries included.  This includes Angular 1.2, Bootstrap 3, AngularUI Bootstrap, AngularUI Utils, FontAwesome 4, JQuery 2, Underscore 1.5, LESS 1.6, and Moment 2.5.  You may of course add to or remove any of these libraries.  But the work to integrate them into the app and into the build process has already been done for you.


text:
# CHANGELOG

## 0.3.1 (2020-12-23)

### Changed

* Package Information (e.g. author, name etc)
* Fixed "Error: Call to undefined method Symfony\Component\BrowserKit\Response::getStatus()" using PR https://github.com/raulr/google-play-scraper/pull/36

### Removed

* Dropped Support for PHP 5.6 & PHP 7.0

## 0.3.0 (2019-06-16)

### Changed

* Update scraping to the current markup in Play Store website.

## 0.2.0 (2016-11-10)

### Added

* `getSearch` and `getDetailSearch` methods.

### Changed

* Convert schemaless URLs into absolute URLs.

## 0.1.1 (2016-04-09)

### Fixed

* Fix throwing InvalidArgumentException on `Scraper::getApp()` when app page contains a `NULL` byte.
* Remove duplicate license field on composer.json.

## 0.1.0 (2016-01-04)

* Initial release.


text:
A value that is specified for a `KeyframeMarker`. Whenever the signal created from [AnimationTrack.GetMarkerReachedSignal](https://developer.roblox.com/api-reference/function/AnimationTrack/GetMarkerReachedSignal) gets fired, this value will be passed into the connected function.

## See also

 - `Keyframe`,  holds the [Poses](https://developer.roblox.com/api-reference/class/Pose) applied to joints in a `Model` at a given point of time in an animation

 - `AnimationTrack`, controls the playback of an animation on a `Humanoid` or `AnimationController`

repository: noahwillcrow/types

gha_language: TypeScript

repo_name: noahwillcrow/types


text:
### Code Style Check
---
Code style violations detected in the following files:
* `spec/fixtures/BadViewController.m`

Execute one of the following actions and commit again:
1. Run `clang-format` on the offending files
2. Apply the suggested patches with `git apply patch`.

#### spec/fixtures/BadViewController.m
```diff
--- spec/fixtures/BadViewController.m
+++ spec/fixtures/BadViewController.m
@@ -1,9 +1,10 @@
-@interface ViewController (  ) @end
+@interface ViewController ()
+@end

 @implementation ViewController
--(void ) viewDidLoad {
-    [super viewDidLoad];
-    NSLog(  @"perfect change!")   ;
+- (void)viewDidLoad {
+  [super viewDidLoad];
+  NSLog(@"perfect change!");
 }

 @end

```

gha_language: Ruby


text:
# NCD Twilio simulator

This project is an IVR stress test tool for Surveda (related issue: [instedd/surveda#1793](https://github.com/instedd/surveda/issues/1793)).

The main difference between this project and its ancestor is that instead of a background service to simulate SMS answers, it will take the shape of a Web API to Simulate phone call answers.

It mocks the following Twilio endpoints used by Verboice:

1. [Fetch an IncomingPhoneNumber resource](https://www.twilio.com/docs/phone-numbers/api/incomingphonenumber-resource#fetch-an-incomingphonenumber-resource)
2. [Initiate an outbound call with Twilio](https://www.twilio.com/docs/voice/make-calls#initiate-an-outbound-call-with-twilio)

## Usage (dev path)

For a local development, we assume that [dockerdev](https://github.com/manastech/dockerdev)
is installed and running.

1. Run `twiliosim`:

   ```bash
   docker compose up web
   ```

2. Your local endpoint is <http://web.ncd_twilio_simulator.lvh.me/>

3. When running against Verboice in staging or somewhere on the Internet, you'll
   have to expose your local to the Internet. You can use `ngrok` to do that:

   ```bash
   docker compose -d up ngrok
   ```

   Your public endpoint is listed on <http://ngrok.ncd_twilio_simulator.lvh.me/>.

   Alternatively, when running a Verboice instance on your local, you'll have to
   configure Verboice. Open the `broker/verboice.config` file and change the
   `twilio_callback_url` to `http://broker.verboice.lvh.me:8080`:

   ```erl
   {twilio_callback_url, "http://broker.verboice.lvh.me:8080/"},
   ```

4. Follow the end-user usage path (starting at point 6) to set up your channel
   in Verboice.

## Usage (end-user path)

1. Pick a port number (in this example: 8081)

2. Run:

   ```bash
   docker run --rm -t -i -p 8081:80 instedd/twiliosim
   ```

3. Your endpoint is: <http://localhost:8081/>

4. Expose your endpoint publicly using [ngrok](https://ngrok.com/docs#getting-started-expose)
   or similar if needed.

   ```bash
   ngrok http 8081
   ```

5. Your public endpoint is listed <http://<IP_ADDRESS>:4040/>.

6. Copy your `https` URL:

   ![ngrok URL](https://user-images.githubusercontent.com/39921597/99557501-88692e00-29a1-11eb-92c5-d27be72885e4.png)

7. Go to your Verboice instance and create a new Twilio Channel using your
   public (or local) endpoint as your channel base URL:

   ![channel base URL](https://user-images.githubusercontent.com/39921597/99560107-442b5d00-29a4-11eb-9f74-e105961b22d5.png)

## Behaviour

This App was born inspired by the [NCD local gateway simulator](https://github.com/instedd/ncd_local_gateway_simulator), a SMS stress test tool for Surveda. So its expected behaviour is similar to [this one](https://github.com/instedd/ncd_local_gateway_simulator#behaviour).

Every Surveda text-to-speech (TTS) question is translated by Verboice to a TTS message. For each message received from Verboice, the following rules apply:

| Input | Reply |
|-|-|
| `#hangup` | hang up |
| `#oneof:N,M,O` | one of the the following numbers: `N`, `M` or `O` |
| `#numeric:N-M` | a number between `N` and `M` (including them) |

If none of these rules apply, the message is ignored.

For creating questionnaire steps these rules should be in the IVR message, make sure the responses are configured accordingly.

The following environment variables controls other aspects of the behavior:

| Variable | Default | Description |
|-|-|-|
| `NO_REPLY_PERCENT` | 0.0 | Percent of respondents that don't reply (doesn't reply to the call, or sends `digits=timeout` on gather when timeout is reached) |
| `DELAY_REPLY_MIN_SECONDS` | 1 | Minimum delay in seconds to reply |
| `DELAY_REPLY_MAX_SECONDS` | 5 | Maximum delay in seconds to reply |
| `INCORRECT_REPLY_PERCENT` | 0.0 | Percent of respondents that reply an incorrect answer |
| `MAX_INCORRECT_REPLY_VALUE` | 99 | Maximum value replied as an incorrect answer |
| `STICKY_RESPONDENTS` | true | If true, once a respondent replies, it will always reply during the same call |
| `DELAY_HANG_UP_SECONDS` | 5 | Delay in seconds to hang up (applying to the `#hangup` rule) |

gha_language: Crystal


text:
---
title: "Why Commits to Forks on GitHub Don't Apply to Contributions"
date: 2016-11-11T13:35:56-06:00
layout: post
tags: [technology, software, github]
---

# Why Commits to Forks on GitHub Don't Apply to Contributions

Recently, I noticed that contributions I made to forks on GitHub were not being counted toward the contribution tracker on my profile's page. I was curious to know why, as I felt like those contributions were just as good as any other.

After a little research, I found [this article](https://help.github.com/articles/why-are-my-contributions-not-showing-up-on-my-profile/) on GitHub's support site. In it, they say:

> Commits made in a fork will not count toward your contributions. To make them count, you must do one of the following:
>
> - Open a pull request to have your changes merged into the parent repository.
> - To detach the fork and turn it into a standalone repository on GitHub, contact GitHub Support.

So I found that there were ways around this, but I still wasn't sure why. So, I emailed GitHub support to get some more information. Their response:

> Hi Isaac,
>
> Forks are designed as temporary places where you can work before merging your commits into the upstream repository. If commits in forks counted towards your contributions, in that workflow you'd end up with two contributions for each commit. One for the fork, one for the upstream repository.
>
> If a fork becomes a different project from it's upstream repository, and will never have commits merged into the upstream repository, then you can ask GitHub Support to detach the fork and turn it into a standalone repository. This will make the commits count towards your contributions. This is described here:
>
> I hope this explains the situation. Let us know if you have any further questions.
>
> Thanks, Alex

This definitely helped my understanding, and makes total sense. I had been using forks incorrectly — not as temporary places to work before merging code back into the master, but as a permanent place for storing work.

So, thank you to Alex at GitHub for clarifying this. I'll be sure to use forks properly, and have correct expectations from now on.

gha_language: SCSS


text:
# Project using Spring MVC

This version of the project is beginning to introduce SpringMVC as a model view controller for the JSP's.
This project illustates a partial migration to Spring where elements of the legacy code still remain but are integrated with the new code using Spring.
This represents a realistic situation where new technology is introduced in controlled stages and the existing code is 'refactored' to use the new framework.

In this example, we have kept the original JSP's and Object Factorys but have introduced new Spring MVC based JSP's running along side.
This means that you can compare the old and new code to see how the new functionality implements the same features as the old.
The old JSP's continue to work along side the new.

## running the application
As usual, build the whole project using
```
mvn clean install
```
Then open the project and all its subprojects in Netbeans and run both the webfacadeexample2-web and webfacadeexample2-web-client projects in the embedded tomcat server.
Please note that there will be a number of errors with the words
```
ClassFormatException: Invalid byte tag in constant pool: 19
```
Ignore these exceptions as they are caused by the old version of Tomcat embedded in Netbeans 8.2

The webfacadeexample2-web-client project is substantially unchanged but a number of changes have been made to the webfacadeexample2-web project as described below.

Browse to http://localhost:8084/basicfacadeweb/

You will see new lines in the index page
```
More Complex Farm Example using Spring MVC
click on farmhome to open application using Spring MVC
```
The springMVC version of the application is accessed from http://localhost:8084/basicfacadeweb/mvc/farmhome

## Changes in this code

The majority of the changes in this example are in the web module as shown below
```
web
   src
      main
         java
            org.solent.com504.factoryandfacade.impl.rest
                RestApp.java
                RestService.java      // NEW ANNOTATIONS FOR BEANS
            org.solent.com504.factoryandfacade.impl.web
                HelloWorld.java       // NEW - SIMPLE CLASS TO HELP EXPLAIN/DEBUG APPLCATIONCONTEXTS- this just logs a message when created.
                ViewController.java   // NEW - SPRING MVC CONTROLLER
                WebObjectFactory.java
      resources
         applicationContext.xml       // NEW - ROOT APPLICATION CONTEXT
         log4j2.xml
         mvc-dispatcher-servelet.xml // NEW - CHILD APPLICATION CONTEXT FOR MVC DISPATCHER
   webapp
      WEB-INF
         views  // NEW JSP'S WHICH ARE ONLY VIEWS WITH NO BUSINESS LOGIC
            farmlist.jsp
            reviewAnimal.jsp
         web.xml // CONFIGURATION TO SET UP MVC DISPATCHER SERVLET
      images
         Village-Farm.svg
      farm.jsp
      farm2.jsp
      index.jsp
      viewAnimal.jsp

```
The project's pom.xml files have also been updated to include some new depencencies.

## Spring application context

The first major change is the introduction of a spring application context [applicationContext.xml](../../week9/webfacadeexample2-spring/web/src/main/resources/applicationContext.xml )
which is intended to replace all references to ObjectFactories in the new code.
When we remove the old code, we will remove all of the object factories but in the interim, the spring code uses the object factories to access the underlying lbraries.

```
applicationContext.xml
...

    <!-- this allows annotations to be picked up by configuration -->
    <context:annotation-config/>
    <context:component-scan base-package="org.solent.com504.factoryandfacade.impl.rest" />

    <!-- this allows a factory to generate a singleton bean -->
    <bean id="serviceFacade" class="org.solent.com504.factoryandfacade.impl.web.WebObjectFactory" factory-method="getServiceFacade"></bean>

    <!-- this just activates and destroys a bean as an example -->
    <bean id="helloWorld" class="org.solent.com504.factoryandfacade.impl.web.HelloWorld" init-method="init" destroy-method="destroy">
        <property name="message" value="main applicationContext" />
    </bean>
```
The first lines in the file cause spring to scan all of the java files in the ```org.solent.com504.factoryandfacade.impl.rest``` package.

The scanner looks for spring annotations such as @Component and @Resource(name="serviceFacade").

When a @Component annotated class is identified, the @Resource(name="serviceFacade") causes Spring to antomatically 'inject' the serviceFacade bean.

### Jersey
The version of jersey has been updated to allow it to work along side Spring MVC.
The pom.xml depencency jars for the later version of Jersey have been changed but the code is substantially the same.

The [RestService.java](../../week9/webfacadeexample2-spring/web/src/main/java/org/solent/com504/factoryandfacade/impl/rest/RestService.java )
class has been updated to use spring to inject the serviceFacade service.

In the RestService.java @Component causes Spring to initialise and register an instance of the RestService s a spring bean.

The @Resource(name="serviceFacade") causes Spring to antomatically 'inject' the serviceFacade bean into the rest service. This means that the RestService class no longer needs to reference the WebObjectFactory directly.

```
@Component // component allows resource to be picked up
@Path("/farmService")
public class RestService {

    // SETS UP LOGGING
    // note that log name will be org.solent.com504.factoryandfacade.impl.rest.RestService
    final static Logger LOG = LogManager.getLogger(RestService.class);

    // This serviceFacade object is automacally injected by Spring
    @Resource(name="serviceFacade")
    FarmFacade serviceFacade = null;

```

The definition for the serviceFacade bean is in the applicationContext.xml file.
Because we are still using the legacy WebObjectFactory in the old jsp code, we have a bean definition in the applicationContext.xml which uses the same WebObjectFactory. You will see that this calls the getServiceFacade() method in the WebObjectFactory to return a reference to a singleton serviceFacade object.

Please compare
[week9 RestService.java](../../week9/webfacadeexample2-spring/web/src/main/java/org/solent/com504/factoryandfacade/impl/rest/RestService.java )
with
[week6 RestService.java](../../week6/webfacadeexample2/web/src/main/java/org/solent/com504/factoryandfacade/impl/rest/RestService.java )

You will see that with the WebObjectFatory removed, the code is simplified in the RestService and the WebObjectFactory is no longer needed.

## Spring MVC
A key problem with the way we have implemented the JSP files so far is that we have a strong linkage between the view and controller functions.
Ideally we want to take all of the java code out of the header of the JSP's and put it in a seperate controller.

A new class, [ViewController.java](../../week9/webfacadeexample2-spring/web/src/main/java/org/solent/com504/factoryandfacade/impl/web/ViewController.java ) has been introduced which acts as the Controller in the Model View Controller implementation.

This controller replaces the controlling java in the jsp's and simply uses the JSP's as view technology with limited control functionality.

An excerpt of this controller class is shown below. In many ways it is very simlilar to hte JAX-RS jersey service we looked at earlier.

The @Controller annotation identifies this class as a controller in the MVC.

@RequestMapping("/mvc") means that the controller will process URL's with /mvc in the classpath.
This is very similar to the @Path annotation in JAX-RS

The @Resource(name = "serviceFacade") annotation injects the farmFacade service bean created by the application context (as discussed above).

The @RequestMapping("/farmhome") annotation maps the call to the farmhome method in the controller. Again this is very like JAX-RS.

```
@Controller
@RequestMapping("/mvc")
public class ViewController {

    final static Logger LOG = LogManager.getLogger(ViewController.class);

    {
        LOG.debug("ViewController created");
    }

    // This serviceFacade object is injected by Spring
    @Resource(name = "serviceFacade")
    FarmFacade serviceFacade = null;

    @RequestMapping("/farmhome")
    public String farmhome(Model m,
            @RequestParam(value = "animalName", required = false) String animalName,
            @RequestParam(value = "animalType", required = false) String animalType) {

        LOG.debug("farmhome called animalType=" + animalType + " animalName=" + animalName);

        List<Animal> animalsList = serviceFacade.getAllAnimals();
        List<String> supportedAnimalTypes = serviceFacade.getSupportedAnimalTypes();
        m.addAttribute("animalsList", animalsList);
        m.addAttribute("supportedAnimalTypes", supportedAnimalTypes);

        // add error / response messages to page
        String errorMessage = "";
        String message = "";
        m.addAttribute("errorMessage", errorMessage);
        m.addAttribute("message", message);

        // render view with jsp
        return "farmlist";
    }
```
If you compare the old
[farm2.jsp](../../week9/webfacadeexample2-spring/web/src/main/webapp/farm2.jsp )
with the new replacement
[farmlist.jsp](../../week9/webfacadeexample2-spring/web/src/main/webapp/WEB-INF/views/farmlist.jsp )
you will see that much of the code for controling the jsp has been moved into the above farmhome() method.

(There is also some extra css and example code at the bottom of the page to illustrate how the jsp view works but this is not central to the simplification of the page.)

In the controller, each request method populates a Model object which contains attributes to be passed into the JSP based view.
The model attributes are passed to the JSP view as request attributes and each request attribute has a name which can be referenced within the page.

The choice of JSP to return as a view is determined by the return line in each controller method. For example
```
  return "farmlist";
```
The controller must be told which JSP's to use, and this is set up in the web application context [mvc-dispatcher-servlet.xml](../../week9/webfacadeexample2-spring/web/src/main/resources/mvc-dispatcher-servlet.xml )

### JSP and JSTL
If you run the web application in Tomcat and browse to http://localhost:8084/basicfacadeweb/mvc/farmhome you will see the first page of the application.
At the bottom of the page you can toggle 'Click to show detail of request variables' which will show you all of the variables in the request object which includes the Model objects the controller has injected.

The page illustrates two ways in which you could reference these model objects in your view.

We could use java directly in the page to access the injected variables. For example;
```
<%
  List<String> supportedAnimalTypes = (List<String>) request.getAttribute("supportedAnimalTypes");
%>
```
Is a declaration which could be used in the page much as before.

However, an alternative which avoids writing any java in the page definition would be to use a markup language called JSP - Standard Tag Library (JSTL).

I have included some basic tags in the page to illustrate the point
```
<%@ taglib uri="http://java.sun.com/jsp/jstl/core" prefix="c" %>
....
            <c:forEach var="animalType" items="${supportedAnimalTypes}">
                <tr>
                    <td>${animalType}</td>
                    <td>
                        <!-- post avoids url encoded parameters -->
                        <form action="./createAnimal" method="post">
                            <input type="hidden" name="animalType" value="${animalType}">
                            <button type="submit" >Create ${animalType}</button>
                        </form>
                    </td>
                </tr>
            </c:forEach>
        </table>
```
Note that the JSTL jar must also be defined in the pom.xml
```
        <dependency>
            <groupId>jstl</groupId>
            <artifactId>jstl</artifactId>
            <version>1.2</version>
        </dependency>
```
You can find out more about JSTL here https://www.tutorialspoint.com/jsp/jsp_standard_tag_library.htm

gha_language: Java


text:
# Boudica

This is the beginnings of a chess engine implemented in Go. I am a mediocre chess player and a mediocre programmer, so I have very low expectations for it. My main motivation behind
the project is to improve my knowledge of Golang and try to learn how to write high-quality, performant, idiomatic code. If I make something cool in the process, that's a huge bonus.

## Todo

- ~Initialise standard starting position~
- ~standard pawn move generation~
- en passant move generation
- sliding piece move generation
- knight move generation
- castling move generation
- implement Perft
- output UCI
- search
- eval

If you're interested in chess programming, the [chess programming wiki](https://www.chessprogramming.org/Main_Page)
is invaluable. It is my primary source of information for writing this engine, and all the techniques I've implemented
are described on the wiki.

repository: jonnyspicer/boudica

repo_name: jonnyspicer/boudica


text:
These are [David Coven's](github.com/mrcoven94) person notes as derived from Michael Hartl's awesome tutorial. The context is developing Ruby on Rails web applications using the [c9.io](c9.io) cloud based editor.

#### Running c9.io server
```ruby
rails s -p $PORT -b $IP
```
#### Running Jekyll on c9.io server
```ruby
jekyll serve --host $IP --port $PORT --baseurl ''
```
#### Adding admin permission via rails console
```ruby
U = User.find(1)
U.update_attribute :status, "admin"
```

#### Adding Image with Paperclip
After you fill in the paperclip gem, and run bundle install, make sure ImageMagick is installed via

```git
sudo apt-get install imagemagick
```

#### Configure Git
```git
$ git config --global user.name "Your Name"
$ git config --global user.email<EMAIL_ADDRESS>$ git config --global push.default matching
$ git config --global alias.co checkout
```
#### Configure BitBucket
1. Sign up for a Bitbucket account if you don’t already have one.
2. Copy your public key to your clipboard. As indicated in Listing 1.11, users of the cloud IDE can view their public key using the cat command, which can then be selected and copied. If you’re using your own system and see no output when running the command in Listing 1.11, follow the instructions on how to install a public key on your Bitbucket account.
3. Add your public key to Bitbucket by clicking on the avatar image in the upper right and selecting “Manage account” and then “SSH keys”
4. Create” to create a new repository
5. Leave the box next to “This is a private repository.” checked.
6. After clicking “Create repository”, follow the instructions under “Command line > I have an existing project”
7. When pushing up the repository, answer yes if you see the question “Are you sure you want to continue connecting (yes/no)?”

Listing 1.11
```git
$ cat ~/.ssh/id_rsa.pub
```
Adding Bitbucket and pushing up the repository.
```git
$ git remote add origin git@bitbucket.org:<username>/hello_app.git
$ git push -u origin --all # pushes up the repo and its refs for the first time
```

#### Configure Heroku
```git
$ heroku version
$ heroku login
$ heroku keys:add
```
Creating a new application at Heroku.
```
$ heroku create
$ git push heroku master
```

Rename Heroku App

```
$ heroku apps:rename newname
```
And then you should see the following in your command line

```
Renaming oldname to newname... done
http://newname.herokuapp.com/ | git@herokuapp.com:newname.git
Git remote heroku updated
```

Sometimes you may get an error saying that it cannot find the app, and that you have to specify. In this case run ```--app APP```. For example I want to rename my app from **whispering-peak-4146** to **recommend-coven**. To do this I run the following.
```
$ heroku apps:rename recommend-coven --app whispering-peak-4146
```

#### initialize  & Push to Git repo
```git
$ git init
$ git add -A
$ git commit -m "Initialize repository"
$ git remote add origin git@bitbucket.org:<username>/sample_app.git
$ git push -u origin --all # pushes up the repo and its refs for the first time
```
#### Push to Heroku repo
```git
$ git commit -am "Add hello"
$ heroku create
$ git push heroku master
```

<hr>
#### Undoing things
Even when you’re very careful, things can sometimes go wrong when developing Rails applications. One common scenario is wanting to undo code generation—for example, when you change your mind on the name of a controller and want to eliminate the generated files. Because Rails creates a substantial number of auxiliary files along with the controller, this isn’t as easy as removing the controller file itself; undoing the generation means removing not only the principal generated file, but all the ancillary files as well.  In particular, these two commands cancel each other out:
```ruby
  $ rails generate controller StaticPages home help
  $ rails destroy  controller StaticPages home help
```

Generating a model can be undone by:
```ruby
  $ rails generate model User name:string email:string
  $ rails destroy model User

```

Undoing migrations. Migrations change the state of the database using the command
```ruby
  $ bundle exec rake db:migrate
  $ bundle exec rake db:rollback
```
To go all the way back to the beginning, we can use
```ruby
  $ bundle exec rake db:migrate VERSION=0
```
Substituting any other number for 0 migrates to that version number, where the version numbers come from listing the migrations sequentially.

#### Error Handling
##### MiniTest reporters
Makes default Rails tests show red and green at the appropriate times. You can find the file in ```test/test_helper.rb```
```ruby
ENV['RAILS_ENV'] ||= 'test'
require File.expand_path('../../config/environment', __FILE__)
require 'rails/test_help'
require "minitest/reporters"
Minitest::Reporters.use!

class ActiveSupport::TestCase
  # Setup all fixtures in test/fixtures/*.yml for all tests in alphabetical
  # order.
  fixtures :all

  # Add more helper methods to be used by all tests here...
end
```
##### Backtrace silencer
Add this to the file ```config/initializers/backtrace_silencers.rb```
```ruby
# Be sure to restart your server when you modify this file.

# You can add backtrace silencers for libraries that you're using but don't
# wish to see in your backtraces.
Rails.backtrace_cleaner.add_silencer { |line| line =~ /rvm/ }

# You can also remove all the silencers if you're trying to debug a problem
# that might stem from framework code.
# Rails.backtrace_cleaner.remove_silencers!
```

##### Automated tests with Guard
For this we need to build the actual Guard file. We can do that by running the following code.
```git
$ bundle exec guard init
```
We then edit the resulting Guardfile so that Guard will run the right tests when the integration tests and views are updated

```ruby
# Defines the matching rules for Guard.
guard :minitest, spring: true, all_on_start: false do
  watch(%r{^test/(.*)/?(.*)_test\.rb$})
  watch('test/test_helper.rb') { 'test' }
  watch('config/routes.rb')    { integration_tests }
  watch(%r{^app/models/(.*?)\.rb$}) do |matches|
    "test/models/#{matches[1]}_test.rb"
  end
  watch(%r{^app/controllers/(.*?)_controller\.rb$}) do |matches|
    resource_tests(matches[1])
  end
  watch(%r{^app/views/([^/]*?)/.*\.html\.erb$}) do |matches|
    ["test/controllers/#{matches[1]}_controller_test.rb"] +
    integration_tests(matches[1])
  end
  watch(%r{^app/helpers/(.*?)_helper\.rb$}) do |matches|
    integration_tests(matches[1])
  end
  watch('app/views/layouts/application.html.erb') do
    'test/integration/site_layout_test.rb'
  end
  watch('app/helpers/sessions_helper.rb') do
    integration_tests << 'test/helpers/sessions_helper_test.rb'
  end
  watch('app/controllers/sessions_controller.rb') do
    ['test/controllers/sessions_controller_test.rb',
     'test/integration/users_login_test.rb']
  end
  watch('app/controllers/account_activations_controller.rb') do
    'test/integration/users_signup_test.rb'
  end
  watch(%r{app/views/users/*}) do
    resource_tests('users') +
    ['test/integration/microposts_interface_test.rb']
  end
end

# Returns the integration tests corresponding to the given resource.
def integration_tests(resource = :all)
  if resource == :all
    Dir["test/integration/*"]
  else
    Dir["test/integration/#{resource}_*.rb"]
  end
end

# Returns the controller tests corresponding to the given resource.
def controller_test(resource)
  "test/controllers/#{resource}_controller_test.rb"
end

# Returns all tests for the given resource.
def resource_tests(resource)
  integration_tests(resource) << controller_test(resource)
end
```

Guard can get buggy and still  slow down processes. So we may have to manually clear them.

##### Unix processes
On Unix-like systems such as Linux and OS X, user and system tasks each take place within a well-defined container called a process. To see all the processes on your system, you can use the ps command with the aux options:

```
$ ps aux
```
To filter the processes by type, you can run the results of ps through the grep pattern-matcher using a Unix pipe |:

 ```
 $ ps aux | grep spring
 ```
  ubuntu 12241 0.3 0.5<PHONE_NUMBER>16 ? Ssl Sep20 1:46
  spring app | sample_app | started 7 hours ago
The result shown gives some details about the process, but the most important thing is the first number, which is the process id, or pid.  In this particular case, you should first try stopping the processes with the spring command itself:

```
$ spring stop
```
Sometimes this doesn’t work, though, and you can kill all the processes with name spring using the pkill command as follows:

```
$ pkill -9 -f spring
```
Any time something isn’t behaving as expected or a process appears to be frozen, it’s a good idea to run ps aux to see what’s going on, and then run kill -9 <pid> or pkill -9 -f <name> to clear things up.

repository: HackerCollective/resources

branch_name: refs/heads/gh-pages

repo_name: HackerCollective/resources


text:
# Yolk Profiler

[![Scrutinizer Code Quality](https://scrutinizer-ci.com/g/simon-downes/yolk-profiler/badges/quality-score.png?b=develop)](https://scrutinizer-ci.com/g/simon-downes/yolk-profiler/?branch=develop)

A simple profiling library for benchmarking code execution speed, memory usage and database queries.

## Requirements

This library requires only PHP 5.4 or later and the Yolk Contracts package (`simon-downes/yolk-contracts`).

## Installation

It is installable and autoloadable via Composer as `simon-downes/yolk-profiler`.

Alternatively, download a release or clone this repository, and add the `\yolk\profiler` namespace to an autoloader.

## License

Yolk Profiler is open-sourced software licensed under the MIT license.

## Quick Start

```php
use yolk\profiler\GenericProfiler;

$profiler = new GenericProfiler();

$profiler->start('Op 1');
lengthyOperation1();
$profiler->stop('Op 1');

$profiler->start('Op 2');
lengthyOperation2();
$profiler->stop('Op 2');

// profile a query
$profiler->start('Query1');
doDatabaseQuery($sql, $params);
$profiler->stop('Query1');

$profiler->query($sql, $params, $profiler->getElapsed('Query1'));

// add some additional info
$profiler->meta('get', $_GET);
$profiler->meta('post', $_POST);

$profiler->stop();

// assign a configuration object to the profiler to be included in the report
$profiler->config(Config $config);

// get a report
print_r(
	$profiler->getData()
);
```

## Timers

`GenericProfiler` makes use of the `GenericTimer` class, which may also be used independently
to record periods of time at microsecond resolution.

```php
use yolk\profiler\GenericTimer;

$t = new GenericTimer();

$t->start();
$t->isRunning();	// returns true
$t->getElapsed();	// returns microseconds since last call to start()
$t->stop();
$t->isRunning();	// returns false
$t->getTotalElapsed();	// returns microseconds between all calls to start()/stop()
```

## Debug Bar

An HTML debug bar is provided that can be rendered to a web page and provide a
pretty representation of the Profiler's data.

The relevant HTML can be obtained by calling the `getHTML()` method on the profiler
and can then be inserted into a template, send to the client, etc.

The debug bar source is self-contained and is designed to be placed just before the
closing `</body>` tag.

repository: simon-downes/yolk-profiler

branch_name: refs/heads/develop

gha_language: PHP

repo_name: simon-downes/yolk-profiler


text:
## Introduction

Videos contain plethora of contextual information. For example, in a movie there are fighting scenes, sentimental scenes, romantic scenes, and many others. In a cricket match, there are wickets, sixes, fours et cetera. With the advent of the data-driven age, amateurs, researchers, and organisations alike require some specific part of this contextual information for their needs; maybe for creating a highlights reel of a sports match or mining data from movies for their machine learning models. This makes parts of certain types of videos very useful. FabBits tries to automate finding them. \\
Following are the things it will be able to detect -

*  Action sequences in movies/shows - ✅\\

*  Summary of movies/shows - ✅\\

*  Actor-specific scenes in movies/shows - ✅\\

*  Jokes in sitcoms - ✅\\

*  Slo-mos in Sports - ❌\\

*  Goals in Soccer - ✅\\

*  Goal misses in Soccer - ⭕\\

*  Three pointers in Basketball - ✅\\

## Links

Project repo - [github.com/achie27/FabBits](https://github.com/achie27/FabBits) \\
Blog posts - [ medium.com/@achie27](http://www.medium.com/@achie27 ) \\
Samples - [ Drive folder]( https://drive.google.com/drive/folders/1lsrLW0c7Pq4FYpWZjrCqKs3RFTwL-1ub )
## Requirements

You need the following things to run FabBits-\\
1. [ Python3]( https://www.python.org/download/releases/3.0/ ) \\
2. [ OpenCV ]( https://opencv.org )- Used for image and video processing \\
3. [ Moviepy ]( https://zulko.github.io/moviepy/ ) - Used for video editing and audio processing \\
4. [ PyQt5 ]( https://www.riverbankcomputing.com/software/pyqt/intro ) - Used to make the GUI \\
5. [ Scipy ]( https://scipy.org ) - Used for audio processing \\
6. [ Tesserocr]( https://github.com/sirfz/tesserocr ) - Used for, well, OCR \\
7. [ Pillow ]( https://pillow.readthedocs.io/en/latest/ ) - Used to preprocess images for OCR \\
The python dependencies can be installed by running - \\

    pip3 install scipy
    pip3 install opencv-python
    pip3 install moviepy
    pip3 install pyqt5
    pip3 install Pillow
    pip3 install tesserocr

or if you are the Anaconda kind -

    conda install -c conda-forge scipy
    conda install -c conda-forge opencv
    conda install -c conda-forge moviepy
    conda install -c anaconda pyqt
    conda install -c conda-forge pillow
    conda install -c simonflueckiger tesserocr

## Usage

Run the main GUI by -
    python3 main.py

To find your FabBit of choice -

*  Click ''MOVIES'' or ''SPORTS'' button for their respective use-cases

*  Select the use-case from the sidebar
    * A pop-up dialog will ask for the actor if actor-specific scene was chosen

*  Click on ''Choose File'' to select the input video

*  Click on ''Find FabBits''

*  Move the slider in the blue areas, which are the extracted FabBits, and play the video

*  Click on ''Save FabBits'' to save the extracted stuff into a video file

You can also run the respective files of use-cases to get their FabBit, like -
`python3 goal_detector.py soccer_match.mp4`

## References

All the references can be found listed in the repository's [readme](https://github.com/achie27/FabBits#readme).

repository: DumbMachine/ccextractor-wiki-test

gha_language: Jupyter Notebook

repo_name: DumbMachine/ccextractor-wiki-test