text:
# explicit

**Got a list of genes and you wonder what TFs regulate their expression? EXPLICIT is the right tool to try.**

The EXPLICIT approach has been developed to construct a gene expression predictor model for the plant species *Arabidopsis thaliana*. The predictor uses the expression of 1,678 transcription factor (TF) genes to predict the expression of 29,182 non-TF genes. It further enables downstream inference of TF regulators for genes and gene modules functioning in diverse plant pathways. Please check the original paper by [Geng *et al.*](https://github.com/MaShisongLab/explicit#Reference) for more details. The EXPLICIT package presented here enables users to 1. Infer TF regulators for their own gene modules; 2. Draw chord diagrams showing TF-target genes regulation for the modules; 3. Create custom gene expression predictors using their own gene expression data. (*Note: below is an example showing the analysis flow-chart for a gene module involved in vascular system development.*)

## Table of Contents
- [Install](https://github.com/MaShisongLab/explicit#Install)
- [Usage](https://github.com/MaShisongLab/explicit#Usage)
   - Infer TF regulators for gene modules
   - Draw chord diagrams showing TF-target genes regulation for the modules
   - Create custom gene expression predictor
- [Reference](https://github.com/MaShisongLab/explicit#Reference)

## Install
This package requires [Perl](https://www.activestate.com/products/perl/downloads/), [R](https://www.r-project.org/), and the [circlize](https://www.rdocumentation.org/packages/circlize/) package in R.

`circlize` can be installed within an R console via the command:

```R
install.packages("circlize")
```
[MATLAB](https://www.mathworks.com/products/matlab.html) is optional. Only required if you want to create your own predictor model using custom expression data.

Once the required software is installed, just download or clone the whole package to a local computer and start using it from the package's home directory.

## Usage

### 1. Infer TF regulators for gene modules

#### (1). Prepare the module file
The file used to store gene modules information is `modules_to_analyze.txt`. It is preloaded with 1,085 gene modules identified from a gene co-expression network described in the paper by [Geng et al.](https://github.com/MaShisongLab/explicit#Reference). The following analysis will proceed with these preloaded modules. On the other hand, you can also edit the file, replacing these modules with your own ones. The file has two tab-separated columns, with the first column being gene ids and the second being module names. For gene ids, only standard Arabidopsis AGI ids are supported. Multiple modules can be analyzed at the same time. Once finish editing, save the file without changing its name.
```
AT5G59360	Module0065
AT5G60690	Module0065
AT5G61480	Module0065
AT5G62710	Module0065
AT5G66300	Module0065
AT1G06830	Module0066
AT1G10380	Module0066
.........   ........
```
#### (2). Conduct enrichment assay to identify TF regulators for the modules
The Perl script `getArabidopsisRegulatorTFs.pl` will do the job. It takes the modules from the file `modules_to_analyze.txt` to conduct enrichment assays to identify potential TF regulators. Results are saved to a file named `results.regulator.tfs.txt`.

Open a command line window or shell terminal, navigate to the home directory of the EXPLICIT package, and type in the following command:
```shell
perl getArabidopsisRegulatorTFs.pl
```

The resulted file `results.regulator.tfs.txt` can be opened and viewed in EXCEL. It lists the potential TF regulators for every input modules. Here is an example:

![alt-text](./data/results_sample.png "Output result example")

### 2. Draw chord diagrams showing TF-target genes regulation for the modules

The `getChordDiagram` function can be used to draw chord diagrams for the modules in R. The function extracts the TF-target gene pairs from the file `results.regulator.tfs.txt` for the input module, and then uses these gene pairs to draw a chord Diagram accordingly. It has four input variables:

`module` - the name of the module
`ratio` - the relative size of the target gene area occupies
`tfnum` - the maximum number of TF genes to be included in the diagram
`targetnum` - the maximum number of target genes to be included in the diagram.

Open an R console and change the working directory to the home directory of the EXPLICIT package. Within the R console, type in the following commands:
```R
# R code
# Load the scripts that define the getChordDiagram function.
source("Rscripts.R")

# The function requires the 'circlize' package
library("circlize")

# To draw a chord diagram for Module0065
getChordDiagram( module="Module0065", ratio = 1, tfnum = 50, targetnum = 15)

# To draw a chord diagram for Module0084
getChordDiagram( module="Module0084", ratio = 1, tfnum = 50, targetnum = 15)

# Change the relative size of the target gene area
getChordDiagram( module="Module0084", ratio = 0.6, tfnum = 50, targetnum = 15)

# To draw a chord diagram for Module0046
getChordDiagram( module="Module0046", ratio = 1, tfnum = 50, targetnum = 15)

# A chord diagram with only the top 30 TFs
getChordDiagram( module="Module0046", ratio = 1, tfnum = 30, targetnum = 15)

# To draw chord diagrams for other modules
getChordDiagram( module="Module0105", ratio = 1, tfnum = 50, targetnum = 15)
getChordDiagram( module="Module0272", ratio = 1, tfnum = 50, targetnum = 15)
getChordDiagram( module="Module0011", ratio = 1, tfnum = 50, targetnum = 15)
```
Here are some output diagrams:
![alt-text](./data/chord_diagrams.png "Output result example")

### 3. Create custom gene expression predictor
We currently only have the gene expression predictor model for *Arabidopsis thaliana*. We are working on predictor models for other species. At the same time, you can also create your own custom gene expression predictor (Note: a large number of samples are required for training the model). The MATLAB function `explicit`, as specified within the file `explicit.m`, is used to create the predictor model. The file can be found within the home folder of the package. The function has the following syntax:
```matlab
explicit( TF_expression, TG_expression, TF_name, TG_name)
```
`TF_expression`: the expression matrix for TF, with samples in rows and genes in columns

`TG_expression`: the expression matrix for target genes, with samples in rows and genes in columns

`TF_name`: the names of the TF genes

`TG_name`: the names of the target genes

Below we describe the detailed procedure to create the Arabidopsis predictor. The procedure can be adapted to analyze other custom gene expression data.

#### (1). Obtain the Arabidopsis gene expression matrix
Download two matrices `At.matrix.demo.h5` and `At.matrix.full.h5` from [Figshare](https://doi.org/10.6084/m9.figshare.14152570.v1) (https://doi.org/10.6084/m9.figshare.14152570.v1) , and place them within the home folder of the EXPLICIT package . We prepared these two matrices by compiling publically available RNA-Seq datasets from NCBI, and used them to train the Arabidopsis predictor. `At.matrix.full.h5` is a full matrix with 24545 samples, while `At.matrix.demo.h5` has 5000 randomly selected samples from the full matrix. We recommend to work with the smaller matrix `At.matrix.demon.h5` first, as it requires less computational resources. Both matrices are saved in hdf5 file format, with the following data structure. Note that both files also contain an additional matrix with independent samples for validation.
```bash
At.matrix.demo.h5
├─expression_log2cpm  		(5000 samples [row] X 38194 genes [column])
├─gene_name			(38194 genes)
├─rnaseq_id			(5000 samples)
├─idx_tf_gene			(specifying TF genes used for model construction)
├─idx_target_gene 		(specifying target genes used for model construction)
└─independent_samples_for_validation
   ├───expression_log2cpm  	(2 samples [row] X 38194 genes [column])
   ├───gene_name		(38194 genes)
   └───sample_id		(2 samples)

At.matrix.full.h5
├─expression_log2cpm  		(24545 samples [row] X 38194 genes [column])
├─gene_name			(38194 genes)
├─rnaseq_id			(24545 samples)
├─idx_tf_gene			(specifying TF genes used for model construction)
├─idx_target_gene 		(specifying target genes used for model construction)
└─independent_samples_for_validation
   ├───expression_log2cpm  (2 samples [row] X 38194 genes [column])
   ├───gene_name		(38194 genes)
   └───sample_id		(2 samples)
```
#### (2). Create the expression predictor
The following analysis builds a gene expression predictor model using the smaller matrix. The analysis is conducted within a MATLAB console.  The analysis is memory-intensive and requires around 20G computer memory.
```matlab
% MATLAB code
% Navigate to and start within the home directory of the EXPLICIT package.

% Obtain the expression matrix and gene names for from At.matrix.demo.h5.
% The matrix contains 5000 samples (in rows) and 38194 genes (in columns)
mtx_demo = h5read("At.matrix.demo.h5","/expression_log2cpm");
gene_name = h5read("At.matrix.demo.h5","/gene_name");

% itf specifies which of the 38194 genes are TFs to be used.  1678 TFs are selected in total.
itf = h5read("At.matrix.demo.h5","/idx_tf_gene") == 1;

% itarget specifies which of the 38194 genes are target genes to be used. 29182 target genes are selected in total.
itarget = h5read("At.matrix.demo.h5","/idx_target_gene") == 1;

% Obtain the TF expression matrix, target gene expression matrix
tf_mtx_demo = mtx_demo(:,itf);
target_mtx_demo = mtx_demo(:,itarget);

% Obtain the TF gene names and target gene names
tf_name = gene_name(itf);
target_name = gene_name(itarget);

% Produce the predictor model
mdl_demo = explicit( tf_mtx_demo, target_mtx_demo, tf_name, target_name);

% Take a look into the predictor model
mdl_demo

% The first 5 significant TF-target gene pairs
mdl_demo.SigEdges(1:5,:)
```

The output is:
```bash
mdl_demo =

  explicit with properties:

                          beta: [1679x29182 double] 	% beta coefficients (1 intercept + 1678 TFs) X 29182 genes
                   beta_pvalue: [1679x29182 double] 	% pValues for the beta coefficiets
                       TF_name: [1x1679 string]     	% names for 1 intercept + 1678 TFs
                   Target_name: [1x29182 string]    	% names for 29182 gene
                         NRMSE: 0.0660			% NRMSE for all the training samples
         Correlation_by_sample: [5000x1 double]		% R of predicted &acutal expression for 5000 samples
    Correlation_by_target_gene: [29182x1 double]    	% R of predicted & actuall exp of each gene across all samples
                           SST: [29182x1 double]    	% SST for regression model of every gene
                           SSR: [29182x1 double]    	% SSR for regression model of every gene
                           SSE: [29182x1 double]	% SSE for regression model of every gene
                         Fstat: [29182x1 double]	% Fstat for regression model of every gene
                       Fpvalue: [29182x1 double]	% Fpvalue for regression model of every gene
                      SigEdges: [252595x4 table]	% Significant TF-target gene pairs with pValue <= 0.00001

mdl_demo.SigEdges(1:5,:) =
       Gene            TF          beta      beta_pvalue
    ___________    ___________    _______    ___________

    "AT1G01020"    "AT1G01010"     0.1231     1.373e-23
    "AT1G01020"    "AT1G15790"    -0.0596     1.074e-06
    "AT1G01020"    "AT1G25580"    -0.0828     2.755e-06
    "AT1G01020"    "AT1G30490"     0.0908     1.053e-06
    "AT1G01020"    "AT2G01060"     0.1191     7.535e-06
```
#### (3). Use the predictor model to predict independent samples
We have generated two independent RNA-Seq datasets from Arabidopsis shoot and root samples. These two datasets were not used in the model training. Below we test how well the predictor model predict these two samples. Note: the order of the gene names within the test samples are the same as those within the demo matrix.
```matlab
% MATLAB code - continued
% Obtain expression matrix for the test samples
test_mtx = h5read("At.matrix.demo.h5","/independent_samples_for_validation/expression_log2cpm");

% test_mtx contains two rows, for 'root' and 'shoot' samples, respetively.
test_sample_id = h5read("At.matrix.demo.h5","/independent_samples_for_validation/sample_id");

test_tf_mtx = test_mtx(:,itf);
actual_target_mtx = test_mtx(:,itarget);

% Predict the expression values for target genes via the formula:
% Predicted expression = [intercept (values of 1) + TF's expression matrix] X beta coefficient matrix
predicted_target_mtx = [ones(size(test_tf_mtx,1),1) test_tf_mtx] * mdl_demo.beta ;

% Calculate the correlation between predicted and actual expression values
corr( actual_target_mtx(1,:)', predicted_target_mtx(1,:)') % The correlation for root is 0.9920
corr( actual_target_mtx(2,:)', predicted_target_mtx(2,:)') % The coorelation for shoot is 0.9900

% Calculate NRMSE (Normalized Root Mean Square Error)
% NRMSE for root & shoot samples are 0.0858 & 0.1016
residual_mtx = predicted_target_mtx - actual_target_mtx ;
NRMSE = sqrt(sum(residual_mtx.^2, 2) ./ sum( actual_target_mtx.^2, 2))
```
Note: If you want to use your own RNA-Seq datasets to test the model, make sure the gene names matched in the same order to the gene names of the TF and target gene matrices used above, and the gene expression value should be log2 transformed (log2(CPM + 1)). Alternatively, you can arrange the TFs and target genes of the demo matrix with the same order as your RNA-Seq datasets, and create the gene expression predictor accordingly.

#### (4). Investigate how the number of training samples affects the predictor power
The number of training samples affects the predictor's predicting power. The function `explicit_eosn`, standing for effect of sample number, investigates such effects. Its inputs are `(TF_expression, Target_expression, TestSampleNum)`, with `TestSampleNum` being the number of samples randomly selected and hold out as test samples.
```matlab
% MATLAB code - continued
% Hold out 500 samples as test samples
mdl_eosn = explicit_eosn( tf_mtx_demo, target_mtx_demo, 500)
mdl_eosn
mdl_eosn.stat
```
The output is:
```bash
    Run    TrainingSampleNum    R_training    NRMSE_training    R_test     NRMSE_test
    ___    _________________    __________    ______________    _______    __________

     1           1700            0.99995        0.0071425       0.68378       0.931
     2           1725            0.99988         0.010676       0.79395     0.63758
     3           1750            0.99983         0.013001       0.82836     0.54658
     4           1775            0.99975         0.015839       0.86977     0.43926
     5           1800            0.99969         0.017487       0.88968     0.38962
     6           1850            0.99958         0.020327       0.91018     0.34773
     7           1900            0.99943         0.023749       0.92621     0.30313
     8           1950             0.9993         0.026351       0.93624     0.27771
     9           2000             0.9992         0.028108       0.94274     0.26096
    10           2100            0.99897         0.031836       0.95118      0.2338
    11           2200            0.99877         0.034816       0.95963     0.20954
    12           2300            0.99854         0.037856       0.96267     0.20082
    13           2400            0.99838         0.039831       0.96647     0.18906
    14           2500            0.99819         0.042092       0.96992     0.17783
    15           3000            0.99737          0.05082       0.97769     0.15087
    16           3500            0.99679         0.056046         0.981     0.13833
    17           4000            0.99629         0.060238       0.98307        0.13
    18           4500            0.99587          0.06356       0.98441     0.12439
```
Eighteen predictor models were built with between 1700 and 4500 training samples, and the predicting accuracy on test samples (R_test) increased along with the number of training samples.
#### (5). Perform K-fold Cross-Validation
K-fold Cross-Validation can be also used to test the predictor's performance. The function `explicit_kfcv` does the job. Its inputs are `(TF_expression, Target_expression, tf_name, target_name, repeats, folds)`, with `repeats` and `folds` being the number of repeats and the folds for the analysis.
```matlab
% MATLAB code - continued
% Perform 5 repeats of 10-fold Cross-Validation
mdl_kfcv = explicit_kfcv(tf_mtx_demo, target_mtx_demo, tf_name, target_name, 5, 10)
mdl_kfcv
mdl_kfcv.CV_Stat
mdl_kfcv.AllEdges(1:5,:)
```
The output is:
```bash
mdl_kfcv =

  explicit_kfcv with properties:

                 Total_repeats: 5
                   Fold_number: 10
    Correlation_of_target_gene: [29182x50 double]	% R of target genes in test samples in all 50 CV runs
              Target_gene_name: {29182x1 cell}
                      AllEdges: [48996578x9 table]	% The statistics for each coefficient across 50 CV runs
                       CV_Stat: [50x6 table]		% R and NRMSE for training and test samples in each CV run

mdl_kfcv.CV_Stat =
   Repeat    Fold_No    NRMSE_training    NRMSE_test    R_training    R_test
    ______    _______    ______________    __________    __________    _______

      1          1          0.063782        0.12305       0.99584      0.98481
      1          2          0.063719        0.12509       0.99585      0.98418
      1          3          0.063326        0.13161        0.9959      0.98232
      1          4          0.063505        0.13506       0.99588      0.98209
      1          5          0.063122        0.14025       0.99592      0.98067
      1          6          0.063771        0.11977       0.99584      0.98526
      1          7          0.063166        0.13044       0.99591      0.98294
      1          8          0.063427         0.1368       0.99589      0.98111
      1          9          0.063389        0.13608       0.99589      0.98168
      1         10           0.06349        0.13053       0.99588      0.98278
      2          1          0.063595        0.13268       0.99586      0.98234
      2          2          0.063495        0.12829       0.99587      0.98335
      2          3          0.063617        0.12701       0.99586      0.98347
      2          4           0.06349        0.12768       0.99588      0.98357
      2          5           0.06351        0.13197       0.99587      0.98255
     ...	...		...	       ...	     ...	  ...

mdl_kfcv.AllEdges(1:5,:) =
     Gene            TF          beta      beta_pvalue    CV_beta_mean    CV_beta_std    beta_bias    beta_relative_bias    relative_std
    ___________    ___________    _______    ___________    ____________    ___________    _________    __________________    ____________

    "AT1G01020"    "intercept"    0.58075       0.02262        0.58822        0.13633       0.00747           0.0129             0.2318
    "AT1G01020"    "AT1G01010"    0.12307     1.373e-23        0.12226        0.01281      -0.00081           0.0066             0.1048
    "AT1G01020"    "AT1G01030"    0.01859       0.08915        0.01981        0.00563       0.00122           0.0656             0.2842
    "AT1G01020"    "AT1G01060"    -0.0292      0.001083       -0.02862        0.00599       0.00058           0.0199            -0.2093
    "AT1G01020"    "AT1G01250"    0.02899       0.00919        0.02742        0.00629      -0.00157           0.0542             0.2294

```
`mdl_kfcv.AllEdges` can be used to check if any of the coefficient is stable or not across the CV runs.
#### (6). Perform Cross-Validation on independent samples
The function `explicit_cv` can be used to conduct Cross-Validation. Its inputs are `(Training_TF_expression, Training_traget_expression, Test_TF_expression, Test_target_expression, Test_sample_ids)`. The function uses the Training TF and target expression matrices to build the predictor model, and then test the model on the Test samples. It provides a covenient way to generate and test different predictor models.
```matlab
% MATLAB code - continued

test_target_mtx = test_mtx(:,itarget);

% Use all 5000 samples in the demo matrix to build the preidctor model,
% and test on the shoot and root RNA-Seq samples.
mdl_cv_5000 = explicit_cv( tf_mtx_demo, target_mtx_demo, test_tf_mtx, test_target_mtx, test_sample_id)

% Use only 4000, 3000, or 2000 samples to build the predictor model
mdl_cv_4000 = explicit_cv( tf_mtx_demo(1:4000,:), target_mtx_demo(1:4000,:), test_tf_mtx, test_target_mtx, test_sample_id)
mdl_cv_3000 = explicit_cv( tf_mtx_demo(1:3000,:), target_mtx_demo(1:3000,:), test_tf_mtx, test_target_mtx, test_sample_id)
mdl_cv_2000 = explicit_cv( tf_mtx_demo(1:2000,:), target_mtx_demo(1:2000,:), test_tf_mtx, test_target_mtx, test_sample_id)
mdl_cv_5000
mdl_cv_5000.Test_Sample_Stat
```
The output is:
```bash
mdl_cv_5000 =

  explicit_cv with properties:

             Training_Sample_Num: 5000
               Actual_Target_Exp: [2x29182 double]
            Predicted_Target_Exp: [2x29182 double]
                 Test_Sample_Num: 2
                Test_Sample_Stat: [2x3 table]
           Test_Sample_NRMSE_all: 0.0933
    Test_Sample_Mean_Correlation: 0.9910

mdl_cv_5000.Test_Sample_Stat =
    Sample     R_test     NRMSE_test
    _______    _______    __________

    'root'     0.99196     0.085795
    'shoot'    0.99002      0.10157
```
When using 5000, 4000, 3000, or 2000 samples to train the predictor model, the mean correlation for test samples are 0.9910, 0.9891, 0.9841, 0.9572, respectively.
#### (7). Use the full matrix to perform the analysis
Next we will conduct the analysis with the full matrix. Significant interacting TF-target genes pairs (SigEdges) will be derived. They are the same SigEdges used in Step 1 for deriving TF regulators for gene modules. Note: Since the matrix is large, it requires a large amount of computational resource. It is recommended to have at least 80G memory available.
```matlab
% MATLAB code
% clear all previous variables to free memory
clearvars ;

% Step A. Create the predictor
mtx_full = h5read("At.matrix.full.h5","/expression_log2cpm");
gene_name = h5read("At.matrix.full.h5","/gene_name");

itf = h5read("At.matrix.full.h5","/idx_tf_gene") == 1;
itarget = h5read("At.matrix.full.h5","/idx_target_gene") == 1;

tf_mtx_full = mtx_full(:,itf);
target_mtx_full = mtx_full(:,itarget);

tf_name = gene_name(itf);
target_name = gene_name(itarget);

% This produces the predictor model.
mdl_full = explicit( tf_mtx_full, target_mtx_full, tf_name, target_name);
mdl_full

% The predictor has 3298936  SigEdges (significant interaction TF-target gene pairs) with pValue <= 0.00001.
% Next the SigEdges with pValue <= 1e-9 are extracted and saved to a file named "Arabidopsis.SigEdges.1e-9.txt".
% The file contains the same SigEdgs as the file "At.SigEdges.txt" within the data directory,
% which is used by the script "getArabidopsisRegulatorTFs.pl" to derive TF-regulators for gene modules in Step 1.
% There are 980736 SigEdges with pValue <= 1e-9.
% These SigEdges are also the same SigEdges reported in the paper by Geng et al.
i = mdl_full.SigEdges{:,4} <= 1e-9 ;
sum(i)
writetable( mdl_full.SigEdges(i,:), "Arabidopsis.SigEdges.1e-9.txt", "Delimiter","tab")

% Step B. Investigate how the number of training samples affects the predictor power
mdl_eosn = explicit_eosn( tf_mtx_full, target_mtx_full, 3000)
mdl_eosn
mdl_eosn.stat

% Step C. K-fold Cross-Validation
mdl_kfcv = explicit_kfcv(tf_mtx_full, target_mtx_full, tf_name, target_name, 5, 10)
mdl_kfcv
mdl_kfcv.CV_Stat
mdl_kfcv.AllEdges(1:5,:)

% Step D. Cross-Validation on independent samples
test_mtx = h5read("At.matrix.full.h5","/independent_samples_for_validation/expression_log2cpm");
test_sample_id = h5read("At.matrix.full.h5","/independent_samples_for_validation/sample_id");
test_tf_mtx = test_mtx(:,itf);
test_target_mtx = test_mtx(:,itarget);
mdl_cv_full = explicit_cv( tf_mtx_full, target_mtx_full, test_tf_mtx, test_target_mtx, test_sample_id)
mdl_cv_full
mdl_cv_full.Test_Sample_Stat
```
The output is:
```bash
mdl_cv_full =

  explicit_cv with properties:

             Training_Sample_Num: 24545
               Actual_Target_Exp: [2x29182 double]
            Predicted_Target_Exp: [2x29182 double]
                 Test_Sample_Num: 2
                Test_Sample_Stat: [2x3 table]
           Test_Sample_NRMSE_all: 0.0796
    Test_Sample_Mean_Correlation: 0.9934

mdl_cv_full.Test_Sample_Stat =
    Sample     R_test     NRMSE_test
    _______    _______    __________

    'root'     0.99449     0.070692
    'shoot'    0.99224     0.089135
```
The mean correlation for the test samples is 0.9934 for the full model, which is better than the model constructed with 5000 samples within the demo matrix.

## Reference

Geng HY, Wang M, Gong JZ, Xu YP and Ma SS, An Arabidopsis expression predictor enables inference of transcriptional regulators for gene modules, Plant Journal, in press.

repository: MaShisongLab/explicit

gha_language: Perl

repo_name: MaShisongLab/explicit


text:
### [CVE-2010-3145](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-3145)
![](https://img.shields.io/static/v1?label=Product&message=n%2Fa&color=blue)
![](https://img.shields.io/static/v1?label=Version&message=n%2Fa&color=blue)
![](https://img.shields.io/static/v1?label=Vulnerability&message=n%2Fa&color=brighgreen)

### Description

Untrusted search path vulnerability in the BitLocker Drive Encryption API, as used in sdclt.exe in Backup Manager in Microsoft Windows Vista SP1 and SP2, allows local users to gain privileges via a Trojan horse fveapi.dll file in the current working directory, as demonstrated by a directory that contains a Windows Backup Catalog (.wbcat) file, aka "Backup Manager Insecure Library Loading Vulnerability."

### POC

#### Reference
- https://docs.microsoft.com/en-us/security-updates/securitybulletins/2011/ms11-001

#### Github
No PoCs found on GitHub currently.

license: MIT

repository: trickest/cve

branch_name: refs/heads/main

extension: md

gha_language: HTML

gha_license_id: MIT

license_type: permissive

repo_name: trickest/cve


text:
# How to Quantize Neural Networks with TensorFlow

When modern neural networks were being developed, the biggest challenge was
getting them to work at all! That meant that accuracy and speed during training
were the top priorities. Using floating point arithmetic was the easiest way to
preserve accuracy, and GPUs were well-equipped to accelerate those calculations,
so it's natural that not much attention was paid to other numerical formats.

These days, we actually have a lot of models being deployed in commercial
applications. The computation demands of training grow with the number of
researchers, but the cycles needed for inference expand in proportion to users.
That means pure inference efficiency has become a burning issue for a lot of
teams.

That is where quantization comes in. It's an umbrella term that covers a lot of
different techniques to store numbers and perform calculations on them in more
compact formats than 32-bit floating point. I am going to focus on eight-bit
fixed point, for reasons I'll go into more detail on later.

[TOC]

## Why does Quantization Work?

Training neural networks is done by applying many tiny nudges to the weights,
and these small increments typically need floating point precision to work
(though there are research efforts to use quantized representations here too).

Taking a pre-trained model and running inference is very different. One of the
magical qualities of deep networks is that they tend to cope very well with high
levels of noise in their inputs. If you think about recognizing an object in a
photo you've just taken, the network has to ignore all the CCD noise, lighting
changes, and other non-essential differences between it and the training
examples it's seen before, and focus on the important similarities instead. This
ability means that they seem to treat low-precision calculations as just another
source of noise, and still produce accurate results even with numerical formats
that hold less information.

## Why Quantize?

Neural network models can take up a lot of space on disk, with the original
AlexNet being over 200 MB in float format for example. Almost all of that size
is taken up with the weights for the neural connections, since there are often
many millions of these in a single model. Because they're all slightly different
floating point numbers, simple compression formats like zip don't compress them
well. They are arranged in large layers though, and within each layer the
weights tend to be normally distributed within a certain range, for example -3.0
to 6.0.

The simplest motivation for quantization is to shrink file sizes by storing the
min and max for each layer, and then compressing each float value to an
eight-bit integer representing the closest real number in a linear set of 256
within the range. For example with the -3.0 to 6.0 range, a 0 byte would
represent -3.0, a 255 would stand for 6.0, and 128 would represent about 1.5.
I'll go into the exact calculations later, since there's some subtleties, but
this means you can get the benefit of a file on disk that's shrunk by 75%, and
then convert back to float after loading so that your existing floating-point
code can work without any changes.

Another reason to quantize is to reduce the computational resources you need to
do the inference calculations, by running them entirely with eight-bit inputs
and outputs. This is a lot more difficult since it requires changes everywhere
you do calculations, but offers a lot of potential rewards. Fetching eight-bit
values only requires 25% of the memory bandwidth of floats, so you'll make much
better use of caches and avoid bottlenecking on RAM access. You can also
typically use SIMD operations that do many more operations per clock cycle. In
some case you'll have a DSP chip available that can accelerate eight-bit
calculations too, which can offer a lot of advantages.

Moving calculations over to eight bit will help you run your models faster, and
use less power (which is especially important on mobile devices). It also opens
the door to a lot of embedded systems that can't run floating point code
efficiently, so it can enable a lot of applications in the IoT world.

## Why Not Train in Lower Precision Directly?

There have been some experiments training at lower bit depths, but the results
seem to indicate that you need higher than eight bit to handle the back
propagation and gradients. That makes implementing the training more
complicated, and so starting with inference made sense. We also already have a
lot of float models already that we use and know well, so being able to convert
them directly is very convenient.

## How Can You Quantize Your Models?

TensorFlow has production-grade support for eight-bit calculations built in. It
also has a process for converting many models trained in floating-point over to
equivalent graphs using quantized calculations for inference. For example,
here's how you can translate the latest GoogLeNet model into a version that uses
eight-bit computations:

```sh
curl -L "https://storage.googleapis.com/download.tensorflow.org/models/inception_v3_2016_08_28_frozen.pb.tar.gz" |
  tar -C tensorflow/examples/label_image/data -xz
bazel build tensorflow/tools/graph_transforms:transform_graph
bazel-bin/tensorflow/tools/graph_transforms/transform_graph \
  --in_graph=tensorflow/examples/label_image/data/inception_v3_2016_08_28_frozen.pb \
  --out_graph=/tmp/quantized_graph.pb \
  --inputs=input \
  --outputs=InceptionV3/Predictions/Reshape_1 \
  --transforms='add_default_attributes strip_unused_nodes(type=float, shape="1,299,299,3")
    remove_nodes(op=Identity, op=CheckNumerics) fold_constants(ignore_errors=true)
    fold_batch_norms fold_old_batch_norms quantize_weights quantize_nodes
    strip_unused_nodes sort_by_execution_order'
```

This will produce a new model that runs the same operations as the original, but
with eight bit calculations internally, and all weights quantized as well. If
you look at the file size, you'll see it's about a quarter of the original (23MB
versus 91MB). You can still run this model using exactly the same inputs and
outputs though, and you should get equivalent results. Here's an example:

```sh
bazel build tensorflow/examples/label_image:label_image
bazel-bin/tensorflow/examples/label_image/label_image \
--graph=/tmp/quantized_graph.pb \
```

You'll see that this runs the newly-quantized graph, and outputs a very similar
answer to the original.

You can run the same process on your own models saved out as GraphDefs, with the
input and output names adapted to those your network requires. I recommend that
you run them through the freeze_graph script first, to convert checkpoints into
constants stored in the file.

## How Does the Quantization Process Work?

We've implemented quantization by writing equivalent eight-bit versions of
operations that are commonly used during inference. These include convolution,
matrix multiplication, activation functions, pooling operations and
concatenation. The conversion script first replaces all the individual ops it
knows about with quantized equivalents. These are small sub-graphs that have
conversion functions before and after to move the data between float and
eight-bit. Below is an example of what they look like. First here's the original
Relu operation, with float inputs and outputs:

![Relu Diagram](https://www.tensorflow.org/images/quantization0.png)

Then, this is the equivalent converted subgraph, still with float inputs and
outputs, but with internal conversions so the calculations are done in eight
bit.

![Converted Diagram](https://www.tensorflow.org/images/quantization1.png)

The min and max operations actually look at the values in the input float
tensor, and then feeds them into the Dequantize operation that converts the
tensor into eight-bits. There are more details on how the quantized representation
works later on.

Once the individual operations have been converted, the next stage is to remove
unnecessary conversions to and from float. If there are consecutive sequences of
operations that all have float equivalents, then there will be a lot of adjacent
Dequantize/Quantize ops. This stage spots that pattern, recognizes that they
cancel each other out, and removes them, like this:

![Stripping Diagram](https://www.tensorflow.org/images/quantization2.png)

Applied on a large scale to models where all of the operations have quantized
equivalents, this gives a graph where all of the tensor calculations are done in
eight bit, without having to convert to float.

## What Representation is Used for Quantized Tensors?

We approach converting floating-point arrays of numbers into eight-bit
representations as a compression problem. We know that the weights and
activation tensors in trained neural network models tend to have values that are
distributed across comparatively small ranges (for example you might have -15 to
+15 for weights, -500 to 1000 for activations on an image model, though the
exact numbers will vary). We also know from experiment that neural nets tend to
be very robust in the face of noise, and so the noise-like error produced by
quantizing down to a small set of values will not hurt the precision of the
overall results very much. We also want to pick a representation that's easy to
perform calculations on, especially the large matrix multiplications that form
the bulk of the work that's needed to run a model.

These led us to pick a representation that has two floats to store the overall
minimum and maximum values that are represented by the lowest and highest
quantized value. Each entry in the quantized array represents a float value in
that range, distributed linearly between the minimum and maximum. For example,
if we have minimum = -10.0, and maximum = 30.0f, and an eight-bit array, here's
what the quantized values represent:

```
Quantized | Float
--------- | -----
0         | -10.0
255       | 30.0
128       | 10.0
```

The advantages of this format are that it can represent arbitrary magnitudes of
ranges, they don't have to be symmetrical, it can represent signed and unsigned
values, and the linear spread makes doing multiplications straightforward. There
are alternatives like [Song Han's code books](http://arxiv.org/pdf/1510.00149.pdf)
that can use lower bit depths by non-linearly distributing the float values
across the representation, but these tend to be more expensive to calculate on.

The advantage of having a strong and clear definition of the quantized format is
that it's always possible to convert back and forth from float for operations
that aren't quantization-ready, or to inspect the tensors for debugging
purposes. One implementation detail in TensorFlow that we're hoping to improve
in the future is that the minimum and maximum float values need to be passed as
separate tensors to the one holding the quantized values, so graphs can get a
bit dense!

The nice thing about the minimum and maximum ranges is that they can often be
pre-calculated. Weight parameters are constants known at load time, so their
ranges can also be stored as constants. We often know the ranges for inputs (for
examples images are usually RGB values in the range 0.0 to 255.0), and many
activation functions have known ranges too. This can avoid having to analyze the
outputs of an operation to determine the range, which we need to do for math ops
like convolution or matrix multiplication which produce 32-bit accumulated
results from 8-bit inputs.

## What's Next?

We've found that we can get extremely good performance on mobile and embedded
devices by using eight-bit arithmetic rather than floating-point. You can see
the framework we use to optimize matrix multiplications at
[gemmlowp](https://github.com/google/gemmlowp). We still need to apply all the
lessons we've learned to the TensorFlow ops to get maximum performance on
mobile, but we're actively working on that. Right now, this quantized
implementation is a reasonably fast and accurate reference implementation that
we're hoping will enable wider support for our eight-bit models on a wider
variety of devices. We also hope that this demonstration will encourage the
community to explore what's possible with low-precision neural networks.

branch_name: refs/heads/master


text:
# **Traffic Sign Recognition**

## Writeup

---

**Build a Traffic Sign Recognition Project**

The goals / steps of this project are the following:
* Load the data set (see below for links to the project data set)
* Explore, summarize and visualize the data set
* Design, train and test a model architecture
* Use the model to make predictions on new images
* Analyze the softmax probabilities of the new images
* Summarize the results with a written report

[//]: # (Image References)

[image1]: ./writeup_images-v2.0/bar_chart_training_data.JPG "Visualization of Training Set"
[image2]: ./writeup_images-v2.0/bar_chart_augmented_training_data.JPG "Visualization of Augmented Training Set"
[image3]: ./writeup_images-v2.0/Blurred_image.JPG "Image Blur"
[image4]: ./writeup_images-v2.0/Alter_brightness.JPG "Altered brightness"
[image5]: ./writeup_images-v2.0/Rotated_Image.JPG "Rotated Image"
[image6]: ./writeup_images-v2.0/preprocess_image.JPG "Preprocessed Image"
[image7]: ./writeup_images-v2.0/sample_images_training_set_cropped.jpg "training set"
[image8]: ./writeup_images-v2.0/Augmented_training_set_cropped.jpg "Augmented training set"
[image9]: ./writeup_images-v2.0/Accuracy_graph.JPG "Accuracy Graph"
[image10]: ./writeup_images-v2.0/loss_graph.JPG "Loss Graph"
[image11]: ./writeup_images-v2.0/preprocess_example_data.JPG "preprocess example data"
[image12]: ./writeup_images-v2.0/conv_layer_1.JPG "conv layer 1"
[image13]: ./writeup_images-v2.0/conv_layer_2.JPG "conv layer 2"
[image14]: ./writeup_images-v2.0/softmax_prob.JPG "softmax prob"
[image15]: ./writeup_images-v2.0/lenet-5.JPG "lenet-5"

[image16]: ./writeup_images-v2.0/traffic_sign/1.JPG "Slippery road"
[image17]: ./writeup_images-v2.0/traffic_sign/2.JPG "Roundabout mandatory"
[image18]: ./writeup_images-v2.0/traffic_sign/3.JPG "Bumpy road"
[image19]: ./writeup_images-v2.0/traffic_sign/4.JPG "Road work"
[image20]: ./writeup_images-v2.0/traffic_sign/5.JPG "Children crossing"
[image21]: ./writeup_images-v2.0/traffic_sign/6.JPG "Speed limit (50km/h)"
[image22]: ./writeup_images-v2.0/traffic_sign/7.JPG "Speed limit (30km/h)"
[image23]: ./writeup_images-v2.0/traffic_sign/8.JPG "Yield"
[image24]: ./writeup_images-v2.0/traffic_sign/9.JPG "Go straight or right"
[image25]: ./writeup_images-v2.0/traffic_sign/10.JPG "Turn right ahead"

## Rubric Points
### Here I will consider the [rubric points](https://review.udacity.com/#!/rubrics/481/view) individually and describe how I addressed each point in my implementation.

---
### Writeup / README

#### 1. Provide a Writeup / README that includes all the rubric points and how you addressed each one. You can submit your writeup as markdown or pdf. You can use this template as a guide for writing the report. The submission includes the project code.

You're reading it!

### Data Set Summary & Exploration

#### 1. Provide a basic summary of the data set. In the code, the analysis should be done using python, numpy and/or pandas methods rather than hardcoding results manually.

I used the pandas library to calculate summary statistics of the traffic
signs data set:

* The size of training set is 34799
* The size of the validation set is 4410
* The size of test set is 12630
* The shape of a traffic sign image is (32, 32, 3)
* The number of unique classes/labels in the data set is 43

#### 2. Include an exploratory visualization of the dataset.

Here is an exploratory visualization of the data set. It is a bar chart showing how the data is distributed in the Training Set

![Distribution of the dataset][image1]

### Design and Test a Model Architecture

#### 1. Describe how you preprocessed the image data. What techniques were chosen and why did you choose these techniques? Consider including images showing the output of each preprocessing technique. Pre-processing refers to techniques such as converting to grayscale, normalization, etc. (OPTIONAL: As described in the "Stand Out Suggestions" part of the rubric, if you generated additional data for training, describe why you decided to generate additional data, how you generated the data, and provide example images of the additional data. Then describe the characteristics of the augmented training set like number of images in the set, number of images for each class, etc.)

## Sample Set before Augmentation

![image7]

I decided to generate additional data because of the following reason(s):
- As shown in the Distribution of the data in the dataset in previous image, classes such as [0,19,37,41,42] have very few data. This might cause the model to be biased around other data .
Unbalanced dataset results in biased prediction.

- low variation in the image set .
i.e. for the model to perform optimally on any image , it should be trained on augmented dataset which will have rotated image, low brightness image.

other examples : Translated Image (partially visible Image), Zoomed in Image, cropped Image, high contrast Image etc.

Firstly , I generated all the augmented Image , such that all classes have almost same number of data points.

## Augmentation Steps

* Blurring the Image.

    ![Image Blur][image3]

* Change the brightness of the Image.

    ![Altered Brightness][image4]

* Rotate the Image.

    ![Rotated Image][image5]

## Sample Set after Augmentation

![Augmented_training_set_cropped][image8]

## Preprocessing all Images in the set

* Convert the Image to grayscale.
* Contrast-limited Adaptive Histogram Equalization.
* Normalize the pixel values [0. - 1.]

![Preprocessed Image][image6]

The difference between the original data set and the augmented data set is the following ,

* Augmented data set contains blurred Image , Altered Brightness Image , Rotated Image, and Combinations of them
* Augmented data set contains 46888 more images than the original Training Set

I decided to augment the dataset in such a way that all classes have almost same number of data points

Distribution of Images in Training Dataset after Image Augmentation

![Visualization of Augmented Training Set][image2]

#### 2. Describe what your final model architecture looks like including model type, layers, layer sizes, connectivity, etc.) Consider including a diagram and/or table describing the final model.

My Lenet model consisted of the following layers:

|          Layer        |              Description	        		    |
|:---------------------:|:---------------------------------------------:|
| Input         		| 32x32x1 Grayscale image   					|
| Convolution 5x5     	| 1x1 stride, VALID padding, outputs 28x28x6 	|
| RELU					|												|
| Max pooling	      	| 2x2 stride,  outputs 14x14x6 				    |
| Convolution 5x5     	| 1x1 stride, VALID padding, outputs 10x10x16 	|
| RELU					|												|
| Max pooling	      	| 2x2 stride,  outputs 5x5x16 				    |
| Flatten               | outputs 400                                   |
| Fully connected       | outputs 120                                   |
| RELU					|												|
| Dropout          	    | rate = 0.5      								|
| Fully connected       | outputs 84                                    |
| RELU					|												|
| Dropout          	    | rate = 0.5      								|
| Fully connected		| outputs 43        							|
| Softmax				|            									|

#### 3. Describe how you trained your model. The discussion can include the type of optimizer, the batch size, number of epochs and any hyperparameters such as learning rate.

To train the model, I used

| Parameters      | Values         |
|:----------------|:--------------:|
| optimizer       | AdamOptimizer  |
| learning_rate   | 0.0005         |
| EPOCHS          | 20             |
| BATCH_SIZE      | 128            |
| beta            | 0.001          |

#### 4. Describe the approach taken for finding a solution and getting the validation set accuracy to be at least 0.93. Include in the discussion the results on the training, validation and test sets and where in the code these were calculated. Your approach may have been an iterative process, in which case, outline the steps you took to get to the final solution and why you chose those steps. Perhaps your solution involved an already well known implementation or architecture. In this case, discuss why you think the architecture is suitable for the current problem.

My final model results were:

* training set accuracy of 97.6%
* validation set accuracy of 97.7%
* test set accuracy of 94.8%

![model Accuracy][image9]

![loss accuracy][image10]

If an iterative approach was chosen:

**What was the first architecture that was tried and why was it chosen?**

![lenet-5][image15]

LeNet-5 is a very simple network. It only has 7 layers, among which there are 3 convolutional layers (C1, C3 and C5), 2 sub-sampling (pooling) layers (S2 and S4), and 2 fully connected layer (F6), that are followed by the output layer. Convolutional layers use 5 by 5 convolutions with stride 1.

**What were some problems with the initial architecture?**

With having so many data points in the training set , basic LeNet-5 Model was not able to train on the images properly.
since the model was generating so many feature maps , the model trained on training set so perfectly that performed poorly on validation set.

**How was the architecture adjusted and why was it adjusted? Typical adjustments could include choosing a different model architecture, adding or taking away layers (pooling, dropout, convolution, etc), using an activation function or changing the activation function. One common justification for adjusting an architecture would be due to overfitting or underfitting. A high accuracy on the training set but low accuracy on the validation set indicates over fitting; a low accuracy on both sets indicates under fitting.**

The Model Overfitting since the validation accuracy was too low , as compared to Training Accuracy. Previously the model was achieving 96% Training Accuracy and 91% validation accuracy.

To adjust this ,I tried a range of learning rates , and Dropout combinations . I tried different batch_size to obtain a cleanly trained model, but failed.

Solution:

- L2_Regularization: I used weight decay to tackle with the situation

 `variables = tf.trainable_variables()
  l2_regularizer = tf.add_n([ tf.nn.l2_loss(v) for v in variables if '_b' not in v.name ])
  cross_entropy = tf.nn.softmax_cross_entropy_with_logits(labels = one_hot_y, logits= logits)
  loss_operation = tf.reduce_sum(cross_entropy) + beta * l2_regularizer`

 _I skipped L2_Regularization of biases as they were small in numbers and they would almost affect neglibily in the training_

**Which parameters were tuned? How were they adjusted and why?**

 Learning_rate was adjusted from 0.00001 to 0.0005 : because the learning curve was converging very slowly
 BATCH_SIZE was adjusted from 32 to 128            : because the learning curve was converging very slowly.
 EPOCHS were restricted to 20 instead of 40        : High Epochs were overfitting the model

**What are some of the important design choices and why were they chosen? For example, why might a convolution layer work well with this problem? How might a dropout layer help with creating a successful model?**

While Designing the model few key considerations were :

- Convolution layer :  CNNs are the foundation of modern state-of-the art deep learning-based computer vision. For the model to have: local receptive fields, shared weights and spacial subsampling Convolution Layer is the best option.
Because of its shared weights there are less number of trainable parameters.

To solve this problem , we need to extract maximum information from an image , ex [lines , shape , grids , combinantions of these forming any part of the image etc.], in this method the trained model can pick features of the targeted image from any part of the image.

- Dropout Layer: Dropout Layer prevents the model from overfitting . It partially removes the neural links going forward , so that the model trains itself, in their absence.

- L2_regularization: Also known as weight decay , helps in reducing the overfitting .It helps the model to have neither high Bias not high Variance.

**How does the final model's accuracy on the training, validation and test set provide evidence that the model is working well?**

Higher Accuracy on Training Set doesn't provide enough evidence to say that it can successfully classify any given data ,
But Higher accuracy on Validation Data proves that we the neural network hyperparameters are tuned to perfection

The Test Accuracy of 94.8 %  shows that the model can now classify any new image with high success rate.

### Test a Model on New Images

#### 1. Choose five German traffic signs found on the web and provide them in the report. For each image, discuss what quality or qualities might be difficult to classify.

Here are 10 German traffic signs that I found on the web:

![Slippery road][image16]        ![Roundabout mandatory][image25] ![Bumpy road][image17]
![Road work][image18]            ![Children crossing][image19]
![Speed limit (50km/h)][image20] ![Speed limit (30km/h)][image21]
![Yield][image22]  ![Go straight or right][image23]
![Turn right ahead][image24]

#### 2. Discuss the model's predictions on these new traffic signs and compare the results to predicting on the test set. At a minimum, discuss what the predictions were, the accuracy on these new predictions, and compare the accuracy to the accuracy on the test set (OPTIONAL: Discuss the results in more detail as described in the "Stand Out Suggestions" part of the rubric).

Here are the results of the prediction:

| Image			        |     Prediction	        					|
|:---------------------:|:---------------------------------------------:|
| Slippery road         | Slippery road 							    |
| Roundabout mandatory	| Roundabout mandatory      					|
| Bumpy road            | No passing                                    |
| Road work             | Road work 								    |
| Children crossing     | Children crossing						        |
| Speed limit (50km/h)	| Roundabout mandatory				 			|
|Speed limit (30km/h)   | Yield                                         |
|Yield                  | Yield                                         |
|Go straight or right   | Go straight or right                          |
|Turn right ahead       | Turn right ahead                              |

Prediction Rate : 70%

The model was able to correctly guess 7 of the 10 traffic signs

#### 3. Describe how certain the model is when predicting on each of the five new images by looking at the softmax probabilities for each prediction. Provide the top 5 softmax probabilities for each image along with the sign type of each probability. (OPTIONAL: as described in the "Stand Out Suggestions" part of the rubric, visualizations can also be provided such as bar charts)

The code for making predictions on my final model is located in the [105th -111th] cell of the Ipython notebook.

![softmax_prob][image14]

The model performed fairly well on the new test set.

### (Optional) Visualizing the Neural Network (See Step 4 of the Ipython notebook for more details)
#### 1. Discuss the visual output of your trained network's feature maps. What characteristics did the neural network use to make classifications?

Convolution Layer 1:

![Conv_layer_1][image12]

Here we can see that for 32x32 grayscale image , for the very first layer the feature maps extracted all the edges of the given Image, and picked the soft , hard edged as well. It picked lines , shape from the image

Convolution Layer 2:

![Conv_Layer_2][image13]

Since the 1st convolution layer was followed by RELU and 2x2 maxpooling , in 2nd Convolution Layer the feature map extracted the pixel information which makeup the lines , shapes and edges in the 1st convolution Layer

path: /writeup -v2.0.md

filename: writeup -v2.0.md


text:
# Coinevents - Progressive web apps

The goal of this project is to refactor the [web app](https://github.com/StanBankras/web-app-from-scratch-2021) I built from scratch from client side to server side render.

I plan to use NodeJS with Express and EJS to achieve this.

During this project I also want to practice using Typescript.

## Table of contents
* [Coinpaprika API](https://github.com/StanBankras/progressive-web-apps-2021#coinpaprika-api)
* [Concept](https://github.com/StanBankras/progressive-web-apps-2021#concept)
* [Features](https://github.com/StanBankras/progressive-web-apps-2021#features)
* [Installation](https://github.com/StanBankras/progressive-web-apps-2021#installation)
* [Resources](https://github.com/StanBankras/progressive-web-apps-2021#resources)
* [License](https://github.com/StanBankras/progressive-web-apps-2021#features)

## :sweet_potato: Coinpaprika API
I have chosen to work with the [Coinpaprika API](https://api.coinpaprika.com/). It's an API that has data of all live cryptocurrencies, ranked based on marketcap. The API is open, always free to use and very friendly to use. Using the API I can retrieve data per coin like on which exchanges it is listed, tweets related to the coin and upcoming events for the coin.

**Base URL**

The base URL of this API is `https://api.coinpaprika.com/v1/`.

**Endpoints used in this project**:
* `/coins` - Retrieve all coins listed on Coinpaprika as object with their corresponding coin-id
* `/coins/${id}/ohlcv/historical` - Retrieve day to day historical price data from a coin
* `/coins/${id}/markets` - Retrieve trading markets that are active for this coin
* `/coins/${id}/twitter` & `/coins/${id}/events` - Retrieve latest Tweets & events by this coin's project

**Rate limit**

Coinpaprika limits the amount of requests per IP to [10 per second](https://api.coinpaprika.com/#section/Rate-limit). Make sure to handle this when using the API.

## :pencil2: Concept
The concept is to make an alternative to [Coinpaprika](https://coinpaprika.com/) where I show upcoming events and most recent tweets of a project, instead of raw data. This could help a normal person to better understand what is about to happen to the project that is behind the coin.

## :memo: Packages used
* Rollup - *Transpile, transform and rename files with different build steps*
* Clean CSS - *Minify CSS*
* Compression - *Enable compression server-side so client receives files more compact*
* Crypto - *Create hashes from strings for cache invalidation*
* ejs - *Templating engine*
* typescript - *Write javascript in a more strict, typed way*
* uglify-js - *Minify javascript*

## :construction_worker: Build steps
1. Delete current dist folder
2. Copy static files from src to dist folder
  - Add hash to CSS and clientside JS for cache invalidation
  - Minify & reference hashed filenames correctly in related files
3. Transpile Typescript to javascript
4. Minify javascript

## :rocket: Features
:heavy_check_mark: Most recent tweet of all top 20 cryptos

:heavy_check_mark: Most latest event of all top 20 cryptos

:heavy_check_mark: Monthly price chart per coin

:heavy_check_mark: Find out which pairs the coin is traded against and where

## :gear: Installation
Install dependencies
```
npm install
// or
yarn install
```
Watch files
```
npm run watch
// or
yarn watch
```
Run server
```
npm run dev
// or
yarn dev
```

## TODO
- [x] "Refresh page to see new content", if new content was found
- [x] Compression serverside
- [x] Clientside javascript encode/minify
- [x] Revision hash static files
- [x] Catch app install event, make custom installation popup

## Resources
* [TSConfig docs](https://www.typescriptlang.org/tsconfig)
* [Options for compiling TS](https://medium.com/@lucksp_22012/3-options-to-compile-typescript-to-js-rollup-tsc-babel-3319977a6946)
* [ESLint docs](https://eslint.org/)
* [Converting JS to TS](https://dev.to/animo/convert-an-express-nodejs-app-from-javascript-to-typescript-302l)
* [NPM run all](https://dev.to/danywalls/how-to-run-multiple-npm-scripts-with-npm-run-all-3ad2)
* [Getting started with Serviceworkers](https://developers.google.com/web/fundamentals/primers/service-workers)
* [Expiration date for cache](https://gomakethings.com/how-to-set-an-expiration-date-for-items-in-a-service-worker-cache/)
* [PWA custom install prompt](https://web.dev/customize-install/)

## License
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)


text:
# Contributing to Mun

Ah! Good to see you are reading this. We feel that developing a programming
language without the input of our target audience is counterproductive.
Therefore we gladly welcome contributions from all developers!

If you haven't already, [join our discord server](https://discord.gg/SfvvcCU).
We can help you get underway with the things you're most excited about.

Some important resources to get started are:

- [Our roadmap](https://trello.com/b/ZcMiREnC/mun-roadmap) gives a very broad
  overview of where we're heading.
- We're usually online on [our discord server](https://discord.gg/SfvvcCU)
  during business hours (UTC+1).
- Bugs? [Report them on GitHub](https://github.com/mun-lang/mun/issues)
- What is our vision on [testing](#testing)?
  - Try to use [Test Driven Development](#test-driven-development)
  - About [snapshot tests](#snapshot-tests)
- Our [Git workflow](#git-workflow)
  1. [Fork](#fork-the-repository) in the repository
  2. Locally [clone](#locally-clone-the-fork) the fork
  3. Create a [branch](#create-a-branch)
  4. [Synchronise](#synchronise-your-branch) your branch
  5. [Commit](#commit-changes) changes
  6. [Push](#push-changes) changes
  7. Create a [Pull Request](#create-a-pull-request)
  8. [Merging](#merging-a-pull-request) a Pull Request

## Testing

We feel that having an extensive [regression
testing](https://en.wikipedia.org/wiki/Regression_testing) suite allows
developers to add new features or change existing features with more confidence;
knowing that changes behave the way you expect them to and without unwittingly
impacting other features.

### Test Driven Development

We try to implement new features using [Test Driven Development
(TDD)](https://en.wikipedia.org/wiki/Test-driven_development). In practice this
means that we write tests - based on requirements - before implementing new
features to ensure that they work. This seamlessly integrates with regression
testing, as there is no extra workload.

### Snapshot Tests

A snapshot test asserts that the output of an operation doesn't change. As such,
we use it to verify that a feature generates (and keeps generating) the correct
output. The [insta crate](https://crates.io/crates/insta) is used throughout the
codebase to enable this functionality. See
[crates/mun_hir/src/ty/tests.rs](crates/mun_hir/src/ty/tests.rs) for an example.

## Git Workflow

We follow a Git workflow similar to
[Kubernetes](https://github.com/kubernetes/community/blob/main/contributors/guide/git_workflow.png).
If you are not familiar with it, please review the following instructions.

### Fork the Repository

1. Visit https://github.com/mun-lang/mun
2. Click the `Fork` button (top right) to establish a cloud-based fork.

You should now have a fork of the Mun repo at `https://github.com/$user/mun`,
where `$user` is your GitHub handle.

### Locally Clone the Fork

To create a local clone of your fork at the `$working_dir` folder, execute the
following Git commands:

```bash
cd $working_dir
git clone https://github.com/$user/mun.git
# or: git clone git@github.com:$user/mun.git

cd $working_dir/mun
git remote add upstream https://github.com/mun-lang/mun.git
# or: git remote add upstream git@github.com:mun-lang/mun.git

# Confirm that your remotes make sense:
git remote -v
```

### Create a Branch

Update your local `main` branch:

```bash
cd $working_dir/mun
git fetch upstream
git checkout main
git rebase upstream/main
```

Branch from `main`:

```bash
git checkout -b feature/my-precious
```

Now you are ready to edit code on the `feature/my-precious` branch.

### Synchronise your Branch

```bash
# While on your feature/my-precious branch
git fetch upstream
git rebase upstream/main
```

Please don't use `git pull` instead of the above `fetch` / `rebase`. By default,
`pull` uses a built-in list of strategies that result in merge commits. These
make the commit history messy and violate the principle that commits ought to be
individually understandable and useful. You can also consider changing your
`.git/config` file via `git config branch.autoSetupRebase always` to change the
behaviour of `git pull` to always use [`--rebase`
merging](https://www.atlassian.com/git/tutorials/rewriting-history/git-rebase).

### Commit changes

Commit your changes.

```bash
git commit
```

Always write a clear log message for your commits. We use the [Conventional
Commits](https://www.conventionalcommits.org/) format, which states that a commit
message should be structured as follows:

```
<type>[optional scope]: <description>

[optional body]

[optional footer(s)]
```

BREAKING CHANGE: a commit that has a footer BREAKING CHANGE:, or appends a !
after the type/scope, introduces a breaking API change (correlating with MAJOR
in semantic versioning). A BREAKING CHANGE can be part of commits of any type.

Recommended *type*s are: `feat`, `fix`, `ci`, `docs`, `style`, `refactor`,
`perf`, `test`, `revert` or `improvement`.

One-line messages are fine for small changes, but bigger changes should include
a body.

#### Examples

```
feat: allow provided config object to extend other configs
```
```
refactor!: drop support for Node 6
```
```
feat(lang): add polish language
```
```
fix: correct minor typos in code

see the issue for details

on typos fixed.

Reviewed-by: Z
Refs #133
```

For more examples, check [recent commit
message](https://github.com/mun-lang/mun/commits/main).

### Push Changes

When ready to review (or to create an off-site backup of your work), push your
branch to your fork:

```bash
git push $your_remote_name feature/my-precious
```

If history was rewritten as a result of a rebase merge, you'll need to [force
push](https://blog.developer.atlassian.com/force-with-lease/) changes.

### Create a Pull Request

Please submit a [GitHub Pull Request to
mun-lang/mun](https://github.com/mun-lang/mun/pull/new/main) with a clear list
of changes (read more about [pull
requests](http://help.github.com/pull-requests/)). When you submit a pull request,
make sure to include tests that validate the implemented feature or bugfix
([read about testing in Mun](testing)). Before committing, please confirm that your code
style is correct (using `cargo fmt`) and all lint warning have been resolved
(using `cargo clippy`). We integrated [cargo-husky](https://github.com/rhysd/cargo-husky)
as a pre-commit hook, to make this process as simple as possible.

Please consider that very small PRs are easy to review, whereas very large PRs
are very difficult to review. The more focused your PR, the shorter the timeline
for approval.

#### Get a Code Review

Once your pull request has been opened, it will be assigned to one or more
reviewers. Those reviewers will do a thorough code review, looking for
correctness, bugs, opportunities for improvement, documentation and comments,
and style.

Please commit changes made in response to review comments to the same branch on
your fork. Feel free to use GitHub's suggestions, but you'll likely have to
clean up your history afterwards.

#### Cleaning up Commit History

After a review, prepare your PR for merging by cleaning up the commit history.
All commits left on your branch after a review should represent meaningful
milestones or units of work. Use commits to add clarity to the development and
review process.

Before merging a PR, squash the following kinds of commits:

- Fixes/review feedback
- Typos
- Merges and rebases
- Work in progress

Aim to have every commit in a PR compile and pass tests independently if you
can, but it's not a requirement. In particular, merge commits must be removed!

To edit or squash your commits, perform an [interactive
rebase](https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History). Start an
interactive rebase using a specific commit hash, or count backwards from your
last commit using `HEAD~<n>`, where `<n>` represents the number of commits to
include in the rebase.

```bash
git rebase -i HEAD~3
```

The output looks similar to:

```bash
pick 2ebe926 feat(memory): add mark-region garbage collector
pick 31f33e9 misc: apply review suggestions
pick b0315fe test(memory): add unit test for mark-region gc

# Rebase 7c34fc9..b0315ff onto 7c34fc9 (3 commands)
#
# Commands:
# p, pick <commit> = use commit
# r, reword <commit> = use commit, but edit the commit message
# e, edit <commit> = use commit, but stop for amending
# s, squash <commit> = use commit, but meld into previous commit
# f, fixup <commit> = like "squash", but discard this commit's log message

...
```

Use a command-line text editor to change the word `pick` to the appropriate
command, e.g. `fixup` for commits that you want to squash:

```bash
pick 2ebe926 feat(memory): add mark-region garbage collector
fixup 31f33e9 misc: apply review suggestions
pick b0315fe test(memory): add unit test for mark-region gc

...
```

Upon a successful rebase, push your changes to the remote branch:

```bash
git push --force-with-lease
```

### Merging a Pull Request

Once your pull request has been reviewed and approved, your PR is ready for
merging. Merging will automatically be taken care of by the Reviewer.

repository: mun-lang/mun

gha_language: Rust

gha_license_id: NOASSERTION

repo_name: mun-lang/mun


text:
# Employee Tracker

## Table of Contents
* [1. Description](#description)
* [2. Installation](#installation)
* [3. License](#license)
* [4. Questions](#questions)

<img width="736" alt="schema" src="https://user-images.githubusercontent.com/74689981/115097855-955f5e80-9efa-11eb-9589-097189ed99b7.png">

## Description
Command line app that allows you to view & add employees, departments, and roles within the database.

## Installation
Clone this repo to your local computer, install necessary dependencies, and run the app by typing "npm start" in your terminal.

## License
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

## Questions
If you have any questions about this project, you can reach me at<EMAIL_ADDRESS>


text:
## Installation

`pip install -r requirements.txt`

## Usage

**Disclaimer:** This was one of the author's first non-trivial projects with Python, so code quality is not too good. The appservice has much better code.

* Run `bridge.py` to generate `config.json`

* Edit `config.json`:

```
{
    "homeserver": "https://matrix.org",
    "username": "@name:matrix.org",
    "password": "my-secret-password",  # Matrix password.
    "token": "my-secret-token",  # Discord bot token.
    "discord_cmd_prefix": "my-command-prefix",
    "bridge": {
                "channel_id": "room_id",
                "channel_id2": "room_id2",  # Bridge multiple rooms.
     },
}
```

This bridge does not use databases for keeping track of bridged rooms to avoid a dependency on persistent storage. This makes it easy to host on something like Heroku with the free tier.

* Logs are saved to the `bridge.log` file in `$PWD`.

* Normal Discord bot functionality like commands can be added to the bot via [cogs](https://discordpy.readthedocs.io/en/latest/ext/commands/cogs.html), example [here](https://gist.github.com/EvieePy/d78c061a4798ae81be9825468fe146be).

**NOTE:** [Privileged Intents](https://discordpy.readthedocs.io/en/latest/intents.html#privileged-intents) must be enabled for your Discord bot.

repository: forksbot/matrix-discord-bridge

repo_name: forksbot/matrix-discord-bridge


text:
# Troubleshooting

## Overview

When integrating with the CAFe endpoints, there are two approaches that teams performing the integration can use to validate the events.

* Review browser network requests
* Search the S3 bucket for the data in question

Each of these methods will be documented below.

In addition, CAFe endpoints emit logging messages when errors are encountered.
It is beyond the scope of this project to implement centralized logging for the API Gateway components.

## Browser Network Requests

For events captured client-side, a first level of validation can be done by monitoring the network request to the endpoint via the browser's developer tools.

Attention should be paid to the following:

* Was the request made when expected (ie, upon the defined trigger)?
* What was the request response?
  * A 200 response indicates a successful post
  * A 400 response indicates a problem with the request body (including event schema validation failures)
  * A 403 response indicates a problem with the request authentication (AWS 'x-api-key' is missing/incorrect)
  * (Other responses may be applicable)
* Does the request body match the requirements?
  * Validate that the event category and action match the specification
  * Validate that all the required contextual data for the given event category/action are present
* Validate that the Client-Side Eventing Guidelines are adhered to

## Search S3 Bucket

There are various tools available to scan through S3 contents.

TODO: Document S3 searching

repository: cjnygard/cafe

repo_name: cjnygard/cafe


text:
# MailMan
---
An email library that makes sense.

This library was created out of frustation with existing popular libraries. They are too complex, not really testable and make it hard to separate responsabilities.

## Sending emails

Sending emails is simple. All you need is an `EmailSender`, write an `Email`, say who is sending it and set its recipients.

```java
EmailSender emailSender = new SmtpEmailSender("smtp.gmail.com");
emailSender.setAuthentication("username", "password");

Email email = new Email();
email.setMessage("Hello, World!");
<EMAIL_ADDRESS>"Me");
email.addRecipient(RecipientType.TO, "You", "you@world.com");

emailSender.send(email);
```

There are two important things to notice here. First, the `EmailSender` creation is completely isolate from the `Email` creation. That is important because it means that when you writing emails, you don't need to know how they'll be sent, you just need someone to give you an `EmailSender` and you are set. The other thing is that `Email` is a POJO, so it is easy to have fun with it. For instance, you can serialize it, persist it, read it from an structured file format, anything.

repository: Ampliato/MailMan

repo_name: Ampliato/MailMan


text:
# OpenX JSON format

This code in this package implements the OpenX JSON SerDe (`org.openx.data.jsonserde.JsonSerDe`)
with extensions. The behavior is based on the implementations maintained by
[Rcongiu](https://github.com/rcongiu/Hive-JSON-Serde) and [Starburst](https://github.com/starburstdata/hive-json-serde).
The Rcongiu implementation is based on very old Hive systems, has many quirks and bugs, and is not
actively maintained (as of this writing). The Starburst version is much more modern, and actively
maintained, but has it own quirks. This implementation generally a super set of both of these
implementations, including support for reading and writing more types, and generally attempts to
normalize the behavior of type mappers, and line processing. Except for some noted cases below,
this implementation will read JSON that the other implementations can read, and can read the JSON
written by the other implementations. Additionally, the JSON written by this implementation can
be read by the other implementations, assuming they support the data type at all.

# Line Format

Hive SerDe operate on a per-line basis, so multi line JSON can not be supported. Each line is
processed as follows:

1. If the first non-whitespace character of the line is not `{` or `]`, a row with all fields
   set to null is produced.
2. The line is parsed using the very lenient [open-json](https://github.com/tdunning/open-json)
   parser. A parser failure can be ignored by setting the `ignore.malformed.json` table property,
   and in this case a row with all fields set to null is produced.
3. Any text after the close of the JSON object or array is ignored.
4. The JSON is parsed using the Type Mapping rules below.

# Type Mapping

The JSON parser will only produce three Java types: `Map`, `List`, and `JsonString`. A `JsonString`
is normal quoted JSON string, or an unquoted JSON value. In valid JSON, unquoted strings are
limited to numbers, and boolean `true` or `false`, but the lenient parser used in OpenX JSON
SerDe allow for field names and value to be any unquoted string.

Specific type mapping rules are described below, but, general guidelines are:

* Parse failures result in `null`.
* Values that are outside the allowed type range result in `null`.
* JSON object or array values for numeric or temporal types, throw an exception.
* Text types support all JSON values by rendering the value as canonical JSON.

## BOOLEAN

`true` only if quoted or unquoted string is equals ignoring case to `true`. All other values,
including array and object, result in `false`.

### Known Differences

* Rcongiu SerDe fails on array or object value.

## BIGINT, INTEGER, SMALLINT, and TINYINT

Values that fail to parse properly, or that are outside the allowed numeric range, result in `null`.

### Formats

* Any value that can be parsed by `new BigDecimal(String)`
* Hex `0x1234`. Negative numbers are not supported.
* Octal `01234`. Negative numbers are not supported.

### Known Differences

* Rcongiu fails on values with decimals or exponents.
* Starburst performs a narrowing cast to expected size.
* Starburst and Rcongiu do not support octal in map keys or quoted octal values.
* Starburst and Rcongiu both have a bug that allows negative hex and octal values formatted like `0x-123` or `0-123`.

## DOUBLE and REAL

Values that fail to parse properly result in `null`.

### Formats

* Any value that can be parsed by `Double.parseDouble` or `Float.parseFloat`
* `NaN`, `Infinity`, `+Infinity`, and `-Infinity`  are supported

### Known Differences

Rcongiu maps keys to lower case when in case-insensitive mode (default), and that causes a failure when
processing the `NaN` and `Infinity` literals.

## DECIMAL

Values are parsed with `BigDecimal`, rounded (half up) to declared scale. Values that fail to parse properly,
or that exceeds the maximum precision, result in `null`.

### Formats

* Any value that can be parsed by `new BigDecimal(String)`

### Known Differences

* Rcongiu SerDe does not support `DECIMAL` type.
* Starburst fails if the value fails to parse.
* Starburst writes all `DECIMAL` values as `null` .

## VARCHAR and CHAR

All JSON values are supported by rendering canonical JSON to text. JSON will be rendered without any
additional whitespace. Unquoted values that can be parsed to a boolean or a number are canonicalized.
Unquoted field names are not canonicalized.

### Coercions

* **Boolean**: either `true` or `false`
* **Number**: the number parsed and formatted with BigDecimal
* **Array**:  value is rendered back to JSON
* **Object**: value is rendered back to JSON

### Known Differences

* Rcongiu fails on unquoted boolean and numbers in field names, but all other unquoted values are allowed.
* Starburst and fails for unquoted field names.
* Rcongiu and Starburst will use `Double` in some cases for canonicalization of numbers which results in
  a slightly different format from `BigDecimal`.
* Rcongiu and Starburst do not support conversions for `CHAR` or bounded `VARCHAR`.
* Rcongiu and Starburst write `CHAR` or bounded `VARCHAR` values as `null` regardless of the actual value.

## VARBINARY

Values are decoded with RFC 4648 Base64. Unquoted strings are not supported and result in `null`.

### Formats

* Base64 decode; `null` if the value is not a quoted string, or cannot be decoded.

### Known Differences

* Rcongiu SerDe does not support `VARBINARY` type.

## DATE

Values that fail to parse properly, or that are outside the allowed range, result in `null`.

### Formats

* `YYYY-MM-DD` with any trailing text after a space
* Decimal number of days since `1970-01-01`
* Hex `0x1234` days. Negative numbers are not supported.
* Octal `01234` days. Negative numbers are not supported.

### Known Differences

* Rcongiu SerDe parses dates using the broken `java.text.DateFormat` code which produces
  incorrect results for dates before 1582-10-15.
* Rcongiu does not support integer days since epoch format.
* Starburst fails for boolean
* Starburst and Rcongiu writes all `DATE` values as `null`

## TIMESTAMP

Values that fail to parse properly, or that are outside the allowed range, result in `null`.

### Text Formats

Then time zone is present the value is converted to the same instant in UTC, and then that local
time is used. The parser uses lenient mode, so values too large for a field are wrapped into the
next field

* `yyyy-MM-dd'T'HH:mm:ss.ffff'Z'` (default)
* `yyyy-MM-dd'T'HH:mm:ss.ffff+HHMM`
* `yyyy-MM-dd hh:mm:ss.ffff`

### Numeric Formats

Numeric formats are the number of seconds or milliseconds since 1970-01-01. Numbers are always
in base 10. Sign and leading zeros are supported. When counting digits sign and leading zeros
are included.

* decimal number: `seconds.millis`
* 13 or more digits: `millis`
* 12 or fewer digits: `seconds`

### Table Properties

* `timestamp.formats`: A comma separated list of `java.time.format.DateTimeFormatter` patterns.
  If the value does not match one of the patterns, the default patterns are used.

### Known Differences

* Rcongiu SerDe parses dates using the broken `java.text.DateFormat` code which produces
  incorrect results for dates before 1582-10-15.
* Rcongiu does not support the `timestamp.formats` property.
* Starburst does not use built in formats when timestamp formats are provided.
* Starburst allows unquoted octal values in numeric formats.
* Rcongiu and Starburst both fail when patterns do not match.
* Rcongiu and Starburst both allow values with exponents in numeric formats.
* Rcongiu and Starburst uses a more strict timestamp parser mode.

## ARRAY

### Coercions

* Empty string is coerced to `null`.
* Primitive or object value (without array wrapper) is coerced to a single element array.

### Known Differences

None

## MAP

Entries where the key evaluates to `null` are skipped. For maps with duplicate keys values
(after parsing), only last value parsed and used. For serialization, map keys must be a
primitive type, but for legacy reasons, there is very limited support for reading structural
map keys due to the implicit string coercions in ARRAY, MAP, and ROW types. It is not
recommended that structural keys are used as they can generally only result in empty maps,
or for ARRAY, a single element array key.

### Coercions

* String containing only whitespace is `null`
* All other types fail

### Known Differences

* Rcongiu SerDe map all keys to lower case during parsing, which affected all parsers.
* Rcongiu fails for duplicate map keys during parse, but duplicates are supported for keys
  that parse to the same value (e.g., doubles with trailing zeros).

## ROW

Keys that do not map to a row field are ignored. For objects with duplicate keys values,
only last value parsed and used.

### Coercions

* String containing only whitespace is coerced to a row with all fields set to `null`
* All other types fail

### Table Properties

* `case.insensitive` (default true): Keys are mapped to fields using a case-insensitive name match
  unless disabled. When case-insensitivity is disabled, fields containing uppercase names will
  never match as Hive field names are always lowercase.
* `dots.in.keys` (default false): If key does not have an exact match, find the first key that
  matches after replacing `.` with `_` in the key. This behavior is affected by the
  `case.insensitivity` property, but not the `mapping` properties.
* `mapping.{field-name}=key`: Matches the field to the specified key. This behavior is affected
  by the `case.insensitivity` property, but not the `dots.in.keys` property.
* `explicit.null` (default false): When enabled, a `null` field is rendered with an explicit JSON
  `null`, otherwise the field is not rendered at all.

### Coercions

* String containing only whitespace is `null`

### Known Differences

* Rcongiu SerDe map all keys to lower case during parsing, which affected all parsers.
* Rcongiu fails for duplicate map keys during parse, but duplicates are supported for keys that
  parse to the same value (e.g., doubles with trailing zeros).
* Rcongiu does not support `explicit.null` option.

repository: trinodb/trino

gha_language: Java

repo_name: trinodb/trino


text:
---
title: "Exams over. Fun week ahead."
category: life
---

After over 25 days of studying, my 6th sem exams finally came to an end.

I'm super excited. I had all these cool ideas during these days which I didn't have time to do.

So here's a *rough* note of some of the things I'll be doing for a week or so.

* Make a programming language.
    - Pretty simple right? :P
    - I already have a pretty sound grasp on the *theory* of compilers. Now it's time to implement one.
    - Might have to refresh on compiler concepts.
* Make a personal photo hosting solution
    - I really need a way to upload images I use instantly.
    - May include screenshot+insta upload.
    - Should include access to api.
    - Possible authentication?
    - Secure possible authentication?
* Make a personal analytics app
    - y u so complicated google analytics?
* Follow through the SICP [video lectures](https://www.youtube.com/playlist?list=PLE18841CABEA24090) from MIT.
    - learn mi some functional

repository: oxalorg/personal-blog

repo_name: oxalorg/personal-blog


text:
# webgl-snow

A particle system with collision detection written in three.js. Click [here](https://gorge-raccoon.github.io/webgl-snow/) to run the simulation.

## Usage

You can also run the simulation locally. Use a web server to avoid CORS issues.

```console
git clone https://github.com/gorge-raccoon/webgl-snow.git
cd webgl-snow
python3 -m http.server 8080&
gio open http://localhost:8080
```

repository: yiidot/webgl-snow

repo_name: yiidot/webgl-snow


text:
---
layout: default
title: Conda
nav_order: 3
description: "Conda commands."
---

# Conda
{: .no_toc }

## Table of contents
{: .no_toc .text-delta }

1. TOC
{:toc}

---

# Create and Update Conda Environments

**Example yaml file.**
```
 name: h5n1day100
 channels:
   - bioconda
   - conda-forge
 dependencies:
   - r=3.5.1
   - snakemake=5.4.5
   - r-tidyverse=1.2.1
   - r-feather=0.3.2
   - r-venndiagram=1.6.20
   - r-tmod=0.40
   - r-biocmanager=1.30
   - bioconductor-biobase=2.42
   - r-logging=0.9_107
```

`conda env create --file=environment.yaml`

`conda env update --file=environment.yaml`

**To activate conda environment** `conda activate h5n1day100`

**To deactivate conda environment** `conda deactivate`

**To remove conda environment** `conda env remove --name=<env name>`

# To Completely remove Anaconda from your Computer

To remove the configs:
```
conda install anaconda-clean
anaconda-clean --yes
```

Once the configs are removed you can delete the anaconda install folder, which is usually under your home dir:
`rm -rf ~/anaconda3`

Also, the `anaconda-clean --yes` command creates a backup in your home directory of the format `~/.anaconda_backup/<timestamp>`. Make sure to delete that one also.

Now if you want to clean all, you will also have to delete the two last lines added to your .bash_profile. They look like:
```
# added by Anaconda3 5.2.0 installer
export PATH="/Users/ody/anaconda3/bin:$PATH"
```

repository: rohitfarmer/docs

gha_language: CSS

repo_name: rohitfarmer/docs


text:
---
id: version-0.4.0-enums
title: Enumeration types
original_id: enums
---

Ariadne supports GraphQL [enumeration types](https://graphql.org/learn/schema/#enumeration-types) which by default are represented as strings in Python logic:

```python
from ariadne import QueryType
from db import get_users

type_defs = """
    type Query{
        users(status: UserStatus): [User]!
    }

    enum UserStatus{
        ACTIVE
        INACTIVE
        BANNED
    }
"""

query = QueryType()

@query.field("users")
def resolve_users(*_, status):
    # Value of UserStatus passed to resolver is represented as Python string
    if status == "ACTIVE":
        return get_users(is_active=True)
    if status == "INACTIVE":
        return get_users(is_active=False)
    if status == "BANNED":
        return get_users(is_banned=True)
```

The above example defines a resolver that returns a list of users based on user status, defined using the `UserStatus` enumerable from the schema.

There is no need for resolver to validate if `status` value belongs to the enum. This is done by GraphQL during query execution. Below query will produce an error:

```graphql
{
    users(status: TEST)
}
```

GraphQL failed to find `TEST` in `UserStatus`, and returned an error without calling `resolve_users`:

```json
{
    "error": {
        "errors": [
            {
                "message": "Argument \"status\" has invalid value TEST.\nExpected type \"UserStatus\", found TEST.",
                "locations": [
                    {
                        "line": 2,
                        "column": 14
                    }
                ]
            }
        ]
    }
}
```

## Mapping to internal values

By default enum values are represented as Python strings, but Ariadne also supports mapping GraphQL enums to custom Python values.

Imagine posts on a social site that can have weights like "standard", "pinned" and "promoted":

```graphql
type Post {
    weight: PostWeight
}

enum PostWeight {
    STANDARD
    PINNED
    PROMOTED
}
```

In the database, the application may store those weights as integers from 0 to 2. Normally, you would have to implement a custom resolver transforming GraphQL representation to the integer but, you would have to remember to use this boilerplate in every resolver.

Ariadne provides an `EnumType` utility thats allows you to delegate this task to GraphQL server:

```python
import enum

from ariadne import EnumType

class PostWeight(enum.IntEnum):
    STANDARD = 0
    PINNED = 1
    PROMOTED = 2

post_weight = EnumType("PostWeight", PostWeight)
```

Include the `post_weight` instance in the list of types passed to `make_executable_schema`:

```python
schema = make_executable_schema(type_defs, some_type, post_weight)
```

This will make the GraphQL server automatically translate `PostWeight` between their GraphQL and Python values:

- If `PostWeight` enum's value is passed in argument of GraphQL field, Python resolver will be called with `PostWeight` member, like `PostWeight.PINNED`.
- If Python resolver for field returning GraphQL enum returns Enum member this value will be converted into GraphQL enum. Eg. returning `PostWeight.PROMOTED` from resolver will appear as `"PROMOTED"` in GraphQL result).
- If Python resolver for field returning GraphQL enum returns a value that's valid value of enum's member, this value will be converted into enum. Eg. returning `1` from resolver will appear as `"PINNED"` in GraphQL result).

Instead of `Enum` you may use plain `dict`:

```python
from ariadne import EnumType

post_weight = EnumType(
    "PostWeight",
    {
        "STANDARD": 0,
        "PINNED": 1,
        "PROMOTED": 2,
    },
)
```

Both `Enum` and `IntEnum` are supported by the `EnumType`.

repository: mirumee/ariadne-website

gha_language: JavaScript

repo_name: mirumee/ariadne-website


text:
---
title: High-latitude maps
excerpt: Plotting seal observations in Antarctica with R and ggplot2.
category: rstats
tags:
  - rstats
  - gis
  - seals
  - mapping
  - sf
header:
  overlay_image: /assets/images/featureSeals.png
  overlay_filter: 0.2
  caption: "Photo by Andrew Shiva - Wikimedia Commons"
---

> ## [Versión en español](https://luisdva.github.io/rstats/mapa-austral/)

This is a brief guide for making maps in R using `ggplot2` and `sf`, tailored specifically for showing high latitude regions effectively. The example data for this are points that show the distribution of two species of seal near the Antarctic peninsula.

We’ll download observation records for Weddell and Southern elephant seals from [gbif](https://www.gbif.org/){:target="_blank"} through `rgbif` and coastline data from Natural Earth with `rnaturalearth`, and plot them with a nice equal-area Azimuthal projection. Working with a suitable projection can really show the location of this study area in context. Projection parameters come from the [Projection Wizard web app](https://projectionwizard.org/){:target="_blank"}, based on John P. Snyder’s selection guideline and implemented by the Cartography and Geovisualization Group at Oregon State University.

Let’s go.

Load packages and download the seal observation data for Southern elephant seals and Weddell seals (querying only points for Chile, Argentina, and Antarctica, using their ISO codes and a semicolon to concatenate the query).

{% highlight r %}
# load packages
library(sf) # CRAN v0.9-1
library(rnaturalearth) # CRAN v0.1.0
library(ggplot2) # CRAN v3.3.0
library(dplyr) # [github::tidyverse/dplyr] v<IP_ADDRESS>02
library(rgbif) # CRAN v2.2.0
library(ggimage) # CRAN v0.2.8

# get occurrence data
mlseals <- occ_data(scientificName = "Mirounga leonina",country = "AR;CL;AQ")
wseals <- occ_data(scientificName = "Leptonychotes weddellii",country = "AR;CL;AQ")
mlsealsdat <- mlseals$data
wsealsdat <-wseals$data
{% endhighlight %}

Now we coerce the data into simple feature objects, simply by specifying which variables hold the coordinates, and what projection they’re in. Here I decided to subset the data by longitude and latitude, before projecting the combined `sf` object and creating a buffered bounding box that is used for plotting later on.

{% highlight r %}

# coerce to simple features objects and subset by lat/long
mlsealssf <-
  mlsealsdat %>% select(scientificName,decimalLongitude,decimalLatitude) %>%
  distinct() %>% dplyr::filter(decimalLatitude < -40) %>%
  dplyr::filter(decimalLongitude> -150 & decimalLongitude < -30 )
wsealssf <-
  wsealsdat %>% select(scientificName,decimalLongitude,decimalLatitude) %>%
  distinct() %>% dplyr::filter(decimalLatitude < -40) %>%
  dplyr::filter(decimalLongitude> -150 & decimalLongitude < -30 )

# bind into a single object
sealsdf <- bind_rows(mlsealssf,wsealssf)

# spatial transformation
seals_spat <- st_as_sf(sealsdf,coords = c("decimalLongitude","decimalLatitude"),crs=4326)
# proj from Projection Wizard website
sealsproj <- st_transform(seals_spat,"+proj=aea +lat_1=-67.64292238209752 +lat_2=-43.70345673689002 +lon_0=-60.46875")
# arbitrary buffer and bounding box
boundss <- st_bbox(st_buffer(sealsproj,500000))
xydatbuffer <- st_as_sf(st_as_sfc(boundss))
{% endhighlight %}

Next, we load country and state polygon outlines from [Natural Earth](https://www.naturalearthdata.com/){:target="_blank"} (a public domain map dataset) and project them.

{% highlight r %}

# load country and state outlines and transform
divpol <- ne_countries(scale = "large",country = c("Chile","Argentina","Antarctica"),returnclass = "sf") %>%
  st_transform("+proj=aea +lat_1=-67.64292238209752 +lat_2=-43.70345673689002 +lon_0=-60.46875")
regs <- ne_states(country = "Chile",returnclass = "sf") %>%
  st_transform("+proj=aea +lat_1=-67.64292238209752 +lat_2=-43.70345673689002 +lon_0=-60.46875")
regsAr <- ne_states(country = "Argentina",returnclass = "sf") %>%
  st_transform("+proj=aea +lat_1=-67.64292238209752 +lat_2=-43.70345673689002 +lon_0=-60.46875")
{% endhighlight %}

We are ready to plot all these data as layers in a `ggplot` call, but to illustrate the use of `ggimage` and to add some flair to the final map, we can make a table with locations and urls for images that can be drawn over the plot. In this case, little pictures of the two seals drawn by geneticist [Julia Saravia](https://twitter.com/JujuSaravia){:target="_blank"} and hosted online.

{% highlight r %}
# plotting
# tibble with location and source url for seal drawings
sealimgs <- tibble(x=c(-2023533 ,1725000),y=c(-8200000,-8200000),imgurl=c("https://raw.githubusercontent.com/luisDVA/luisdva.github.io/master/assets/images/pup/mirounga.png",
                                  "https://raw.githubusercontent.com/luisDVA/luisdva.github.io/master/assets/images/pup/wedd.png"))
{% endhighlight %}

Ready for plotting. `sf` does most of the work with the spatial objects. I made a few arbitrary choices for the colors and the plotting extent, mostly for aesthetics.

{% highlight r %}
ggplot()+
  geom_sf(data=divpol,size=0.2,color="black",fill="#e7d8c9")+
  geom_sf(data=regs,size=0.3,color="black",fill="transparent",lty=2)+
  geom_sf(data=regsAr,size=0.3,color="black",fill="transparent",lty=2)+
  geom_sf(data=sealsproj,pch=21,color="black",aes(fill=scientificName),size=3,stroke=1)+
  geom_image(data=sealimgs[1,],aes(x=x,y=y,image=imgurl),size=0.12)+
  geom_image(data=sealimgs[2,],aes(x=x,y=y,image=imgurl),size=0.14)+
    coord_sf(
    xlim = c(boundss["xmin"]-900000, boundss["xmax"]+900000),
    ylim = c(boundss["ymin"]-800000, boundss["ymax"]))+
  scale_x_continuous(expand = c(0,0))+
  scale_y_continuous(expand = c(0,0))+
  scale_fill_manual(values = c("blue","orange"),name="",labels=c("Southern elephant seal","Weddell seal"))+
  labs(x="",y="")+
  theme(panel.background = element_rect(fill="#c6def1"),
        text = element_text(family = "Loma",size = 18),
        legend.position = "bottom",legend.text = element_text(size = 24))
{% endhighlight %}

The plot:

photobomb!

Looks pretty good!
-feedback welcome-


text:
# vpx-encode

Rust interface to libvpx encoder

This crate provides a Rust API to use
[libvpx](https://en.wikipedia.org/wiki/Libvpx) for encoding images.

It it based entirely on code from [srs](https://crates.io/crates/srs).
Compared to the original `srs`, this code has been simplified for use as a
library and updated to add support for both the VP8 codec and (optionally)
the VP9 codec.

## Optional features

Compile with the cargo feature `vp9` to enable support for the VP9 codec.

## Example

An example of using `vpx-encode` can be found in the
[`record-screen`](https://crates.io/crates/record-screen) program. The source
code of record screen is
[here](https://github.com/astraw/vpx-encode/tree/master/record-screen).

## Contributing

All contributions are appreciated.

License: MIT

repository: astraw/vpx-encode

repo_name: astraw/vpx-encode


text:
# Promocode Generator

Generates save and unique (when used with AR) codes which can be handed to customers for promotional campaigns.

## Installation

Add this line to your application's Gemfile:

    gem 'promocode_generator'

And then execute:

    $ bundle

Or install it yourself as:

    $ gem install promocode_generator

## Usage

Without Rails:

  Use PromocodeGenerator.generate(length) to produce a promotional code of the desired length.

With Rails:

  In your model, use the promocode_attribute method to automatically generate a promotional code of length 8 in the before_create hook:

  ```ruby
  class Campaign < ActiveRecord::Base
    promocode_attribute :code
  end
  ```

  Basic uniqueness validation comes by default so that codes are generated until a non-existing code is found in the db. Provide the :reject_if option to customize:
  `promocode_attribute :code, :reject_if => Proc.new { |code| ModelA.where(:code => code).any? }`

## Contributing

1. Fork it
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Add some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
5. Create new Pull Request