text:
# ApiBundle

:package: Base classes and helper services to build API application via Symfony.

[![Scrutinizer Quality Score](https://img.shields.io/scrutinizer/g/stfalcon-studio/ApiBundle.svg?style=flat-square)](https://scrutinizer-ci.com/g/stfalcon-studio/ApiBundle/)
[![Build Status](https://img.shields.io/github/workflow/status/stfalcon-studio/ApiBundle/CI/main?style=flat-square)](https://github.com/stfalcon-studio/ApiBundle/actions?query=workflow%3ACI+branch%3Amain+)
[![CodeCov](https://img.shields.io/codecov/c/github/stfalcon-studio/ApiBundle.svg?style=flat-square)](https://codecov.io/github/stfalcon-studio/ApiBundle)
[![License](https://img.shields.io/packagist/l/stfalcon-studio/api-bundle.svg?style=flat-square)](https://packagist.org/packages/stfalcon-studio/api-bundle)
[![Latest Stable Version](https://img.shields.io/packagist/v/stfalcon-studio/api-bundle.svg?style=flat-square)](https://packagist.org/packages/stfalcon-studio/api-bundle)
[![Total Downloads](https://img.shields.io/packagist/dt/stfalcon-studio/api-bundle.svg?style=flat-square)](https://packagist.org/packages/stfalcon-studio/api-bundle)
[![StyleCI](https://styleci.io/repos/257974142/shield?style=flat-square)](https://styleci.io/repos/257974142)

## Installation

```composer req stfalcon-studio/api-bundle='0.7.0'```

#### Check the `config/bundles.php` file

By default, Symfony Flex adds this bundle to the `config/bundles.php` file.
In case when you ignored `contrib-recipe` during bundle installation it would not be added. In this case add the bundle manually.

```php
# config/bundles.php

return [
    // Other bundles...
    StfalconStudio\ApiBundle\StfalconApiBundle::class => ['all' => true],
    // Other bundles...
];
```

## Set Up Steps

#### Add mappings to Doctrine ORM config

```yaml
doctrine:
    orm:
        mappings:
            StfalconApiBundle:
                is_bundle: true
                type: annotation
```

#### Setup a directory for JSON schemas

The default directory is `./src/Json/Schema/`. Or you can change it via bundle configuration.

```yaml
stfalcon_api:
     json_schema_dir: '%kernel.project_dir%/src/Json/Schema/'
```

## Contributing

Read the [CONTRIBUTING](https://github.com/stfalcon-studio/ApiBundle/blob/master/.github/CONTRIBUTING.md) file.

repository: metalslave/ApiBundle

repo_name: metalslave/ApiBundle


text:
# warping-surfaces
<h5> Description </h5>
This matlab program warps 3d surface function and visually display it on 3d graph.

Key Features
Program in Matlab
Integration and interpolation between different surface function

This progam creates a neat animation between four surface function.
The numerical and symbolic integration are shown alongside the changing surface!

<h5> Video Demonstration </h5>
https://youtu.be/lpwA3oiPLaQ

branch_name: refs/heads/master

extension: md

license_type: permissive


text:
# Exploring alternatives to age pyramids

See [this post](https://gorelik.net/2020/09/02/exploring-alternatives-to-population-pyramids/) for details

license: MIT


text:
## Sorting
#### [Download as zip](https://grapecity.github.io/DownGit/#/home?url=https://github.com/GrapeCity/ComponentOne-WinForms-Samples/tree/master/NetFramework\FlexGrid\CS\Sorting)
____
#### Shows several sorting techniques.
____
1. Use the EditorSetup event to sort items in a drop down list editor.
2. Use the BeforeSort event to add an 'AsDisplayed' flag to the sort flags.
3. Use a custom IComparer to sort the grid.

repository: KaptenJon/ComponentOne-WinForms-Samples

repo_name: KaptenJon/ComponentOne-WinForms-Samples


text:
# Backlog

This file will serve as the backlog of bugs, user stories, and tasks.

## Adding and Removing Items

Please feel free to add more ehancement requests or report bugs. If you contribute towards any of the item in this file, remove it from this file and add it to the [CHANGELOG.md](CHANGELOG.md)

## Pending

### Types

- New Feature
- Feature Enhancement
- Bug Fix

| ID | Type | Description | Status |
|---|---|---|---|
| 1 | New Feature | Load a rules library from a datastore | done |

repository: SSAgov/object-property-rule-engine

repo_name: SSAgov/object-property-rule-engine


text:
chip8emu
========

A Chip-8 interpreter I wrote in a weekend for fun.

You may view a video of it running here:
https://www.youtube.com/watch?v=Q8zCpPjTbS0

The code has three files:

* Main.java
The entry point. Creates a MainWindow instance, creates an emulator instance, then runs the emulator.

* MainWindow.java
The window displayed to the user showing the screen of the Chip-8.

* Emulator.java
The main code for interpreting the Chip-8 system.

It needs a lot of clean-up, which I'm going to do eventually.

Thanks for viewing!

(Released under the zlib license. Basically, do whatever you want except say that you wrote the original software / code, and everything'll be OK.)

license: Zlib


text:
# Login Form Experiment

This is a set of very basic pages to help validate potential new <abbr title="W3C Accessibility Guidelines">WCAG</abbr> 3 conformance. They purposely have numerous accessibility defects in them.

Please note that none of the forms submit any kind of data.

repository: fstrr/login-form-experiment

branch_name: refs/heads/main

repo_name: fstrr/login-form-experiment


text:
# Proto for Vercel Node APIs

[![npm](https://badgen.net/npm/v/@peterjskaltsis/proto)](https://www.npmjs.com/package/@peterjskaltsis/proto)

This is a super-simple (not production ready) method/middleware routing layer designed for running [Node.js](https://nodejs.org/api/http.html) APIs with [Vercel](https://vercel.com/).

Example repositories:

- [Inv-noice API](https://www.github.com/peterjskaltsis/inv-noice/)
- [Vercel Node.js API](https://www.github.com/peterjskaltsis/experiment-node-vercel/)

This can probably also be used with [Next.js](https://nextjs.org/) too.

## Installation

```bash
npm install @peterjskaltsis/proto
// or
yarn add @peterjskaltsis/proto
```

## Usage

`proto` allows you to define functions to be called for different HTTP methods on a single endpoint, in a clean and simple way.

```javascript
// api/index.js
import proto from '@peterjskaltsis/proto'

const router = proto()

router.get((req, res) => {
  res.send('Hello there!')
})

router.post((req, res) => {
  res.json({ hello: 'there!' })
})

router.put(async (req, res) => {
  res.end('async/await enabled')
})

router.patch(async (req, res) => {
  throw new Error('Errors can be caught and handled.')
})

export default router
```

## Contributing

Please see my [contributing.md](CONTRIBUTING.md).

## License

[MIT](LICENSE)

repository: peterjskaltsis/proto

repo_name: peterjskaltsis/proto


text:
---
layout : post
title : K-Means Clustering with Visualization tool
desc :
Pascal

Machine Learning

 This algorithm gives labels to unlabeled datas and put them into group.
img  : ../public/post-assets/Kmeans/title.gif
---

Pascal

Machine Learning

**Content:**

- [1. Objective](#1-objective)
- [2. Introduction](#2-introduction)
- [3. Mathematical Analysis](#3-mathematical-analysis)
	- [3.1. Input and Output](#31-input-and-output)
	- [3.2. Lost Function and Optimization Problem](#32-lost-function-and-optimization-problem)
	- [3.3. Solving Optimization Problem](#33-solving-optimization-problem)
		- [3.3.1. Fixed $$\mathbf{M}$$, center of observation group](#331-fixed-%5Cmathbfm-center-of-observation-group)
		- [3.3.2. Fixed $$\mathbf{Y}$$, label vector of each observation](#332-fixed-%5Cmathbfy-label-vector-of-each-observation)
	- [3.4. Algorithm Summary and Flowchart](#34-algorithm-summary-and-flowchart)
		- [3.4.1. Summary](#341-summary)
		- [3.4.2. Flowchart](#342-flowchart)
	- [3.5. Discussion](#35-discussion)
		- [3.5.1. Convergence](#351-convergence)
		- [3.5.2. Sensitiveness to initial cluster](#352-sensitiveness-to-initial-cluster)
- [4. Application in Data Compression](#4-application-in-data-compression)
- [5. Conclusion](#5-conclusion)
- [6. Acknowledgement](#6-acknowledgement)
	- [6.1. Reference](#61-reference)
- [7. Source Code and PDF report](#7-source-code-and-pdf-report)
	- [7.1. Declare Data types](#71-declare-data-types)

## 1. Objective
After reading this blog, you will be able to
1. Understand a clustering method (unsupervised learning) namely K-means algorithm from mathematical perspective.
2. **Source code** of visualization tool (written in **Pascal**), demo below.
3. **Source code** of image compression, image segmentation tool, applied K-Means Algorithm (written in **Pascal**).
<embed class="video" width="1600" height="400" style="width: 100%" src="/public/post-assets/Kmeans/demo.mp4" scale="aspect" controller="true">

Demo video

##### 1.1. ***Abstract***
Clustering problem is a task of dividing a set of objects (also called members) into different groups (called clusters) based on object's characteristics. Members of a group will have more similarities in comparison with those in other group. This report discusses a traditional clustering method called K-Means algorithm from mathematical perspective. Additionally, an experiment is provided to examine the algorithm in two dimensional space then an application in image compressing.

## 2. Introduction
Clustering problems arise in many different applications: machine learning data mining and knowledge discovery, data compression and vector quantization, pattern recognition and pattern classification.

The goal of K-Means Algorithm is to correctly separating objects in a dataset into groups based on object's properties. For instance, objects could be house and their properties are size, number of floor, location, power consumption per year, etc. The goal is to classify house dataset into groups which are luxury, average, poor. In that case, all properties of houses have to be processed to turn into number to create a vector, this process is called vectorization.

Another example, take each points in a panel as a objects and each object has two properties which are x-axis and y-axis location, with input $$K=3$$. The algorithm correctly finds the cluster ([fig 1](#fig1)).

Fig 1: Running K-means algorithm to find 3 cluster

## 3. Mathematical Analysis

### 3.1. Input and Output
The K-Means Algorithm takes a set of observations $$X=[x_1,x_2,...,x_N]$$ $$\in$$ $$R^{d \times N}$$ where each observation is a $$d$$-dimensional vector, $$N$$ is the number of observations (members) and the number of group $$(K, K<N)$$ as two input. The algorithm outputs the center of $$K$$ group $$[m_1,m_2,...,m_K] \in R^{d \times K}$$ and the index or name of group that each member belonged to (label of the group).

### 3.2. Lost Function and Optimization Problem
Suppose $$x_i$$ $$(i\in[1,N])$$ belong to cluster $$k$$ $$(k \in [1,K]$$, the lost value of observation $$x_i$$ is the distance from observation $$x_i$$ to center $$m_k$$ in euclidean space, defined by $$(x_i-m_k)$$.
Let's $$y_i=[y_{i1},y_{i2},...,y_{iK}]$$ be the label vector of each observation $$x_i$$, $$y_{ik}=1$$ if $$x_i$$ belongs to group $$k$$ and $$y_{ij}=0$$ $$\forall j\neq k$$.
Label vector of each observation contains only one digit $$1$$ because each observation belongs to only one group which leads to the following equation.
\begin{equation} \tag{1}\label{eq:1}
\sum_{k=1}^{K} y_{ik} = 1
\end{equation}
The objective is to minimize the within-cluster sum of squares (variance), also known as square errors of, where each square error of an observation $$x_i$$ from group $$m_k$$ is defined by:
\begin{equation} \tag{2}\label{eq:2}
||x_{i}-m_{k}||^2=y_{ik}||x_{i}-{m_k}||^2
\end{equation}
From the equation \eqref{eq:1}, sum of all elements in a label vector is equal to 1. The square error of an observation is:
\begin{equation} \tag{3}\label{eq:3}
y_{ik}||x_{i}-{m_k}||^2=\sum_{j=1}^{K}y_{ij}||x_{i}-m_{j}||^2
\end{equation}
The square error of all observation is the sum of every square error of in the given set of observation. The goal is minimize the lost function, equation \eqref{eq:4} where $$Y=[y_1,y_2,...,y_N]$$ be the matrix contains all label vector of $$N$$ observation and $$M=[m_1,m_2,...,m_K]$$ be the center of $$K$$ groups (clusters).
\begin{equation} \tag{4}\label{eq:4}
f(Y,M)=\sum_{i=1}^{N}\sum_{j=1}^{K}y_{ij}||x_i-m_j||^2
\end{equation}
The objective is also to find the center and label vector of each observation which are Y and M, the two outputs that are mentioned in [input and output](#input-and-output).
\begin{equation} \tag{5}\label{eq:5}
Y,M=argmin_{Y,M}\sum_{i=1}^{N}\sum_{j=1}^{K}y_{ij}||x_i-m_j||^2
\end{equation}

### 3.3. Solving Optimization Problem
There are two variable in equation \eqref{eq:5} which are center of each group of observation and label vector of each observation. The problem could be solved by fixed each variable then minimize the other variable.

#### 3.3.1. Fixed $$\mathbf{M}$$, center of observation group
Because all centers ($$M$$) are constant, the objective is to correctly identify label vector which is identifying the group that each observation belonged to so that the square error in equation \eqref{eq:4} is minimized.
\begin{equation} \tag{6}\label{eq:6}
y_i = argmin_{y_i}\sum_{j=1}^{K}y_{ij}||x_i-m_j||^2
\end{equation}
Retrieving from equation \eqref{eq:1}. Because only one element in vector $$y_i$$ $$i\in[1,K]=1$$. Equation \eqref{eq:6} could be rewritten as:
\begin{equation} \tag{7}\label{eq:7}
j=argmin_{j}||x_i-m_j]]^2
\end{equation}
The value of $$||x_i-m_j||^2$$ is the square of distance from observation to center of group in euclidean space. Concretely, when M is constant, equation \eqeqref{eq:7} shows that minimizing the sum of square error could be achieved by choosing label vector so that the center are closest to observation.

#### 3.3.2. Fixed $$\mathbf{Y}$$, label vector of each observation
When label vector ($$Y$$) are constant, the objective is to correctly identify the center so that the square error in equation \eqref{eq:4} is minimized. In this case, the optimization problem in equation \eqref{eq:5} could be rewritten by the following equation.
\begin{equation} \tag{8}\label{eq:8}
m_j=argmin_{m_j}\sum_{i=1}^{N}y_{ij}||x_i-m_j||^2
\end{equation}
The equation \eqref{eq:8} is a convex function and differentiable for each $$i\in[1,N]$$. Hence equation \eqref{eq:8} could be solved by finding the root of the partial derivative function. This approach will make sure that the root is the the value that make the function reach a optimum.
Let's $$g(m_j)=\sum_{i=1}^{N}y_{ij}||x_i-m_j||^2$$ (retrieving from equation \eqref{eq:8} and take the partial derivative of $$g(m_j)$$:
\begin{equation} \tag{9}\label{eq:9}
\frac{\partial g(m_j)}{\partial m_j}=2 \sum_{i=1}^{N}y_{ij}(m_j-x_i)
\end{equation}
The equation \eqref{eq:9} is equal 0 is equivalent to:
\begin{equation} \tag{10}\label{eq:10}
m_j\sum_{i=1}^{N}y_{ij}=\sum_{i=1}^{N}y_{ij}x_{i}
\end{equation}
\begin{equation} \tag{11}\label{eq:11}
\Leftrightarrow m_j=\frac{\sum_{i=1}^{N}y_{ij}x_{i}}{\sum_{i=1}^{N}y_{ij}}
\end{equation}
The value of $$y_{ij}=1$$ when observation $$x_i$$ belongs to group $$m_j$$. Hence, the denominator of equation \eqref{eq:11} $$\sum_{i=1}^{N}y_{ij}$$ is the number of observations that belonging to group $$m_j$$ and the nominator $$\sum_{i=1}^{N}y_{ij}x_{i}$$ is the sum of all observations belonging to group $$m_j$$.
In other word, when Y is constant, the square errors could be minimize by assigning the centers to the means of observations in the groups that the observations belonging to.

### 3.4. Algorithm Summary and Flowchart

#### 3.4.1. Summary
The algorithm can be done by continuously constantize Y and M, one each a time as discussed in [Fixed $$M$$](#fixed-%5C%5C%5Cmathbfm%5C%5C-center-of-observation-group)and [Fixed $$Y$$](#fixed-%5C%5C%5Cmathbfy%5C%5C-label-vector-of-each-observation).

Step 1. Select K points as cluster centers randomly. (K is the number of group/cluster)

Step 2.	Assign all data points to corresponding cluster center created in step 1 based on Euculidean distance.

Step 3.	Move cluster center, new cluster centers are means of coordination of all data points among same group.

Step 4. Repeat step 2 until convergence (when all cluster centers remain unchanged after an iteration).

Step 5.	Clustering process is done, if we have new data points, caculate distance between that data point to the cluster centers that we found and see what group the data point belongs to.

#### 3.4.2. Flowchart
The following chart describe K-Means Algorithm.

Fig 2: K-Means Algorithm Flowchart

### 3.5. Discussion

#### 3.5.1. Convergence
The algorithm will stop after a certain number of iteration because the square error function is a strictly decreasing sequence and the square error is always greater than 0. But this algorithm will not make sure that it will find a global optimum because solving the equation \eqref{eq:8} by finding the root when the partial derivative is equal 0 will only return the value for local optima but not make sure that local optima will be a global minimum.
The following figure describe a case where poorly seeding leads to a local optimum.

Fig 3: Poorly seeded leads to inaccurate result

This result leads to a very high square error compared to the result in [fig 1](#fig1).

#### 3.5.2. Sensitiveness to initial cluster
K-Means algorithm requires careful seeding, which means the final result is very sensitive to the initial value of cluster. Numerous efforts have been made to improving K-Means clustering algorithm due to its drawbacks [$$^[2]$$](#ref2).

## 4. Application in Data Compression

Left image: 900 KB, Right image (K=16 color): 70KB

More detail will be described in the upcomming blog Image Compression

## 5. Conclusion
K-Means Algorithm could be very simple and quick to be implemented, the clustering problems where all clusters are centroids and separated can be solved by the algorithms.

This report doesn't come with new idea to improve the effectiveness of the algorithm, the aim of the report is to introduce the readers to a basic clustering method with some visual examples on 2-dimensional and 3-dimensional data.

## 6. Acknowledgement
The ideal of K-means Algorithm is leaned by me from [Machine Learning course](https://www.coursera.org/learn/machine-learning) by [Prof. Andrew Ng](http://www.andrewng.org/) from [coursera](https://www.coursera.org).

The mathematical approach to solve the [optimization problem](#lost-function-and-optimization-problem) is learned by me from the article on K-means algorithm ([machine learning fundamental](https://machinelearningcoban.com/2017/01/01/kmeans/)) by [Tiep Vu](http://www.personal.psu.edu/thv102/).

### 6.1. Reference

[1]. Joaqun Prez Ortega, Ma. Del, Roco Boone Rojas, and Mara J.Somodevilla ***"Research issues on k-means algorithm: An experimental trial using matlab"***.

[2]. Arthur,~D.,~Vassilvitskii,~S., 2016. ***k-Means++: The Advantages of Careful Seeding***, Technical Report, Stanford.

## 7. Source Code and PDF report
This part will guide you through all the code in this program, if you understand the algorithm, you could implement it on any programming language not just pascal.
 Declare Data types
```pascal
const
  POINT_RADIUS = 5;
  X_DIFF = 20; //location panel and point location
  Y_DIFF = 160; //location panel and point location

type
  // type used for application
  RGB = array[0..2] of Single; // x-axis  // store red, green and blue of an image
  CollumnPixels = array of RGB; // y-axis, rowPixels
  imgArray = array of CollumnPixels; // 3-dimensional array used to store red, green, blue color of each pixel in an image
  IntArray = array of Integer; // label

  SingleColorArray = array of Color; // 1 dimensional array used to store color
  ColorArray = array of SingleColorArray; // 2 dimensional array used to store color

  // typed used for Visualization
  PointsRecord = Record
    x    : Single; // x-axit on panel
    y    : Single; // y-axit on panel
    idx  : Integer; // store label of each point, instead of create another array or enumeration to store it
  end;

  PointsArray = array of PointsRecord; // Store all the points on panel and the init points
```
In the panel, each point's data type is `PointsRecord`, it has `x` and `y` properties, they are the x-axis and y-axis of each point, `idx` is the label of each point, `idx` keep changing overtime during the iterations.

Continue Reading... Please download source code!

[Download PDF Report](/public/post-assets/Kmeans/Report.pdf)
Full code in Pascal: https://github.com/DungLai/Image-Compression-Segmentation

gha_language: TeX

gha_license_id: NOASSERTION


text:
# SylectralSender
Send sms texts using Twillio as SMS service

## Installation
1. Download the folder and serve it to the web.
2. Use database.sql to configure the needed tables.
3. Set the database connection in **database_connection/connection.php**
   * Open the connection.php file in the database_connection folder
   * Set the servername, the username, the password and the dbname
4. Set your values in the following files:
   * **incoming_message.php**: replace domain.com/ with the path to this folder
   * **mailer.php**: set the mail recipient to receive the incoming messages
5. Set the Account Sid and Auth Token from twilio.com/user/account in the cron_check.php file
6. Set your twillio application to call **/incoming_message.php** when an incoming message is detected.

## Usage
### Outbound
1. Send messages by calling **prepare_message.php** with following GET variables
   * `text`: This variable takes the message to send to the recipient. The message can contain all ascii characters.
   * `contact`: This variable contains the mobile phone number of the recipient. The following format should be used: ##4123123 (Where ## is the country code without leading zeroes.). E.g.: 324123123 (Belgian mobile number)
   * `name`: This variable is optional and contains the name of the recipient. If the name is given, the mail sent to you will contain the name of the recipient.
   * `date`: This variable is optional and contains the timestamp of when the message should be sent. Format: YYYY-MM-DD HH:mm.

2. Set a cron job to call **cron_check.php**

### Inbound
1. Inbound messages are sent to the mail address set in the **mailer.php** file.

## Development
I'm actively working on this project. If you have questions, ideas for enhancement or if you find any bugs in the scripts, feel free to open an issue!

## Copyright
This project is licensed under the MIT license. Please feel free to use this code, modify it or redistribute it.


text:
# pipetteshow

## Project setup

```
npm install
```
Remember to put your Google Drive API-Keys into a .env.local-file at project root level (see .env.local.example).

### Compiles and hot-reloads for development
```
npm run serve
```

### Compiles and minifies for production
```
npm run build
```

### Lints and fixes files
```
npm run lint
```

### Customize configuration
See [Configuration Reference](https://cli.vuejs.org/config/).

repository: DrMarkEy/PipetteShow

repo_name: DrMarkEy/PipetteShow


text:
# [MusicTuner](http://alexa.amazon.com/#skills/amzn1.echo-sdk-ams.app.22ee8098-8c05-495c-bccf-8dff2fd7bb14)
![2.5 stars](../../images/ic_star_black_18dp_1x.png)![2.5 stars](../../images/ic_star_black_18dp_1x.png)![2.5 stars](../../images/ic_star_half_black_18dp_1x.png)![2.5 stars](../../images/ic_star_border_black_18dp_1x.png)![2.5 stars](../../images/ic_star_border_black_18dp_1x.png) 3

To use the MusicTuner skill, try saying...

* *Alexa, ask music tuner to play a middle c.*

* *Alexa, ask music tuner to give me a long a.*

* *Alexa, ask music tuner to play a d.*

This app is intended to help while tuning instruments, or any other situation where a pure note is needed. It does not do chords. The default is to play a 10 second tone, but if the user requests a 'long note' then it will play for 30 seconds.

***

### Skill Details

* **Invocation Name:** music tuner
* **Category:** null
* **ID:** amzn1.echo-sdk-ams.app.22ee8098-8c05-495c-bccf-8dff2fd7bb14
* **ASIN:** B01GSE4R8M
* **Author:** Erik Lane
* **Release Date:** June 17, 2016 @ 06:45:34
* **In-App Purchasing:** No


text:
# 2. Use Apache Kafka as Integration Backbone

Date: 2019-04-30

## Status

Accepted

## Context

Multiple different platforms currently integrate with each other in custom way, leading to multiple different integration mechanisms and approaches. It is required to standardize the way integration is implemented, while ensuring loose coupling between different systems.

## Decision

Use Apache Kafka based integration backbone that will provide the required infrastructure to implement event based integrations instead of pull/push type of integrations.

## Consequences

- All Frappe based platforms can use a standard reusable application for publishing and consuming the events
- All producer platforms will only be required to decide on the events to be published and not how these should be consumed or processed.
- Individual consumer platforms will only be required to decide on the action to be taken when a particular event is received, ideally, irrespective of the source of the event.


text:
# Camel case property names

This sample uses a [CamelCaseNamingStrategy](/api/newtonsoft/json/serialization/camelcasenamingstrategy/) specified using a contract resolver to camel case serialized property names.

## Sample

:::code source="../../../Src/Newtonsoft.Json.Tests/Documentation/Samples/Serializer/NamingStrategyCamelCase.cs" region="Types" title="Types" :::

:::code source="../../../Src/Newtonsoft.Json.Tests/Documentation/Samples/Serializer/NamingStrategyCamelCase.cs" region="Usage" title="Usage" :::

gha_language: C#

gha_license_id: MIT


text:
---
title: "addGlobalNotification (Client API reference) in model-driven apps| MicrosoftDocs"
description: Includes description and supported parameters for the addGlobalNotification method.
ms.date: 04/21/2021
ms.service: powerapps
ms.topic: "reference"
author: "Nkrb"
ms.author: "nabuthuk"
manager: "kvivek"
search.audienceType:
  - developer
search.app:
  - PowerApps
---
# addGlobalNotification (Client API reference)

[!INCLUDE[./includes/addGlobalNotification-description.md](./includes/addGlobalNotification-description.md)]

## Syntax

`Xrm.App.addGlobalNotification(notification).then(successCallback, errorCallback);`

## Parameters

Name
Type
Required
Description

notification
Object
Yes
The notification to add. The object contains the following values:

action: (Optional) Object. Contains the following values:

actionLabel: (Optional) String. The label for the action in the message.

eventHandler: (Optional) Function reference. The function to execute when the action label is clicked.

level: Number. Defines the level of notification. Valid values are:

1: Success

2: Error

3: Warning

4: Information

message: String. The message to display in the notification.

showCloseButton: (Optional) Boolean. Indicates whether or not the user can close or dismiss the notification. If you don't specify this parameter, users can't close or dismiss the notification by default.

type: Number. Defines the type of notification. Currently, only a value of 2 is supported, which displays a message bar at the top of the app.

successCallback
Function
No

A function to call when notification is displayed. A GUID value is passed to uniquely identify the notification. You can use the GUID value to close or dismiss the notification using the clearGlobalNotification method.

errorCallback
Function
No
A function to call when the operation fails.

## Return Value

On success, returns a promise object containing a GUID value to uniquely identify the notification as described earlier in the description of the **successCallback** parameter.

## Examples

### Display an error notification that can't be closed or dismissed by user

```JavaScript
// define notification object
var notification =
{
  type: 2,
  level: 2, //error
  message: "Test error notification"
}

Xrm.App.addGlobalNotification(notification).then(
    function success(result) {
        console.log("Notification created with ID: " + result);
        // perform other operations as required on notification display
    },
    function (error) {
        console.log(error.message);
        // handle error conditions
    }
);
```

This is how the error notification will appear in the app:

![Example error notification](media/add-global-error.png "Example error notification")

### Display a warning notification that can be closed or dismissed by user

```JavaScript
// define notification object
var notification =
{
  type: 2,
  level: 3, //warning
  message: "Test warning notification",
  showCloseButton: true
}

Xrm.App.addGlobalNotification(notification).then(
    function success(result) {
        console.log("Notification created with ID: " + result);
        // perform other operations as required on notification display
    },
    function (error) {
        console.log(error.message);
        // handle error conditions
    }
);
```

This is how the warning notification will appear in the app:

![Example warning notification](media/add-global-warning.png "Example warning notification")

### Display an information notification with a "Learn more" link that can be clicked by users

```javascript
// define action object
var myAction =
{
  actionLabel: "Learn more",
  eventHandler: function () {
        Xrm.Navigation.openUrl("https://docs.microsoft.com/powerapps/");
        // perform other operations as required on clicking
    }
}

// define notification object
var notification =
{
  type: 2,
  level: 4, // information
  message: "Test information notification",
  action: myAction
}

Xrm.App.addGlobalNotification(notification).then(
    function success(result) {
        console.log("Notification created with ID: " + result);
        // perform other operations as required on notification display
    },
    function (error) {
        console.log(error.message);
        // handle error conditions
    }
);
```

This is how the information notification will appear in the app:

![Example information notification](media/add-global-information.png "Example information notification")

### See also

[clearGlobalNotification](clearGlobalnotification.md)

[!INCLUDE[footer-include](../../../../../includes/footer-banner.md)]

repository: phuocle/Dynamics-Crm-DevKit

repo_name: phuocle/Dynamics-Crm-DevKit


text:
# SharpGenTools

[![Build Status](https://dev.azure.com/SharpGenTools/SharpGenTools/_apis/build/status/SharpGenTools?branchName=main)](https://dev.azure.com/SharpGenTools/SharpGenTools/_build/latest?definitionId=1&branchName=main) [![MyGet Pre Release](https://img.shields.io/myget/sharpgentools/vpre/SharpGenTools.Sdk.svg)](https://www.myget.org/feed/Packages/sharpgentools) [![NuGet](https://img.shields.io/nuget/v/SharpGenTools.Sdk.svg)](https://www.nuget.org/packages/SharpGenTools.Sdk) [![Docs](https://readthedocs.org/projects/sharpgentools/badge/?version=latest)](https://sharpgentools.readthedocs.io/en/latest/) [![codecov](https://codecov.io/gh/SharpGenTools/SharpGenTools/branch/main/graph/badge.svg)](https://codecov.io/gh/SharpGenTools/SharpGenTools) [![CodeFactor](https://www.codefactor.io/repository/github/sharpgentools/sharpgentools/badge)](https://www.codefactor.io/repository/github/sharpgentools/sharpgentools)

Accurate and high performance C++ interop code generator for C#.

## Features

* Accurate, fast code-gen for C++ and COM interfaces from their C++ headers.
* No dependencies on .NET Runtime COM support
* Supports passing code-gen information through MSBuild project and package references
* Pluggable Runtime Library Name and runtime support type names (default library is SharpGen.Runtime)
  * Projects that implement their own SharpGenTools runtime support classes can add mapping rules to ensure that their generated code uses their runtime support library.

## Components in this Repo

* SharpGen
  * The code-gen engine that runs CastXML to parse the C++ and then generates the C# interop code.
  * API for SharpGen SDK plugins, e.g. documentation providers.
* SharpGenTools.Sdk
  * MSBuild tooling to integrate SharpGen directly into projects.
  * Build-time code generation plugin platform.
* SharpGen.Runtime
  * Runtime support classes for code generated by SharpGen.
* SharpGen.Runtime.COM
  * Mapped classes for the core interfaces in the COM and Windows Runtime.

## Requirements

### To Use

* SDK-style (CPS) MSBuild projects
* .NET environment, at least one of the following:
  * .NET SDK (6 or newer)
  * .NET Core SDK (3.1 or newer)
  * Visual Studio 2017.3 with desktop .NET workload, .NET Framework 4.7.2 SDK or newer
* Make any mapping files a `SharpGenMapping` item in your `.csproj`.

### To Build

* .NET SDK: 6.0 or newer.
* CMake: 3.0 or newer.
* SDK tests require x64 Windows, VS2019 with x86 and x64 C++ compilers, recent PowerShell version.

## Nightly (CI) builds
Add SharpGenTools MyGet feed to your NuGet.config:

```xml
<configuration>
  <packageSources>
    <!-- ... -->
    <add key="SharpGenTools" value="https://www.myget.org/F/sharpgentools/api/v3/index.json" />
  </packageSources>
</configuration>
```

repository: SharpGenTools/SharpGenTools

gha_language: C

repo_name: SharpGenTools/SharpGenTools


text:
# Rebalancing Tool
Rebalancing tool is stateful Processing Unit that tries to keep other PUs evenly distributed across the grid.
Tool maintains PU zones configuration, GSA zones configuration to find overloaded/underloaded GSAs and relocates
PU instances to make GSAs equally loaded. Both stateful and stateless processing units can be balanced with a help of
Rebalancing tool. In case of stateful PU tool takes care of high availability and doesn't put primary and backup to the same node.
Tool rebalances first primaries by locating empty and low loaded containers, and then, according to the primaries hosts, rebalances backups.

### Config
Rebalancing tool doesn't require any additional configuration. It just has to be deployed (grid-rebalancing.jar file) to the grid
and then tool finds locators/groups from system properties or grid configuration.

### Rebalancing tool runs:
* when Rebalancing Tool is deployed;
* when GSA is added to the grid;
* when GSA is removed from the grid.
* when tool switched on (only for XAP Manager)

### Plugin
If XAP manager is used, with REST API and ZooKeeper, the plugin submodule can be built and put to specific directory to add
ability enable/disable rebalancing tool without redeploy. Plugin also persists tool state (ON/OFF) into ZooKeeper, so if
the tool will be restarted, it's state saved.
Used endpoints:

http://<xap_host>:8090/v1/controller/start?appName=grid-rebalancing

http://<xap_host>:8090/v1/controller/stop?appName=grid-rebalancing

where appName parameter is the actual name of tool (if tool was deployed with -override-name option and has different name).
By hitting start endpoint, tool will be triggered to rebalance instances.

If rebalancer was disabled, the rebalancing can be done manually, using endpoint

http://<xap_host>:8090/v1/controller/rebalance?appName=grid-rebalancing

By sending GET request the app will return current grid state (balanced = true or false), and using POST request on the same URL,
single rebalance will be initiated. The grid is in unbalanced state if at least 1 processing unit is unbalanced on his primaries or backups.

### API within jar
If XAP version is lower than 12.2 and REST API extension isn't supported, api.jar can come in handy. This jar is a simple web server,
that supports the same endpoint as plugin, but doesn't require 12.2 (or higher) version. It can be start from terminal using plain command

java -jar path/to/api.jar 1 2 ,

where placeholder 1 suppose to be port, that will be listened by server, and placeholder
2 is a lookup locator address in format host:port.

> The patterns and source code available in this repository are provided as is. While most of them are used in real life use cases and productions environments, and GigaSpaces is making its best effort to keep them up to date, they should not be considered as fully productized artifacts, and you should test them in your own environment before using them.

The Apache v2 license applies. Here's [the fine print](../license.txt).


text:
# Quickstart for belvo-python

[![Deploy](https://www.herokucdn.com/deploy/button.svg)](https://heroku.com/deploy?template=https://github.com/belvo-finance/quickstart/tree/master/python)

To run this application locally, first install it and then run the app with the instructions below.

## Installing the quickstart app

```
git clone https://github.com/belvo-finance/quickstart.git
cd quickstart/python

pip install -r requirements.txt

cd client
npm install
cd ..
```

## Run the backend
```
# Start the Quickstart app with your sandbox API keys from the Dashboard
# https://dashboard.belvo.co/

BELVO_SECRET_ID='BELVO_SECRET_ID' \
BELVO_SECRET_PASSWORD='BELVO_SECRET_PASSWORD' \
BELVO_ENV=sandbox \
python server.py
```

## Run the client
```
cd client
npm run serve

# Configure the widget to Belvo by adding your private
# IP and the port as a new URL in the Connect Widget
# area (e.g: <IP_ADDRESS>:8080)
# https://dashboard.belvo.co/configuration/widget/
```

## Create a link
The quickstart app is using the `Sandbox` environment by default.

**You can use any combination of username and password to create a link with an institution in `Sandbox`.**

Read more about creating links in sandbox in our [dedicated guide](https://developers.belvo.co/docs/test-in-sandbox#create-links-in-sandbox).

repository: jaraujoduarte/quickstart

repo_name: jaraujoduarte/quickstart


text:
# Exercise-7

In this exercise, you'll work with a dataset of **NBA team
statistics** from a recent season. Here, you'll practice data
wrangling using DPLYR functions while reading in real world data.

As in previous exercises, you should fork and clone this repository,
follow the instructions in `exercise.R` file, then `add` and `commit`
your changes so that you can `push` them back up to Github.


text:
### 🌇 GitHub Skyline 3D calendar

> ⚠ This plugin significantly increase file size, prefer using it as standalone.

The *skyline* plugin lets you display your 3D commits calendar from [skyline.github.com](https://skyline.github.com/).

This uses puppeteer to generate collect image frames, and use CSS animations to create an animated rendering (GIF images are not animated in GitHub flavored markdown rendering which is why this design choice was made).

#### ℹ Examples workflows

[➡ Available options for this plugin](metadata.yml)

```yaml
- uses: lowlighter/metrics@latest
  with:
    # ... other options
    plugin_skyline: yes
    plugin_skyline_year: 0            # Set to 0 to display current year
    plugin_skyline_frames: 60         # Use 60 frames (half-loop)
    plugin_skyline_quality: 0.5       # Set image quality
    plugin_skyline_compatibility: yes # Support additional browsers (⚠ increases file size and reduce optimization)
```


text:
### [CVE-2019-14053](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-14053)
![](https://img.shields.io/static/v1?label=Product&message=Snapdragon%20Auto%2C%20Snapdragon%20Compute%2C%20Snapdragon%20Consumer%20Electronics%20Connectivity%2C%20Snapdragon%20Consumer%20IOT%2C%20Snapdragon%20Industrial%20IOT%2C%20Snapdragon%20IoT%2C%20Snapdragon%20Mobile%2C%20Snapdragon%20Voice%20%26%20Music%2C%20Snapdragon%20Wearables%2C%20Snapdragon%20Wired%20Infrastructure%20and%20Networking&color=blue)
![](https://img.shields.io/static/v1?label=Version&message=n%2Fa&color=blue)
![](https://img.shields.io/static/v1?label=Vulnerability&message=Buffer%20Over-read%20Issue%20in%20HLOS%20Data&color=brighgreen)

### Description

When attempting to create a new XFRM policy, a stack out-of-bounds read will occur if the user provides a template where the mode is set to a value that does not resolve to a valid XFRM mode in Snapdragon Auto, Snapdragon Compute, Snapdragon Consumer Electronics Connectivity, Snapdragon Consumer IOT, Snapdragon Industrial IOT, Snapdragon IoT, Snapdragon Mobile, Snapdragon Voice & Music, Snapdragon Wearables, Snapdragon Wired Infrastructure and Networking in APQ8009, APQ8053, APQ8096AU, APQ8098, IPQ4019, IPQ8074, MDM9206, MDM9207C, MDM9607, MDM9640, MDM9650, MSM8905, MSM8909W, MSM8917, MSM8953, MSM8996AU, QCA4531, QCN7605, QCS605, QM215, SA415M, SC8180X, SDA660, SDA845, SDM429, SDM429W, SDM439, SDM450, SDM630, SDM632, SDM636, SDM660, SDM845, SDX20, SDX24, SDX55, SM6150, SM7150, SM8150, SM8250, SXR2130

### POC

#### Reference
- https://www.qualcomm.com/company/product-security/bulletins/may-2020-bulletin

#### Github
No PoCs found on GitHub currently.

repository: trickest/cve

gha_language: HTML

repo_name: trickest/cve


text:
# Template MVC in PHP

 If you are looking for a how to assemble an MVC structure, or looking for an easy to use structure. :+1:

## Don't you know what MVC is?

MVC is used in many projects due to its architecture, which makes it possible to divide the project into very well defined layers. Each of them, Model, Controller and View, executes what is defined and nothing more than that.

## To start

Change the access to the bank in the global variable BASE_URL in the config.php file.

Check the .htacess file too, if you are not able to access the fourth line.


text:
# db-oracle: Oracle database bindings for Node.js #

For detailed information about this and other Node.js
database bindings visit the [Node.js db-mysql homepage] [homepage].

## INSTALL ##

### OCI libraries ###

Before proceeding with installation, you need to have the
[OCI instant client] [oci] [libraries] [oci-lib] and [include files] [oci-inc].
For example, you download the `instantclient-basic-linux.x64-<IP_ADDRESS>.0.zip`
library file, and the `instantclient-sdk-linux.x64-<IP_ADDRESS>.0.zip` SDK file,
and save them in your `$HOME/Downloads` directory. You would then uncompress
both files, and move the generated directory to your `/opt` path:

```bash
$ cd $HOME/Downloads
$ unzip instantclient-basic-linux.x64-<IP_ADDRESS>.0.zip
$ unzip instantclient-sdk-linux.x64-<IP_ADDRESS>.0.zip
$ sudo mv instantclient_11_2/ /opt/instantclient
```

After uncompressing you will probably need to create symbolink links:

```bash
$ cd /opt/instantclient
$ sudo ln -s libocci.so.11.1 libocci.so
$ sudo ln -s libclntsh.so.11.1 libclntsh.so
```

You will also need `libaio`. In **Arch Linux** this can easily be installed with:

```bash
$ sudo pacman -S libaio
```

On **Debian** based distros:

```bash
$ sudo apt-get install libaio
```

### Configuring OCI ###

Once you have the library and include files installed, and in order for the
installation script to locate them properly, you'll need to set the
`OCI_INCLUDE_DIR` and `OCI_LIB_DIR` environment variables. For example:

```bash
$ export OCI_INCLUDE_DIR=/opt/instantclient/sdk/include/
$ export OCI_LIB_DIR=/opt/instantclient
```

### Install ###

Once the environment variables are set, install with npm:

```bash
$ npm install db-oracle
```

You should also add the OCI library path to your `LD_LIBRARY_PATH` environment:

```bash
export LD_LIBRARY_PATH=/opt/instantclient
```

## QUICK START ##

```javascript
var oracle = require('db-oracle');
new oracle.Database({
    hostname: 'localhost',
    user: 'root',
    password: 'password',
    database: 'node'
}).connect(function(error) {
    if (error) {
        return console.log("CONNECTION ERROR: " + error);
    }

    this.query().select('*').from('users').execute(function(error, rows) {
        if (error) {
            return console.log('ERROR: ' + error);
        }
        console.log(rows.length + ' ROWS');
    });
});
```

## LICENSE ##

This module is released under the [MIT License] [license].

[homepage]: http://nodejsdb.org/db-mysql
[license]: http://www.opensource.org/licenses/mit-license.php
[oci]: http://www.oracle.com/technetwork/database/features/oci/index.html
[oci-lib]: http://www.oracle.com/technetwork/topics/linuxx86-64soft-092277.html
[oci-inc]: http://www.oracle.com/technetwork/topics/linuxx86-64soft-092277.html

repository: tojocky/node-db-oracle

repo_name: tojocky/node-db-oracle


text:
# Typed usb-barcode scanner
A typed library for using USB barcode scanners in Node.js with support for Typescript.

On Linux and Mac, you have to run your application with root privileges.

## Intallation
You can install this package by running the following command:
``` bash
npm i usb-barcode-scanner --save
```
Or using yarn:
``` bash
yarn add usb-barcode-scanner
```

## Usage Typescript

You can list all devices using the following code:
``` typescript
import { getDevices } from 'usb-barcode-scanner';

console.log(getDevices());
```

``` typescript
import { UsbScanner } from 'usb-barcode-scanner';

let scanner = new UsbScanner({
    vendorId: 1155,
    productId: 22352
    /** You could also initialize the scanner by giving entering the path variable:
     *  path: 'IOService:/AppleACPI etc...'
    **/
});

scanner.on('data', (data) => {
    console.log(data);
});

scanner.startScanning();
```

To stop scanning, you can use:
``` typescript
scanner.stopScanning();
```

## Usage Vanilla JS
You can list all devices using the following code:
``` javascript
let getDevices = require('usb-barcode-scanner').getDevices;

console.log(getDevices());
```

``` javascript
let UsbScanner = require('usb-barcode-scanner').UsbScanner;

let scanner = new UsbScanner({
    vendorId: 1155,
    productId: 22352
});

scanner.on('data', (data) => {
    console.log(data);
});

scanner.startScanning();
```

To stop scanning, you can use:
``` typescript
scanner.stopScanning();
```

repository: macyabbey/usb-barcode-scanner

repo_name: macyabbey/usb-barcode-scanner


text:
# resxtojson
Converts resx to JSON (with just a few dependencies).

Usage:

```
resxtojson MyApp*.resx resources
```

This will scan all resx files in the current folder and send the output to the
folder resources of the current folder.

Options:

-m, --regx-match <value>

Used to set a key pattern match for filtering resources, e.g.:

```
resxtojson -m \\bMyPrefix_.*\\b MyApp*.resx resources
```

Only resource keys starting with MyPrefix_ will be converted.

repository: ericontilt/resxtojson

repo_name: ericontilt/resxtojson


text:
# How To Split A Fuzzer-Generated Input Into Several

This document describes several recommended strategies for splitting
a single fuzzer-generated input into several parts (sub-inputs).

# Examples
Splitting a fuzzer-generated input into several independent
parts (sub-inputs) is required very often. Some examples:

* Fuzzing a regular expression library requires
   * The regular expression (RE)
   * Flags for RE compilation and matching
   * A string to search the RE in

* Fuzzing an audio/video format decoder often requires
  * Decoding flags
  * Several frames

* Fuzzing a XSLT or CSS library requires
  * The stylesheet input
  * The XML/HTML input

* Fuzzing a font-rendering library requires
  * The font file
  * The text to render
  * The rendering flags

* Fuzzing a database library may require
  * The query text
  * The database state

TODO: more examples?

# Common Data Format

When trying to split the fuzzer-generated input into several parts,
the first question one needs to ask is whether the input format is common,
i.e. is it used or processed by other libraries, APIs, of fuzz targets.

If the data format is common (e.g. a widely used media format or network packet
format) then it is highly desirable for a fuzz target to consume exactly this
data format, and not some custom modification.
This way it will be easier to procure a seed corpus for this fuzz target
and to use the generated corpus to test/fuzz other targets.

## Multiple Options
If the data format may be processed by a fuzz target in a small number of different ways,
it is often the best approach to split the fuzz target into several ones,
each processing the input in exactly one way.

Make sure to [cross-pollinate](https://github.com/google/fuzzing/blob/master/docs/glossary.md#cross-pollination)
the corpora between these targets.
OSS-Fuzz does that automatically.

## Embedding / Comments
When a fuzz target for a common data format requires some flags, options, or
additional auxiliary sub-input(s), it is sometimes possible to embed the extra input
inside a custom section or a comment of the main data format.

Examples:
* PNG allows custom "chunks", and so a fuzz target for a PNG decoder can
  hide the flags used during PNG processing
  in a separate PNG chunk, e.g. `fUZz`
  ([example](https://github.com/google/oss-fuzz/blob/master/projects/libpng-proto/libpng_transforms_fuzzer.cc)).
* When fuzzing C/C++/Java/JavaScript inputs one may hide a sub-input in a single-line `//` comment. TODO: example?

## Hash
When only one small fixed-size sub-input is required (such as flags / options),
the fuzz target may compute a hash function on the full input and use it as the flag bits.
This option is very easy to implement, but it's applicability is limited to
relatively simple cases. The major problem is that a small local mutation of the input
leads to a large change in the sub-input, which often makes fuzzing less
efficient. Try this approach if the flags are individual bits and the input
type allows some bit flips in the inputs (e.g. a plain text).

TODO: example.

# Custom Serialization Format

If you **do not intend to share the corpus** with any other API or fuzz targets,
then a custom serialization format might be a good option for a multi-input fuzz
target.

## First / Last Bytes

When only one fixed-size sub-input is required (such as flags / options),
it is possible to treat the first (or last) `K` bytes of the input as sub-input,
and the rest of the bytes as the main input.

Just remember to copy the main input into a separate heap buffer of `Size - K`
bytes, so that buffer under/overflows on the main input are detected.

TODO: example.

## Magic separator

Choose a 4-byte (or 8-byte) magic constant that will serve as a separator
between the inputs.
In the fuzz target, split the input using this separator. Use `memmem` to
find the separator in the input -- `memmem` is known to be friendly to fuzzing
engines, at least to libFuzzer.

Example (see full code [here](https://github.com/llvm-mirror/compiler-rt/blob/6cd423889971c0d97801a9f3b9b5afb91ae9c137/test/fuzzer/MagicSeparatorTest.cpp)):
```cpp
// Splits [data,data+size) into a vector of strings using a "magic" Separator.
std::vector<std::vector<uint8_t>> SplitInput(const uint8_t *Data, size_t Size,
                                     const uint8_t *Separator,
                                     size_t SeparatorSize) { ... }

extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
  const uint8_t Separator[] = {0xDE, 0xAD, 0xBE, 0xEF};
  auto Inputs = SplitInput(Data, Size, Separator, sizeof(Separator));
  // Use Inputs.size(), Inputs[0], Inputs[1], ...
}
```

It is relatively easy for a modern fuzzing engine to discover the separator,
but nevertheless we recommend to provide several seed inputs with the desired number
of separators.

## Fuzzed Data Provider

[FuzzedDataProvider] (*FDP*) is a single-header C++ library that is helpful for
splitting a fuzz input into multiple parts of various types. It is a part of
LLVM and can be included via `#include <fuzzer/FuzzedDataProvider.h>` directive.
If your compiler doesn't have this header (in case it's an older Clang version
or some other compiler), you can copy the header from [here] and add it to your
project manually. It should just work, as the header doesn't depend on LLVM.

An advantage and disadvantage of using this library is that the input splitting
happens dynamically, i.e. you don't need to define any structure of the input.
This might be very helpful in certain cases, but would also make the corpus to
be no longer in a particular format. For example, if you fuzz an image parser
and split the fuzz input into several parts, the corpus elements will no longer
be valid image files, and you won't be able to simply add image files to your
corpus.

### Main concepts

* [FuzzedDataProvider] is a class whose constructor accepts `const uint8_t*,
  size_t` arguments. Usually, you would call it in the beginning of your
  `LLVMFuzzerTestOneInput` and pass the `data, size` parameters provided by the
  fuzzing engine.
* Once an FDP object is constructed using the fuzz input, you can consume the
  data from the input by calling the FDP methods listed below.
* If there is not enough data left to consume, FDP will consume all the
  remaining bytes. For example, if you call `ConsumeBytes(10)` when there are
  only `4` bytes left in the fuzz input, FDP will return a vector of length `4`.
* If there is no data left, FDP will return the default value for the requested
  type or an empty container (when consuming a sequence of bytes).
* If you consume data from FDP in a loop, make sure to check the value returned
  by `remaining_bytes()` between loop iterations.
* Do not use the methods that return `std::string` unless your API requires a
  string object or a C-style string with a trailing null byte. This is a common
  mistake that hides off-by-one buffer overflows from AddressSanitizer.

### Methods for extracting individual values

* `ConsumeBool`, `ConsumeIntegral`, `ConsumeIntegralInRange` methods are helpful
  for extracting a single boolean or integer value (the exact type is defined by
  a template parameter), e.g. some flag for the target API, or a number of
  iterations for a loop, or length of a part of the fuzz input.
* `ConsumeProbability`, `ConsumeFloatingPoint`, `ConsumeFloatingPointInRange`
  methods are very similar to the ones mentioned above. The difference is that
  these methods return a floating point value.
* `ConsumeEnum` and `PickValueInArray` methods are handy when the fuzz input
  needs to be selected from a predefined set of values, such as an enum or an
  array.

These methods are using the last bytes of the fuzz input for deriving the
requested values. This allows to use valid / test files as a seed corpus in some
cases.

### Methods for extracting sequences of bytes

Many of these methods have a length argument. You can always know how many bytes
are left inside the provider object by calling `remaining_bytes()` method on it.

* `ConsumeBytes` and `ConsumeBytesWithTerminator` methods return a `std::vector`
  of the requested size. These methods are helpful when you know how long a
  certain part of the fuzz input should be. Use `.data()` and `.size()` methods
  of the resulting object if your API works with raw memory arguments.
* `ConsumeBytesAsString` method returns a `std::string` of the requested length.
  This is useful when you need a null-terminated C-string. Calling `c_str()` on
  the resulting object is the best way to obtain it.
* `ConsumeRandomLengthString` method returns a `std::string` as well, but its
  length is derived from the fuzz input and typically is hard to predict, though
  always deterministic. The caller can provide the max length argument.
* `ConsumeRemainingBytes` and `ConsumeRemainingBytesAsString` methods return
  `std::vector` and `std::string` objects respectively, initialized with all the
  bytes from the fuzz input that left unused.
* `ConsumeData` method copies the requested number of bytes from the fuzz input
  to the given pointer (`void *destination`). The method is useful when you need
  to fill an existing buffer or object (e.g. a struct) with fuzzing data.

For more information about the methods, their arguments and implementation
details, please refer to the [FuzzedDataProvider] source code. Every method has
a detailed comment in that file, and the implementation is relatively small.

### Examples of fuzz targets using `FuzzedDataProvider`

* [net_verify_name_match_fuzzer] splits the fuzz input into two parts.
* [net_http2_frame_decoder_fuzzer] reads data in small chunks in a loop in order
  to emulate a sequence of frames coming from the network connection.
* [net_crl_set_fuzzer] initialized multiple parameters and uses the rest of the
  fuzz input for the main argument (i.e. data to be parsed / processed). Note
  that using [Protobufs](#Protobufs) based fuzzing might be more efficient for
  such a target.
* [net_parse_cookie_line_fuzzer] is a slightly more sophisticated fuzz target
  that emulates different actions with different parameters initialized with the
  fuzz input.

[FuzzedDataProvider]: https://github.com/llvm/llvm-project/blob/main/compiler-rt/include/fuzzer/FuzzedDataProvider.h
[here]: https://raw.githubusercontent.com/llvm/llvm-project/main/compiler-rt/include/fuzzer/FuzzedDataProvider.h
[net_crl_set_fuzzer]: https://cs.chromium.org/chromium/src/net/cert/crl_set_fuzzer.cc?rcl=0be62a8d95f7fa1455fce1a76f0fa5b8484d0c8c
[net_http2_frame_decoder_fuzzer]: https://cs.chromium.org/chromium/src/net/spdy/fuzzing/http2_frame_decoder_fuzzer.cc?rcl=0be62a8d95f7fa1455fce1a76f0fa5b8484d0c8c
[net_parse_cookie_line_fuzzer]: https://cs.chromium.org/chromium/src/net/cookies/parse_cookie_line_fuzzer.cc?rcl=0be62a8d95f7fa1455fce1a76f0fa5b8484d0c8c
[net_verify_name_match_fuzzer]: https://cs.chromium.org/chromium/src/net/cert/internal/verify_name_match_fuzzer.cc?rcl=0be62a8d95f7fa1455fce1a76f0fa5b8484d0c8c

## Type-length-value
A custom [Type-length-value](https://en.wikipedia.org/wiki/Type-length-value), or TLV,
may sound like a good solution. However, we typically **do not recommend using a custom TLV**
to split your fuzzer-generated input for the following reasons:
* This is more test-only code for you to maintain, and easy to get wrong
* Typical mutations performed by fuzzing engines, such as inserting a byte,
  will break the TLV structure too often, making fuzzing less efficient

However, a TLV input combined with a custom mutator might be a good option.
See [Structure-Aware Fuzzing](structure-aware-fuzzing.md).

## Protobufs

Yet another option is to use one of the general-purpose serialization formats,
such as Protobufs, in combination with a custom mutator.
See [Structure-Aware Fuzzing](structure-aware-fuzzing.md).

repository: google/fuzzing

repo_name: google/fuzzing


text:
# iOWN Token

An ERC-20 Ethereum based token for iOWN project.

[![Telegram icon](./assets/telegram-follow.png)](https://t.me/iOWNToken)

## More information

More information about iOWN available here:

* [Website](https://www.iowntoken.com) - iOWN Token official website.
* [Whitepaper](https://www.iowntoken.com/whitepaper) - iOWN Token official whitepaper.

## Ethereum Token Address

iOWN Token V3 is deployed on this address (not distributed yet).
* [ERC20](https://bscscan.com/token/0x5d681b9839e237c4f1dc7d7486e6cb0a12b9654f) - iOWN Token ERC20 Contract Address (BNB/BSC Chain).

iOWN Token is deployed on this address:
* [ERC223](https://etherscan.io/token/0x555d051538c7a13712f1f590fa6b4c176ca4529f) - iOWN Token ERC223 Contract Address (To be deactivated).

Important: Original Smart contract (ERC20) was deployed here:
* [ERC20](https://etherscan.io/token/0x870ff0b9214ee330674dd143bc1836f8b11a627a) - iOWN Token ERC20 (Inactive Cotnract).

## About iOWN

iOWN is a Blockchain based crowdfunding platform concept coupled with a comprehensive ecosystem,
aiming to bridge the gaps between investors and business seeking funding in tranditional investment environments.

The platform serves a marketplace for listing and participating in fundraising campaigns.
The ecosystem realizes the platform services including qualification process of campaigns, auditing, account, operations, etc...

### Prerequisites

This reporistory depends on the following:

* [node](https://nodejs.org) - NodeJS v11.8.0 (npm v6.9.0)
* [truffle](https://truffleframework.com) - Truffle Framework v5.0.13
* [Infura](https://infura.io) - Infura

Packages should be pre-installed

Other requirements:

* [truffle-flattener](https://github.com/nomiclabs/truffle-flattener) - Truffle Flattener v1.3.0
* [EthLint](https://github.com/duaraghav8/Ethlint) - EthLint (Solium) v1.2.4

All steps and environment is based on Ubuntu 18.04

### Installing

Install and preparing:

```
npm install
```

## Compiling

Install and preparing:

```
truffle compile (--all)
```

### Migrations

Before running truffle migrate

The following files are required:

```
vi ~/.keys/owner.key
chmod 600 ~/.keys/owner.key
```

Infura configuration

```
vi ~/.keys/infura.key
chmod 600 ~/.keys/infura.key
```

Alternatively edit `truffle-config.js` to remove references for files

## Deployment

Smart contracts can be deployed via truffle, but verification process will be troublesome on [Etherscan](https://etherscan.io) because of multi-file contarcts (despite flattening)

As such its recommended to "flatten" the contracts using tuffle-flattner:

```
truffle-flattener ./contracts/iown/IownToken.sol > ./build/IownToken-flattended.sol
```

Then you can use `./build/IownToken-flattended.sol` on [Remix](https://remix.ethereum.org) along with [MetaMask](https://metamask.io).
The same flattended file can then be used on etherscan verification process.

To generate constructor arguments (abi encoded), [HashEx](https://abi.hashex.org) can be used.

## Authors

* **Alexander Sayegh** - *Initial work* - [LinkedIn](https://www.linkedin.com/in/alexandersayegh/)
* **Mohammad Saleh** - *BNB Chain Upgrade* - [LinkedIn](https://www.linkedin.com/in/clrmas/)

## Contact Us

You can also reach us here [info@iowntoken.com](mailto:info@iowntoken.com).

## License

This project is licensed under the MIT License.

## Acknowledgments

* Hat tip to anyone worked on OpenZeppelin, Ethereum and all frameworks and code used here
Remix

repository: iOWNToken/iOWNToken

repo_name: iOWNToken/iOWNToken


text:
# Module Installation
The Broadleaf Amazon module requires [configuration](#configuration-changes) and [third-party property configuration](#third-party-property-configuration)

## Broadleaf Dependency

- Version 2.1.x-GA requires Broadleaf 5.2 or later.
- Version 2.0.x-GA requires Broadleaf 5.0 or 5.1
- Version 1.1.x-GA requires Broadleaf 4.0 or 4.1
- Version 1.0.x-GA requires Broadleaf 3.1

## Configuration Changes
**Step 1.**  Add the dependency into your `core/pom.xml`:

```xml
<dependency>
    <groupId>org.broadleafcommerce</groupId>
    <artifactId>broadleaf-amazon</artifactId>
</dependency>
```

## Third Party Property Configuration
This module requires you to configure properties specific to your amazon account.

### Amazon Credentials
Broadleaf requires access to your Amazon AWS account. See [About Amazon Credentials](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) for more information.

**AccessKeyId / SecretKey**
One approach for setting up credentials is to use an accessKeyId and secretKey.  When you create the access keys on amazon you will need to copy the values and add the following properties to your `common-shared.properties` file located in your core project.

    aws.s3.accessKeyId=

    aws.s3.secretKey=

**Instance Profiles**
If you are hosting on EC2, you can now use Instance Profiles.  See [Creating Instance Profiles](http://docs.aws.amazon.com/codedeploy/latest/userguide/getting-started-create-iam-instance-profile.html)

If you are using Instance Profiles, then the following property should be set:

    aws.s3.useInstanceProfile=true

### Amazon File Storage Location Information
Broadleaf needs to know the specific location within your S3 account to store the files.   You will need to set the bucket name and the bucket location.  These properties are described on the [Amazon S3 Location Selection page](http://docs.aws.amazon.com/AmazonS3/latest/dev/LocationSelection.html).

_The bucket name must be unique across all of Amazon_

    aws.s3.defaultBucketName=

_If you would like to store files inside of a folder within the bucket (like 'img'), set the following property:_

    aws.s3.bucketSubDirectory=img

> Starting and trailing slashes will be stripped from this value to build the file path so they are not necessary

_The Amazon module will default to the "us-west-2" region of S3. You can override the region by setting the following property._

    aws.s3.defaultBucketRegion=us-west-2

_If your uploads require Server-Side Encryption, enable with this property. The default value is "false"._

    aws.s3.sse=true

repository: BroadleafCommerce/blc-amazon

repo_name: BroadleafCommerce/blc-amazon


text:
# The Chicago Housing Search Utility
This project is designed to save individuals countless hours by customizing your very own housing recommendation model. Data is sourced from [Redfin](https://www.redfin.com/), [Zillow's API](https://www.zillow.com/howto/api/APIOverview.htm), [Google Directions API](https://developers.google.com/maps/documentation/directions/start), [ARCGIS API](https://developers.arcgis.com/python/), and [Chicago's Data Portal](https://data.cityofchicago.org/). This project was created using python 3.7 64-bit.

## Commute Features Available
Input raw RedFin search data into program called "get_commute.py". Inside the file "get_commute.py", you will need to specify your [Google API key](https://developers.google.com/maps/documentation/javascript/get-api-key) in order for the program to properly work.
- Total commute time to specified address (public transit)
- Total walking time to specified address (public transit)
- Route path to specified address (ex. TRAIN - BUS - WALKING)
- Total number of transfers to specified address (public transit)

## Grocery Store Features Available
Input raw RedFin search data into program called "get_grocery.py". Some prerequisites would be the geocoordinates of your favorite grocery stores. The csv file called "illinois_ZIP.csv" contains all the zip codes in Illinois. Using the zip codes and for example "get_wholeFoods_data.py" use ARCGIS API to search through zip codes containing key words. This will return a dataset with all the geocoordinates of the grocery stores.
- Number of grocery stores within 1/2 mile (walking distance)
- List of grocery stores within 1/2 mile (walking distance)
- Indicator if Aldi's, Jewel-Osco, Mariano's, Trader Joe's, or WholeFoods are within 1/2 mile (walking distance)
- Number of grocery stores within 2 miles (driving distance)
- List of grocery stores within 2 miles (driving distance)
- Indicator if Aldi's, Jewel-Osco, Mariano's, Trader Joe's, or WholeFoods are within 2 miles (driving distance)

## Zillow Features Available
Input raw RedFin's address and zipcode into Zillow's API will get you several features.
- Zillow ID
- Zillow URL
- Last Tax Year
- Last Tax Assessment Amount
- Estimated Yearly Tax (based on Cook County average tax rate 2.117%)
- Last Sold Date
- Last Sold Price
- Zestimate

## Tasks to Complete
- Text analysis on crime statistics
- Closest gun crime (distance)
- Text analysis on census data
- Number of parks in a fixed range
- Closest park (distance)
- Liquor Stores and Tobacco Stores

## Contributing
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

## License
[MIT](https://choosealicense.com/licenses/mit/)

repository: mmachen/housing-search-Chicago

gha_language: Python

repo_name: mmachen/housing-search-Chicago


text:
# Advent of Code 2015 - Golang

<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
**Table of Contents**

- [01](#01)
  - [Lessons](#lessons)
- [02](#02)
  - [Lessons](#lessons-1)
- [03](#03)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

## 01

```bash
# part 1
$ nodemon -w ./01/part-1/main.go --exec 'go run ./01/part-1/main.go ../01.txt || exit 1'

# part 2
$ nodemon -w ./01/part-2/main.go --exec 'go run ./01/part-2/main.go ../01.txt || exit 1'
```

### Lessons

- to slice a slice or array, use `xs[start:end]`.
- one gets arguments passed to the process via `os.Args` - this is like
    `process.env` in Node
- to get the current working directory, i.e. the directory in which the
    executable lies, we use `os.getWd()`
- to join paths one uses `path.Join(path1, path2, [...])`. This is like
    `require(path.resolve(path1, path2))` in Node
- `io/ioutil` is useful for reading in file data
- `ioutil.ReadFile` returns a byte array - attempting to write it straight out
    may not be meaningful. `ioutil.ReadFile` is similar to `fs.readFile` in Node
- `string(myByteArray)` will format the byte array as a string
- `fmt.Pintf("%s", myByteArray)` will format the byte array as a string without
    the additional `string` call
- the go docs seeem to favour using `log.Fatal(err)` for logging errors -
    `panic` is used in Go By Example
    - `log.Fatal` is equivalent to `fmt.Println(msg); os.Exit(1)`
    - `panic(msg)` will exit once all deferred invocations have completed
        executing

## 02

```bash
# part 1
$ nodemon -w ./02/part-1/main.go --exec 'go run ./02/part-1/main.go ../02.txt || exit 1'

# part 2
$ nodemon -w ./02/part-2/main.go --exec 'go run ./02/part-2/main.go ../02.txt || exit 1'
```

### Lessons

- `backticks` denote string literals, as they do in Javascript. There doesn't
    seem to be string interpolation, however
- "double-quotes" denote interpreted string literals. Special characters, such
    as backslashes, need to be escaped
- the `regexp` package allows for one to define regular expressions
    - start by defining your pattern using `regexp.Compile` or
        `regexp.MustCompile`, followed by using the resulting regular expression
        to perform a match
        - this is like `new RegExp(/pattern/)` in Javascript
    - to find all occurrences in a string:

        ```golang
        newlineRx := regexp.MustCompile(`\n`)

        // or
        newlineRx := regexp.MustCompile("\\n")

        matches := newlineRx.FindAll(str)
        ```
- the `strings` package allows for text manipulation, such as splitting a string
    by a specific character:

    ```golang
    words := strings.Split(str)
    ```

    - `strings.Split` is like `String.prototype.split` in Javascript
- use `strconv.Atoi` to parse integers from alphanumeric strings
    - `strconv.Atoi` is similar to `parseInt` in Javascript
- the `sort` package is useful for sorting slices of different types:

    ```golang
    xs := []int{3,2,1}
    // sorts and mutates xs
    sort.Ints(xs)
    ```

    - `sort` is similar to `Array.prototype.sort` in Javascript, but with
        explicitly typed functions

## 03

```bash
# part 1
$ nodemon -w ./03/part-1/main.go --exec 'go run ./03/part-1/main.go ../03.txt || exit 1'
```

license: Unlicense

repository: larrybotha/coding-challenge-solutions

gha_language: Go

gha_license_id: Unlicense

repo_name: larrybotha/coding-challenge-solutions


text:
---
layout: post
title: Connect to Office 365 with PowerShell on Linux
tags: [PowerShell]
author: Matt McNabb
comments: true
---

[ConnectExchange]: https://technet.microsoft.com/en-us/library/jj984289(v=exchg.160).aspx
[ConnectCompliance]: https://technet.microsoft.com/en-us/library/mt587092(v=exchg.160).aspx
[ProgBar]: /assets/img/O365linuxProgBar.png

This Wednesday Microsoft announced the release of PowerShell 6.0 Alpha 17. One new feature in particular intrigued, and that's the capability to [connect to custom remoting configurations](https://github.com/PowerShell/psl-omi-provider/releases/tag/v<IP_ADDRESS>). This opens up the possibility of connecting to an Exchange endpoint, including Office 365. I just had to give this a try to see how and if it works. My set up for testing is a Hyper-V VM with Ubuntu 16.04 installed, and I installed PowerShell directly from the [Github releases page](https://github.com/PowerShell/PowerShell/releases/tag/v6.0.0-alpha.17).

Here's how you can create an implicit remoting connection to Exchange Online:

<!--more-->

{% highlight PowerShell %}
PS /home/matt> $O365Credential = Get-Credential
PS /home/matt> $Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://outlook.office365.com/powershell-liveid/ -Credential $O365Credential -Authentication Basic -AllowRedirection
PS /home/matt> Import-PSSession -Session $Session
{% endhighlight %}

You'll see a funky progress bar detailing the commands being imported from the remoting session:

![][ProgBar]

Now you can run any commands from the Exchage Online module:

{% highlight PowerShell %}
PS /home/matt> get-mailbox 'matt'

Name                      Alias                ServerName       ProhibitSendQuota
----                      -----                ----------       -----------------
Matt McNabb            matt                 bn3pr07mb2755    49.5 GB (53,150,220,288 bytes)
{% endhighlight %}

And the process for connecting to the Compliance Center is very similar:

{% highlight PowerShell %}
PS /home/matt> $Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://ps.compliance.protection.outlook.com/powershell-liveid/ -Credential $O365Credential -Authentication Basic -AllowRedirection
PS /home/matt> Import-PSSession -Session $Session
{% endhighlight %}

> NOTE: Note that I've added the prefix "CC" to the commands imported from the Compliance Center - that's because some of the included command names overlap with existing commands in the Exchange Online module.

Now in the release notes linked above the author states that this will allow connections to Office 365 from Linux, but that's not strictly true. It will allow connections to Exchange Online and the Compliance Center since both of these service rely on implicit remoting to an Exchange endpoint. But Office 365 is more than just Exchange - on Windows we can also use PowerShell to manage Azure Active Directory, Sharepoint, and Skype For Business. These other services rely on installers or assemblies that only work in Windows, so for now I think we'll only be able to use PowerShell to connect to the administrative services that are supported strictly with implicit remoting. Hopefully this will improve when PowerShell 6.0 is released and we see more folks using PowerShell for administration on Linux or OSX.

Time for a shameless plug: You can check out my module for making it easier to connect to all Office 365 services from PowerShell by running this command:

{% highlight PowerShell %}
PS C:> Install-Module O365Extensions
{% endhighlight %}

This module currently supports connections to Exchange,Azure AD, Sharepoint, Compliance Center, and Skype using a simple connection command. Support for Linux and OSX is coming soon, along with some other features that may make your day-to-day Office 365 management much easier.


text:
# Digital Cell Sorter

[![DOI](https://badge.fury.io/gh/sdomanskyi%2FDigitalCellSorter.svg)](https://badge.fury.io/gh/sdomanskyi%2FDigitalCellSorter)
[![DOI](https://badge.fury.io/py/DigitalCellSorter.svg)](https://pypi.org/project/DigitalCellSorter)
[![DOI](https://readthedocs.org/projects/digital-cell-sorter/badge/?version=latest)](https://digital-cell-sorter.readthedocs.io/en/latest/?badge=latest)
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.2533377.svg)](https://doi.org/10.5281/zenodo.2533377)

Digital Cell Sorter (DCS): a single cell RNA-seq analysis toolkit for clustering, cell type identification, and anomaly detection.

> [Digital Cell Sorter (DCS): a cell type identification, anomaly detection, and Hopfield landscapes toolkit for single-cell transcriptomics](https://doi.org/10.7717/peerj.10670 "Digital Cell Sorter (DCS): a cell type identification, anomaly detection, and Hopfield landscapes toolkit for single-cell transcriptomics."),
> Sergii Domanskyi, Alex Hakansson, Thomas Bertus, Giovanni Paternostro & Carlo Piermarocchi,
> *PeerJ*, 9:e10670 (**2021**).

> **The previous publication describing the methodology of cell types identification:**
> [Polled Digital Cell Sorter (p-DCS): Automatic identification of hematological cell types from single cell RNA-sequencing clusters](
> https://doi.org/10.1186/s12859-019-2951-x
> "Polled Digital Cell Sorter (p-DCS): Automatic identification of hematological cell types from single cell RNA-sequencing clusters"),
> Sergii Domanskyi, Anthony Szedlak, Nathaniel T Hawkins, Jiayin Wang, Giovanni Paternostro & Carlo Piermarocchi,
> *BMC Bioinformatics* volume 20, Article number: 369 (**2019**).

> **Note:** If you want to access the package detailed in our 2019 publication of Polled Digital Cell Sorter
> go to https://zenodo.org/record/2603265 and download the package (v1.1).

The documentation is available at https://digital-cell-sorter.readthedocs.io/.

- [Getting Started](#getting-started)
  * [Prerequisites](#prerequisites)
  * [Loading the package](#loading-the-package)
  * [Gene Expression Data Format](#gene-expression-data-format)
  * [Other Data](#other-data)
- [Functionality](#functionality)
  * [Overall](#overall)
  * [Visualization](#visualization)
- [Demo](#demo)
  * [Usage](#usage)
    + [Main cell types](#main-cell-types)
    + [Cell sub-types](#cell-sub-types)
  * [Output](#output)

## Getting Started

These instructions will get you a copy of the project up and running on your machine for data analysis, development or testing purposes.

### Prerequisites

#### Environment setup
The software runs in Python >= 3.7

It is highly recommended to install Anaconda.
Installers are available at https://www.anaconda.com/distribution/
Whether you already had Anaconda installed or just installed it we recommend to
update all packages by running:

	conda update conda

With conda, you can create, export, list, remove, and update environments that
have different versions of Python and/or packages installed in them.
Switching or moving between environments is called activating the environment.

	conda create --name DCS
	conda activate DCS

Now, in your new environment, the packages can be installed or updated without affecting
your other environments. Note, environments use is not necessary, and the
default ```(base)``` is used if you dont set up any other. For more information see
https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html

> Note: use of conda environments (for instance DCS exemplified above)
> with a high performance computer
> such as MSU HPCC in a batch job, i.e. not on a development node but
> submitted to a SLURM queue, requires the following steps. In the slurm
> script, before line calling your python script,
> add ```conda deactivate``` to deactivate base environnment
> and ```conda activate DCS```. After calling the script
> do ```conda deactivate```. The example testing script is shown below.

SLURM script example:

	#!/bin/bash --login
	########## Define Resources Needed with SBATCH Lines ##########
	#SBATCH --time=00:01:00             # limit of wall clock time - how long the job will run (same as -t)
	#SBATCH --ntasks=1                  # number of tasks - how many tasks (nodes) that you require (same as -n)
	#SBATCH --cpus-per-task=1           # number of CPUs (or cores) per task (same as -c)
	#SBATCH --mem=1G                    # memory required per node - amount of memory (in bytes)
	##SBATCH --job-name Name_of_Job     # you can give your job a name for easier identification (same as -J)

	########## Command Lines to Run ##########
	conda deactivate
	conda activate DCS
	cd ./                               ### change to the directory where your code is located
	python test.py        		    ### call your executable
	scontrol show job $SLURM_JOB_ID     ### write job information to output file
	conda deactivate

where ```test.py``` is the python script where you import and use
```DigitalCellSorter```.

#### Installation of the DigitalCellSorter package

Install ```DigitalCellSorter``` with ```pip```. Most of the dependencies packages
are automatically installed with installation of the  latest release
of ```DigitalCellSorter```:

	pip install DigitalCellSorter

Alternatively, you can clone and install this module directly from GitHub using:

	pip install git+https://github.com/sdomanskyi/DigitalCellSorter

Similarly, one can create a local copy of this project for development purposes, and
install the package from the cloned directory:

	git clone https://github.com/sdomanskyi/DigitalCellSorter
	python setup.py install

Our software uses packages ```numpy```, ```pandas```, ```matplotlib```,
```scikit-learn```, ```scipy```, ```mygene```, ```fftw```,
```fitsne```, ```adjustText``` and a few other standard Python packages.
Some of the packages used in ```DigitalCellSorter``` are not installed by default,
and should by installed by separately if using certain functionality with
Digital Cell Sorter. For example, for network-based clustering
install packages ```pynndescent```, ```networkx```, ```python-louvain```.
Other packages that have to be installed separately are ```fitsne```, ```umap```,
```phate``` and ```orca```. The detailed instructions are below.

#### t-SNE
With datasets containing less than 2000 cells ```sklearn.manifold.TSNE``` is used.
For large datasets Fast Fourier Transform-accelerated Interpolation-based t-SNE (FIt-SNE)
implemented by **KlugerLab** is used (https://github.com/KlugerLab/FIt-SNE).
To use FIt-SNE the following need to be installed. First update ```cython``` by

	pip install --upgrade cython

Then install ```fftw``` from the ```conda-forge``` channel
add ```conda-forge``` to your channels, and install ```fftw```:

	conda config --add channels conda-forge
	conda install fftw

The next installation step is platform specific. To install FI-tSNE for Linux:

	pip install fitsne

On macOS Mojave C++ compiler has to be specified explicitly:

	env CC=clang CXX=clang++ pip install fitsne

On Windows the FI-tSNE wrapper and executable are already
included with ```DigitalCellSorter```.

#### Other layouts

To use UMAP layout

	pip install umap-learn

To use PHATE

	pip install phate

> Note, if neither ```fitsne```, ```umap``` nor ```phate``` are installed
> ```DigitalCellSorter``` defaults to PCA two largest principal components for
> visualization layout.

#### Interactive HTML figures
To use Sankey diagrams that are part of Digital Cell Sorter
install ```plotly``` and ```orca```:

    conda install -c plotly plotly-orca
    conda install -c anaconda psutil

See
[interactive Hopfield landscape figure](http://htmlpreview.github.io/?https://github.com/sdomanskyi/DigitalCellSorter/blob/master/docs/examples/dataName_energy_landscape_PC1_vs_PC0.html "Hopfield attractors figure")
and
[interactive Sankey diagram figure](http://htmlpreview.github.io/?https://github.com/sdomanskyi/DigitalCellSorter/blob/master/docs/examples/PanglaoDB_Sankey_SRS3296611.html "Sankey diagram of cell annotation")
in a browser.

```orca	``` is necessary to convert Sankey diagrams to static images.
If for any reason ```orca``` is unavailable the Sankey diagrams will be saved as
ineractive HTML figure, that can be opened in a browser (Chrome, Firefox etc.) and
saved as static image. The visualization of ```DigitalCellSorter``` are implemented
with ```matplotlib```, allowing all the figures to be saved in either raster or
vactor format. Since ```plotly``` can convert simple ```matplotlib``` figures
(scatter, line, bar plots, but not heatmaps, splines or other complex patch objects) to
ineractive HTML format ```DigitalCellSorter``` can attempt to save any of its figures
as HTML. This is particulatly useful with ```Projection``` plots, even though the color
bars are not rendered in HTML figures.

### Loading the package

Use the latest release of PyPI package.

For a quick-start demo with a dataset of ~5k PBMCs execute
in the terminal and follow prompts:

	python -m DigitalCellSorter

The second, more detailed demonstration analysis with step-by-step
explanation is discussed here
and in the demo section at the end of this README.
In your script import the package:

	import DigitalCellSorter

Create an instance of class ```DigitalCellSorter```. Here, for simplicity, we use Default parameter values:

	DCS = DigitalCellSorter.DigitalCellSorter()

During the initialization a number of parameters can be specified. For detailed list see documentation.
Many of these parameters are transfered to DCS attributes thus can be modified after initialization using, e.g.:

	DCS.toggleMakeStackedBarplot = False

### Gene Expression Data Format

The input gene expression data is expected in one of the following formats:

1. Spreadsheet of comma-separated values ```csv``` containing condensed matrix in a form ```('cell', 'gene', 'expr')```.
If there are batches in the data the matrix has to be of the form ```('batch', 'cell', 'gene', 'expr')```. Columns order can be arbitrary.

Examples:

| cell | gene | expr |
|------|------|------|
| C1   | G1   | 3    |
| C1   | G2   | 2    |
| C1   | G3   | 1    |
| C2   | G1   | 1    |
| C2   | G4   | 5    |
| ...  | ...  | ...  |

or:

| batch  | cell | gene | expr |
|--------|------|------|------|
| batch0 | C1   | G1   | 3    |
| batch0 | C1   | G2   | 2    |
| batch0 | C1   | G3   | 1    |
| batch1 | C2   | G1   | 1    |
| batch1 | C2   | G4   | 5    |
| ...    | ...  | ...  | ...  |

2. Spreadsheet of comma-separated values ```csv``` where rows are genes, columns are cells with gene expression counts.
If there are batches in the data the spreadsheet the first row should be ```'batch'``` and the second ```'cell'```.

Examples:

| cell  | C1     | C2     | C3     | C4     |
|-------|--------|--------|--------|--------|
| G1    |        | 3      | 1      | 7      |
| G2    | 2      | 2      |        | 2      |
| G3    | 3      | 1      |        | 5      |
| G4    | 10     |        | 5      | 4      |
| ...   | ...    | ...    | ...    | ...    |

or:

| batch | batch0 | batch0 | batch1 | batch1 |
|-------|--------|--------|--------|--------|
| cell  | C1     | C2     | C3     | C4     |
| G1    |        | 3      | 1      | 7      |
| G2    | 2      | 2      |        | 2      |
| G3    | 3      | 1      |        | 5      |
| G4    | 10     |        | 5      | 4      |
| ...   | ...    | ...    | ...    | ...    |

3. ```Pandas DataFrame``` where ```axis 0``` is genes and ```axis 1``` are cells.
If the are batched in the data then the index of ```axis 1``` should have two levels, e.g. ```('batch', 'cell')```,
with the first level indicating patient, batch or expreriment where that cell was sequenced, and the
second level containing cell barcodes for identification.

Examples:

    df = pd.DataFrame(data=[[2,np.nan],[3,8],[3,5],[np.nan,1]],
                      index=['G1','G2','G3','G4'],
                      columns=pd.MultiIndex.from_arrays([['batch0','batch1'],['C1','C2']], names=['batch', 'cell']))

4. ```Pandas Series ``` where index should have two levels, e.g. ```('cell', 'gene')```. If there are batched in the data
the first level should be indicating patient, batch or expreriment where that cell was sequenced, the second level cell barcodes for
identification and the third level gene names.

Examples:

    se = pd.Series(data=[1,8,3,5,5],
                   index=pd.MultiIndex.from_arrays([['batch0','batch0','batch1','batch1','batch1'],
                                                    ['C1','C1','C1','C2','C2'],
                                                    ['G1','G2','G3','G1','G4']], names=['batch', 'cell', 'gene']))

Any of the data types outlined above need to be prepared/validated with a function ```prepare()```.
Let us demonstrate this on the input of type 1:

	df_expr = DCS.prepare('data/testData/dataFileCondensedWithBatches.tsv')

### Other Data

```markersDCS.xlsx```: An excel book with marker data. Rows are markers and columns are cell types.
'1' means that the gene is a marker for that cell type, '-1' means that this gene is not expressed in this cell type, and '0' otherwise.
This gene marker file included in the package is used by Default.
If you use your own file it has to be prepared in the same format (including the two-line header). Note that only the first worksheet will be read,
and its name can be arbitrary. The first column should contain gene names. The second row should contain cell types, and the first row how
those cell types are grouped. If any of the cell types need to be skipped, have "NA" in the corresponding cell of the first row of that cell type.

Example:

|A       |B            |C             |D           |E          |F                |G                         |H                           |I                        |J                         |K                  |L               |M                 |...      |
|--------|-------------|--------------|------------|-----------|-----------------|--------------------------|----------------------------|-------------------------|--------------------------|-------------------|----------------|------------------|---------|
|        |B cells      |B cells       |B cells     |T cells    |T cells          |T cells                   |T cells                     |T cells                  |T cells                   |T cells            |NK cells        |NK cells          |...      |
|Marker  |B cells naive|B cells memory|Plasma cells|T cells CD8|T cells CD4 naive|T cells CD4 memory resting|T cells CD4 memory activated|T cells follicular helper|T cells regulatory (Tregs)|T cells gamma delta|NK cells resting|NK cells activated|...      |
|ABCB4   |1            |0             |0           |0          |0                |0                         |0                           |0                        |0                         |0                  |0               |0                 |...      |
|ABCB9   |0            |0             |1           |0          |0                |0                         |0                           |0                        |0                         |0                  |0               |0                 |...      |
|ACAP1   |0            |0             |0           |0          |1                |0                         |0                           |0                        |0                         |0                  |0               |0                 |...      |
|ACHE    |0            |0             |0           |0          |0                |0                         |0                           |0                        |0                         |0                  |0               |0                 |...      |
|ACP5    |0            |0             |0           |0          |0                |0                         |0                           |0                        |0                         |0                  |0               |0                 |...      |
|ADAM28  |1            |1             |0           |0          |0                |0                         |0                           |0                        |0                         |0                  |0               |0                 |...      |
|ADAMDEC1|0            |0             |0           |0          |0                |0                         |0                           |0                        |0                         |0                  |0               |0                 |...      |
|ADAMTS3 |0            |0             |0           |0          |0                |0                         |0                           |0                        |0                         |0                  |0               |0                 |...      |
|ADRB2   |0            |0             |0           |0          |0                |0                         |0                           |0                        |0                         |0                  |0               |0                 |...      |
|AIF1    |0            |0             |0           |0          |0                |0                         |0                           |0                        |0                         |0                  |0               |0                 |...      |
|AIM2    |0            |1             |0           |0          |0                |0                         |0                           |0                        |0                         |0                  |0               |0                 |...      |
|ALOX15  |0            |0             |0           |0          |0                |0                         |0                           |0                        |0                         |0                  |0               |0                 |...      |
|ALOX5   |0            |1             |0           |0          |0                |0                         |0                           |0                        |0                         |0                  |0               |0                 |...      |
|AMPD1   |0            |0             |1           |0          |0                |0                         |0                           |0                        |0                         |0                  |0               |0                 |...      |
|ANGPT4  |0            |0             |1           |0          |0                |0                         |0                           |0                        |0                         |0                  |0               |0                 |...      |
|...     |...          |...           |...         |...        |...              |...                       |...                         |...                      |...                       |...                |...             |...               |...      |

```Human.MitoCarta2.0.csv```: An ```csv``` spreadsheet with human mitochondrial genes, created within work
[MitoCarta2.0: an updated inventory of mammalian mitochondrial proteins](https://doi.org/10.1093/nar/gkv1003 "MitoCarta2.0")
Sarah E. Calvo, Karl R. Clauser, Vamsi K. Mootha, *Nucleic Acids Research*, Volume 44, Issue D1, 4 January 2016.

## Functionality

### Overall

The main class, DigitalCellSorter, includes tools for:

  1. **Pre-preprocessing**
  2. **Quality control**
  3. **Batch effects correction**
  4. **Cells anomaly score evaluation**
  4. **Dimensionality reduction**
  5. **Clustering**
  6. **Annotating cell types**
  7. **Vizualization**
  8. **Post-processing**.

### Visualization

Function ```visualize()``` will produce most of the necessary files for post-analysis of the data.

See examples of the visualization tools below.

The visualization tools include:

- ```makeMarkerExpressionPlot()```: a heatmap that shows all markers and their expression levels in the clusters,
in addition this figure contains relative (%) and absolute (cell counts) cluster sizes

- ```getIndividualGeneExpressionPlot()```:  2D layout colored by individual gene's expression

- ```makeVotingResultsMatrixPlot()```: z-scores of the voting results for each input cell type and each cluster,
in addition this figure contains relative (%) and absolute (cell counts) cluster sizes

- ```makeHistogramNullDistributionPlot()```: null distribution for each cluster and each cell type illustrating
the "machinery" of the Digital Cell Sorter

- ```makeQualityControlHistogramPlot()```: Quality control histogram plots

- ```makeProjectionPlot()```: 2D layout colored by number of unique genes expressed,
number of counts measured, and a faraction of mitochondrial genes..

Effect of batch correction demostrated on combining BM1, BM2, BM3 and processing the data jointly without (left) and with (right) batch correction option:

- ```makeStackedBarplot()```: plot with fractions of various cell types

- ```makeSankeyDiagram()```: river plot to compare various results

- ```getAnomalyScoresPlot()```: plot with anomaly scores per cell

Calculate and plot anomaly scores for an arbitrary cell type or cluster:

- ```getIndividualGeneTtestPlot()```: Produce heatmap plot of t-test p-Values calculated gene-pair-wise
        from the annotated clusters

- ```makePlotOfNewMarkers()```: genes significantly expressed in the annotated cell types

## Demo

### Usage

We have made an example execution file ```demo.py``` that shows how to use ```DigitalCellSorter```.

In the demo, folder ```data``` is intentionally left empty.
The data file (cc95ff89-2e68-4a08-a234-480eca21ce79.homo_sapiens.mtx.zip) is about 2.4Gb in size and
will be downloaded with the ```demo.py``` script.

> Previously the HCA preview data was consolidated in file ```ica_bone_marrow_h5.h5``` and downloadable
> from https://preview.data.humancellatlas.org/ (Raw Counts Matrix - Bone Marrow).
> That file was ~485Mb and containing 378000 cells from 8 bone marrow donors (BM1-BM8).

See details of the script ```demo.py``` at:

> [Example walkthrough of demo.py script](https://github.com/sdomanskyi/DigitalCellSorter/blob/master/docs/examples/ "Examples")

To execute the complete script ```demo.py``` run:

	python demo.py

*Note that the HCA BM1 data contains ~50000 sequenced cells, requiring more than 60Gb of RAM (we recommend to use High Performance Computers).
If you want to run our example on a regular PC or a laptop, you can use a randomly chosen number of cells:

    df_expr.sample(n=5000, axis=1)

### Output

All the output files are saved in ```output``` directory inside the directory where the ```demo.py``` script is.
If you specify any other directory, the results will be generetaed in it.
If you do not provide any directory the results will appear in the root where the script was executed.

repository: sdomanskyi/DigitalCellSorter

repo_name: sdomanskyi/DigitalCellSorter


text:
![Saliens logo](https://i.imgur.com/5eu9a5P.png)

# SalienProgress

This project lists the current progression of Steam's Saliens minigame during the Steam Summer Sale of 2018.

All data required is locally retrieved using Steam's API and put through a proxy (allorigins.me) to prevent Same-Origin-Policy errors.

You can visit the project by visiting [wtzb.github.io/SalienProgress](https://wtzb.github.io/SalienProgress/).

All feedback is welcome, be it bugs, feature requests or simply a comment. Feel free to open an issue or [add me on steam](https://steamcommunity.com/id/Klaaiklut/).

Thanks to [@dimitarnestorov](https://github.com/dimitarnestorov) and [@MrLuit](https://github.com/MrLuit) for their various helpful contributions.

repository: wtzb/SalienProgress

repo_name: wtzb/SalienProgress