text:
# DataDonkey

Processes XML, CSV and Excel files.

DataDonkey is an OpenSource project by scireum GmbH. It is a scriptable ETL tool which processes various files.

Download the latest datadonkey.jar from: https://oss.sonatype.org/content/groups/public/com/scireum/datadonkey/

## Example

See [demo/example.js](demo/example.js) for an example script which combines an XML source ([demo/example.xml](demo/example.xml)) with a
CSV file ([demo/example.csv](demo/example.csv)) and generates an Excel output.

See [demo/xml-output/xml-example.js](demo/xml-output/xml-example.js) for an example on how to output XML.

## Usage

> java -jar datadonkey.jar myscript.js

## Reference

* All relevant functions are defined in [donkey.js](src/main/resources/donkey.js) (autoloaded on startup)
* XML nodes are presented as [StructuredNode](http://sirius-lib.net/apidocs/sirius-kernel/sirius/kernel/xml/StructuredNode.html)
* Rows of Excel or CVS files are represented as: [Values](http://sirius-lib.net/apidocs/sirius-kernel/sirius/kernel/commons/Values.html)
* **queryValue(String)** of StructuredNode and **at(String)** of Values return: [Value](http://sirius-lib.net/apidocs/sirius-kernel/sirius/kernel/commons/Value.html)

## License

DataDonkey is licensed under the MIT License:

> Permission is hereby granted, free of charge, to any person obtaining a copy
> of this software and associated documentation files (the "Software"), to deal
> in the Software without restriction, including without limitation the rights
> to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
> copies of the Software, and to permit persons to whom the Software is
> furnished to do so, subject to the following conditions:
>
> The above copyright notice and this permission notice shall be included in
> all copies or substantial portions of the Software.
>
> THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
> AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
> LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
> OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
> THE SOFTWARE.

repository: scireum/datadonkey

gha_language: Java

repo_name: scireum/datadonkey


text:
# Ramble on Soft Actor-Critic (SAC)

Recently, the implementation of algorithm Soft Actor-Critic (SAC) makes me aware of some critical problems that I haven't noticed before.

## Background

The Soft Actor-Critic algorithm is currently state-of-the-art model-free off-policy algorithm for reinforcement learning.

There are usually three main parts in SAC: the actor (policy) network, the state value network and its target network, and two state-action value networks (with Clipped Double Q-learning trick). The update of SAC can be briefly summarized as: (with implementations in Pytorch )

* For the state-action value networks:

![L(\phi_i, {\mathcal D}) = \underset{(s,a,r,s',d) \sim {\mathcal D}}{{\mathrm E}}\left[     \Bigg( Q_{\phi_i}(s,a) - \left(r + \gamma (1 - d) V_{\psi_{\text{targ}}}(s') \right) \Bigg)^2     \right].](https://spinningup.openai.com/en/latest/_images/math/0a1fc500475d85a71984c03f94462b075da698c3.svg)

	And its implementation:

```python
predicted_q_value1 = soft_q_net1(state, action)
predicted_q_value2 = soft_q_net2(state, action)
target_value = target_value_net(next_state)
target_q_value = reward + (1 - done) * gamma * target_value
soft_q_criterion1 = nn.MSELoss()
soft_q_criterion2 = nn.MSELoss()
q_value_loss1 = soft_q_criterion1(predicted_q_value1, target_q_value.detach())
q_value_loss2 = soft_q_criterion2(predicted_q_value2, target_q_value.detach())
```

* For the state value network: (with Clipped Double Q-learning trick)

![L(\psi, {\mathcal D}) = \underE{s \sim \mathcal{D} \\ \tilde{a} \sim \pi_{\theta}}{\Bigg(V_{\psi}(s) - \left(\min_{i=1,2} Q_{\phi_i}(s,\tilde{a}) - \alpha \log \pi_{\theta}(\tilde{a}|s) \right)\Bigg)^2}.](https://spinningup.openai.com/en/latest/_images/math/d44a3cfb903ec8530a5880aa718aa5cf0dad28f8.svg)

  	And its implementation:

```python
predicted_new_q_value = torch.min(soft_q_net1(state, new_action),soft_q_net2(state, new_action))
target_value_func = predicted_new_q_value - alpha * log_prob
value_criterion  = nn.MSELoss()
value_loss = value_criterion(predicted_value, target_value_func.detach())
```

* For the policy network: (maximize)

  ![\underE{a \sim \pi}{Q^{\pi}(s,a) - \alpha \log \pi(a|s)}.](https://spinningup.openai.com/en/latest/_images/math/d3deca00a02e211a85278358f902e3ab0683c8a5.svg)

  And its implementation: (three different types)

  ```python
  # 1. Berkely rlkit implementation
  policy_loss = (alpha * log_prob - predicted_new_q_value).mean()
  # 2. Openai Spinning Up implementation
  policy_loss = (alpha * log_prob - soft_q_net1(state, new_action)).mean()
  # 3. max Advantage instead of Q to prevent the Q-value drifted high
  policy_loss = (alpha * log_prob - (predicted_new_q_value - predicted_value.detach())).mean()

  ```

## Quick Facts

* For value-based methods like applying Q(s,a) or V(s), the absolute value doesn't matter, what matters is their relative value.
* Reward normalization can be critical sometimes.
* Sometimes the decreasing policy loss may not indicate better behaviours, several factors may cause that, depending on the format of policy loss functions.

## Details

When I first implemented the soft actor-critic algorithm with the first type of implementation of policy loss, I got a learning curve on my environment like following: (the blue line is reward, and orange line is a  predicted Q value)

![sac (copy)](/home/quantumiracle/research/RL_Robotics2/Soft_Actor_Critic/SAC/sac (copy).png)

<img src="https://github.com/quantumiracle/Reinforcement-Learning-for-Robotics/blob/master/img/3000step_sparse2.pdf" width="80%">

The _alpha_ is set to be 0 to remove the effects of entropy. The reward curve shows that the agent doesn't learn anything useful while the policy loss is actually keep decreasing. This can be derived as the policy loss is negative Q(s,a\_) in this case and the Q(s,a\_) is keep increasing. And it tells us a potential problem in SAC, **if the evaluation of Q and V keeps increasing together, then it will have chances to achieve a small Q loss and small V loss, even more, a decreasing policy loss**. And this is exactly the case happened in my first trial.

The core of the problem lies in that

And it is like jump on a soft ground.

Other algorithms like DQN may not suffer from this problem.

* For the Q-learning:

  ![L(\phi, {\mathcal D}) = \underset{(s,a,r,s',d) \sim {\mathcal D}}{{\mathrm E}}\left[     \Bigg( Q_{\phi}(s,a) - \left(r + \gamma (1 - d) \max_{a'} Q_{\phi}(s',a') \right) \Bigg)^2     \right]](https://spinningup.openai.com/en/latest/_images/math/d193a1fae2f39357adc458987f0301518f3cd669.svg)

* For the policy:

  ![\max_{\theta} \underset{s \sim {\mathcal D}}{{\mathrm E}}\left[ Q_{\phi}(s, \mu_{\theta}(s)) \right].](https://spinningup.openai.com/en/latest/_images/math/ec6f2bd53f08da666607434c871e250f86010241.svg)

This two functions cannot be satisfied with only an increasing value estimation of the Q-value network, because the policy loss requires update the policy parameters to maximize the Q-value with fixed Q-value network parameters. (This can be realized with variable\_scope and var\_list in optimizer with Tensorflow or Pytorch) Therefore it provides a solid update for the policy.

A key problem in the Q-value update in DQN is that if the rewards are all positive, the update function will keep drifting the Q-value to be larger and larger. This may not hurt the update of the policy, but will make the update of the Q-value function to chase a moving goal. To solve this problem, the rewards need to be normalized to have mean 0, like with a batch normalization.

repository: quantumiracle/Popular-RL-Algorithms

branch_name: refs/heads/master

extension: md

gha_language: Jupyter Notebook

license_type: permissive

repo_name: quantumiracle/Popular-RL-Algorithms


text:
---
title: "Manjari version 1.910 released"
author: Santhosh Thottingal
type: post
date: 2020-08-28T10:10:00+05:30
url: /blog/2020/08/28/manjari-version-1-910/
categories:
  - Projects
  - Malayalam
---

A new version of Manjari typeface is available now. New version adds about 25 Latin glyphs that are considered important by Google Fonts checks. Manjari is now integrated with [Fontbakery][1] font quality check in its CI. Some bugs reported by Fontbakery is also fixed.

It is available at [SMC website][2]. [Change log][3] is available in gitlab

SMC also started to publish the font releases in a new release file server - [releases.smc.org.in][4]. Thanks to Balasankar and Ashik for the release workflow automation.

[1]: https://github.com/googlefonts/fontbakery/
[2]: https://smc.org.in/fonts/manjari
[3]: https://gitlab.com/smc/fonts/manjari/-/releases/Version1.910
[4]: https://releases.smc.org.in/

license: MIT

repository: santhoshtr/website

repo_name: santhoshtr/website


text:
---
title: Enable or Disable Application Registration for a Service
no_version: true
content-type: how-to
---

To grant developers access to [register an application](/konnect/dev-portal/applications/dev-reg-app-service), you must enable application registration for a service version.
When you enable application registration, {{site.konnect_saas}} enables two plugins automatically: [ACL](/hub/kong-inc/acl), and your choice of [Key Authentication](/hub/kong-inc/key-auth)
or [OIDC](/hub/kong-inc/openid-connect). These plugins run in the background to support application registration for the service and are managed by
{{site.konnect_saas}}.

## Prerequisites

- A service that is versioned and published to the
  {{site.konnect_short_name}} Dev Portal so that it appears in the catalog.

- The service version must be in the `default` runtime group.
- The service version must have an [implementation](/konnect/servicehub/service-implementations).

- If you are using [OpenID Connect](#oidc-flow) for your authorization:

  - Set up your application, claims, and scopes in your OpenID identity provider. Refer to your IdP/OP provider's documentation for instructions.

  - Edit the **Reference ID** field in the Dev Portal
    [Update Application](/konnect/dev-portal/applications/dev-apps#edit-my-app)
    dialog to match to your third-party OAuth2 claim.

{:.note}
> **Note:** For instructions on configuring {{site.konnect_short_name}} declaratively, read our [declarative guide](/konnect/runtime-manager/runtime-groups/declarative-config).

## Enable app registration with key authentication {#key-auth-flow}

To enable app registration with key authentication, from the {{site.konnect_short_name}} menu, click {% konnect_icon servicehub %} **Service Hub**, select a
service, and follow these steps:

1. Click **Versions** to select a version.

2. From the **Version actions** drop-down menu, select **Enable app registration**.

3. Select `key-auth` from the **Auth Type** list.

4. Optional: click to enable [**Auto Approve**](/konnect/dev-portal/access-and-approval/auto-approve-devs-apps/) for application registration requests.

5. Click **Enable**.

    All versions of this service now include
    read-only entries for the `acl` and `key-auth` plugins.

## Enable app registration with OpenID Connect {#oidc-flow}

To enable app registration with OpenID Connect, from the {{site.konnect_short_name}} menu, click {% konnect_icon servicehub %} **Service Hub**, select a
service, and follow these steps:

1. Click **Versions** to select a version.

2. From the **Version actions** drop-down menu, select **Enable app registration**.

3. Select `openid-connect` from the **Auth Type** list.

   Refer to the [configuration parameters section](#openid-config-parameters) for information
   about each field.

4. Click **Enable**.

    All versions of this service now include
    read-only entries for the  `acl` and `oidc` plugins.

###  OpenID Connect configuration parameters {#openid-config-parameters}

   | Form Parameter | Description                                                                       |Required |
   |:---------------|:----------------------------------------------------------------------------------|--|
   | `Issuer` | The issuer URL from which the OpenID Connect configuration can be discovered. For example: `https://dev-1234567.okta.com/oauth2/default`.  |**True** |
   | `Scopes` | The scopes to be requested from the OpenID Provider. Enter one or more scopes separated by spaces, for example: `open_id` `myscope1`.  | **False**
   | `Consumer claims` |  Name of the claim that is used to find a consumer. | **True**
   | `Auth method` | The supported authentication method(s) you want to enable. This field should contain only the authentication methods that you need to use. Individual entries must be separated by commas. Available options: `password`, `client_credentials`, `authorization_code`, `bearer`, `introspection`, `kong_oauth2`, `refresh_token`, `session`. | **True**
   | `Hide Credentials` |**Default: disabled**
  Hide the credential from the upstream service. If enabled, the plugin strips the credential from the request header, query string, or request body, before proxying it. | **False** |
   | `Auto Approve`| **Default: disabled**
Automatically approve developer application requests for an application.| **False**

   For more background information about OpenID Connect plugin parameters, see
   [Important Configuration Parameters](/hub/kong-inc/openid-connect/#important-configuration-parameters).

## Disable application registration for a service {#disable}

Disabling application registration removes all plugins that were initially enabled through application registration for this service.
To remove a plugin by disabling application registration, follow these steps:

1. Click a service to open the **Service** menu.

2. From the **Service** menu, select **Version** to display all of the registered versions.

3. Click the version you intend to disable.

4. From the **Version actions** drop-down menu, select **Disable app registration**.

5. Click **Disable** from the pop-up modal.

You can
[re-enable application registration](/konnect/dev-portal/applications/enable-app-reg)
at any time.

branch_name: refs/heads/main

gha_license_id: MIT


text:
# json-sorter
Extracting a sorted result from a given JSON object or array even string, parse and stringify with [JSON3](https://github.com/bestiejs/json3).

# Usage
``` javascript

'use srtrict';

const jsonSorter = require('json-sorter');

jsonSorter(json [,options]);

```

The first parameter `json` could be an object or array, even a json string.

By default, `options.parse` is `true`, you can set it to `false` so that it will not try to parse string to object or array.
Of course you can pass an user-defined parse function which will trigger a json-string parameter.

`options.stringify` is used to judge whether to stringify the sorted result, and usually you don't need to set it.

If you don't set `options.compare`, the json with be sorted according to the dictionary. It with be used to sort the keys of the parsed object like `Array.prototype.sort`.

See [Array.prototype.sort](https://developer.mozilla.org/zh-CN/docs/Web/JavaScript/Reference/Global_Objects/Array/sort) for more information.

---

A complex example is as follows:
``` javascript
const jsonSorter = require('json-sorter');
const _ = require('underscore');

jsonSorter([{key: 1}, {key: 3}, {key: 2}, {key: {c: 1, d: 2, a: 3, b: 4}}], {
    compare: function (x, y) {
        if (_.isObject(x)) {
            return x.key < y.key;
        }

        return x < y;
    }
}));

// outputs : '[{"key":3},{"key":2},{"key":1},{"key":{"d":2,"c":1,"b":4,"a":3}}]'
```

---

Yeah, as yo see it's very simple.

# License
MIT License (Expat)

repository: Raiyee/json-sorter

repo_name: Raiyee/json-sorter


text:
# mojs-player

<img src="https://github.com/legomushroom/mojs-player/blob/master/mockups/mojs-player@x4.png?raw=true" alt="mojs-player" />

Player controls for [mojs](mojs.io). Intended to help you to craft `mojs` animation sequences. To be clear, this player is not needed to play `mojs` animations. It is just a debug tool that gives you the ability to:
 - control your sequences with `GUI` while working on them
 - it saves the current progress of your animation thus you don't loose the `focus`
 - it gives you `bounds` to constrain `focus point` of your animation
 - it gives you the control over the speed of animation
 - it gives you the ability to seek animations freely
 - it saves player's state and settings thus they not get lost when page reloaded

## Installation

The `MojsPlayer` depends on `mojs >= 0.225.2` so make sure you link it first.

[CDN](https://www.jsdelivr.com/):

```
<script src="//cdn.jsdelivr.net/mojs-player/latest/mojs-player.min.js"></script>
```

[NPM](https://www.npmjs.com/):

```
[sudo] npm install mojs-player
```

[Bower](http://bower.io/):

```
bower install mojs-player
```

Import `MojsPlayer` constructor to your code (depends on your environment) :

```javascript
const MojsPlayer = require('mojs-player').default;
// or
import MojsPlayer from 'mojs-player';
```

If you installed it with script link - you should have `MojsPlayer` global.

## Usage

Construct `MojsPlayer` and pass your main `Tween/Timeline` as the `add` option:

```javascript
// const mainTimeline = new mojs.Timeline({});
// .. some code regarding timeline ..
const mojsPlayer = new MojsPlayer({ add: mainTimeline });
```

The `add` option is the only required option to launch. Player's controls should appear at the bottom of the page.

You can also set other player initial state:

```javascript
// ..
const mojsPlayer = new MojsPlayer({
  add:      mainTimeline,
  // options
  className:    '',         // class name to add to main HTMLElement
  isSaveState:  true,       // determines if should preserve state on page reload
  isPlaying:    false,      // playback state
  progress:     0,          // initial progress
  isRepeat:     false,      // determines if it should repeat after completion
  isBounds:     false,      // determines if it should have bounds
  leftBound:    0,          // left bound position  [0...1]
  rightBound:   1,          // right bound position [0...1]
  isSpeed:      false,      // determines if speed control should be open
  speed:        1,          // `speed` value
  isHidden:     false,      // determines if the player should be hidden
  precision:    0.1,        // step size for player handle - for instance, after page reload - player should restore timeline progress - the whole timeline will be updated incrementally with the `precision` step size until the progress will be met.
  name:         'mojs-player' // name for the player - mainly used for localstorage identifier, use to distuguish between multiple local players
});
```

## Shortcuts

- `alt + p`  - toggle `play`/`pause` playback state
- `alt + - `  - decrease progress by `1/100`
- `alt + +` - increase progress by `1/100`
- `shift + alt + -` - decrease progress by `1/10`
- `shift + alt + +` - increase progress by `1/10`
- `alt + s` - `stop` playback
- `alt + r` - toggle `repeat` state
- `alt + b` - toggle `bounds` state
- `alt + h` - toggle `show`/`hide` player state
- `alt + q` - reset `speed` to `1x`
- `alt + 2` - decrease `speed` by `1/50`
- `alt + 3` - increase `speed` by `1/50`
- `shift + alt + 2` - decrease `speed` by `1/10`
- `shift + alt + 3` - increase `speed` by `1/10`

## Development

Install [webpack](https://webpack.github.io/) globally:

```tiddlywiki
[sudo] npm install webpack -g
```

Install dependencies with [npm](https://www.npmjs.com/):

```
[sudo] npm install
```

Run [webpack](https://webpack.github.io/):

```
webpack
```

Please make sure you are on the `dev` branch before making changes.

## License

(The MIT License)

Copyright (c) Oleg Solomka [@LegoMushroom](https://twitter.com/legomushroom) [legomushroom<EMAIL_ADDRESS>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the 'Software'), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

repository: quangogage/killstreak

repo_name: quangogage/killstreak


text:
# Local development

## Prerequisites

For the development environment you need to install these tools:

* [.NET Core 3.1 SDK](https://dotnet.microsoft.com/download)
* Docker: [Windows](https://docs.docker.com/docker-for-windows/install/) | [Mac OS](https://docs.docker.com/docker-for-mac/install/) | [Linux](https://docs.docker.com/install/linux/docker-ce/ubuntu/)

## Dependencies

To execute tests and run the project for debugging you also need instances of the following running and configured on the right ports:

* [NBXplorer](https://github.com/dgarage/NBxplorer)
* [Postgres](https://www.postgresql.org/)
* [Bitcoin Core](https://bitcoincore.org/) (and optionally other full nodes)

We recommend you use our dockerized setup to bootstrap the development environment:
The file [BTCPayServer.Tests/docker-compose.yml](https://github.com/btcpayserver/btcpayserver/blob/master/BTCPayServer.Tests/docker-compose.yml) can be used to spin everything up:

```bash
git clone https://github.com/btcpayserver/btcpayserver.git
cd btcpayserver/BTCPayServer.Tests
docker-compose up dev
```

To get started see the [README of the test project](https://github.com/btcpayserver/btcpayserver/blob/master/BTCPayServer.Tests/README.md).

## Which IDE?

We recommend using Visual Studio Code (cross platform) or Visual Studio 2019 (Windows Only) or Rider (cross platform).
You can of course use VIM if you are hardcore, .NET Core is command-line environment friendly.

Visual Studio Code, Visual Studio and Rider will run the debug profile Docker-Regtest.
This will run a BTCPayServer instance connecting to the services in your Docker service, so you can easily debug and step through the code.

## Videos

For more information check out these videos:

* [How to contribute to BTCPay Server Development (Windows)](https://youtube.com/watch?v=ZePbMPSIvHM) by Nicolas Dorier
* [Setting up BTCPayServer development environment on Linux (Ubuntu)](https://youtube.com/watch?v=j486T_Rk-yw) by RockStarDev
* [BTCPay Server Development - Testing pull request, payments (MacOS)](https://youtube.com/watch?v=GWR_CcMsEV0) by Pavlenex

and these notes:

* [How to get started with development](https://github.com/btcpayserver/btcpayserver/blob/master/BTCPayServer.Tests/README.md) by Nicolas Dorier (covering relevant docker commands, paying regtest invoices)


text:
# UnofficialALTestApp
Display logged in user's anime or manga list.

**Note: You will need an [Anilist](https://anilist.co) account to use this app.**

Steps:
1. Open index.html in the browser and log in using the link.
2. Click the `Authorize` button.
3. Paste the code provided in place of `{code}` on js/authorization.js.
4. Install node and type `node js/authorization.js` (or just `node authorization.js` if already in the js directory) in the terminal.
5. Copy the token from the terminal.
6. Paste the token provided in place of `{token}` on js/script.js.
7. Open index.html in the browser (or refresh if already on index.html).
8. The rest are self-explanatory.

repository: josephyhu/UnofficialALTestApp

repo_name: josephyhu/UnofficialALTestApp


text:
Document: "virtualNetworkGateway"

Path: "/tmp/azure-rest-api-specs/specification/network/resource-manager/Microsoft.Network/stable/2018-11-01/virtualNetworkGateway.json")

## VirtualNetworkGatewayConnection

A common class for general resource information

```puppet
azure_virtual_network_gateway_connection {
  api_version => "api_version",
  etag => "etag (optional)",
  id => "id (optional)",
  location => "location (optional)",
  parameters => "parameters",
  properties => $azure_virtual_network_gateway_connection_properties_format
  resource_group_name => "resource_group_name",
  subscription_id => "subscription_id",
  tags => "tags (optional)",
}
```

| Name        | Type           | Required       | Description       |
| ------------- | ------------- | ------------- | ------------- |
|api_version | String | true | Client API version. |
|etag | String | false | Gets a unique read-only string that changes whenever the resource is updated. |
|id | String | false | Resource ID. |
|location | String | false | Resource location. |
|parameters | Hash | true | Parameters supplied to the create or update virtual network gateway connection operation. |
|properties | [VirtualNetworkGatewayConnectionPropertiesFormat](#virtualnetworkgatewayconnectionpropertiesformat) | true | Properties of the virtual network gateway connection. |
|resource_group_name | String | true | The name of the resource group. |
|subscription_id | String | true | The subscription credentials which uniquely identify the Microsoft Azure subscription. The subscription ID forms part of the URI for every service call. |
|tags | Hash | false | Resource tags. |

## VirtualNetworkGatewayConnectionPropertiesFormat

VirtualNetworkGatewayConnection properties

```puppet
$azure_virtual_network_gateway_connection_properties_format = {
  authorizationKey => "authorizationKey (optional)",
  connectionProtocol => $azure_connection_protocol
  connectionType => "connectionType",
  enableBgp => "enableBgp (optional)",
  expressRouteGatewayBypass => "expressRouteGatewayBypass (optional)",
  ipsecPolicies => $azure_ipsec_policy
  localNetworkGateway2 => $azure_local_network_gateway
  peer => $azure_sub_resource
  resourceGuid => "resourceGuid (optional)",
  routingWeight => "1234 (optional)",
  sharedKey => "sharedKey (optional)",
  tunnelConnectionStatus => $azure_tunnel_connection_health
  usePolicyBasedTrafficSelectors => "usePolicyBasedTrafficSelectors (optional)",
  virtualNetworkGateway1 => $azure_virtual_network_gateway
  virtualNetworkGateway2 => $azure_virtual_network_gateway
}
```

| Name        | Type           | Required       | Description       |
| ------------- | ------------- | ------------- | ------------- |
|authorizationKey | String | false | The authorizationKey. |
|connectionProtocol | [ConnectionProtocol](#connectionprotocol) | false | Connection protocol used for this connection |
|connectionType | String | true | Gateway connection type. Possible values are: 'Ipsec','Vnet2Vnet','ExpressRoute', and 'VPNClient. |
|enableBgp | Boolean | false | EnableBgp flag |
|expressRouteGatewayBypass | Boolean | false | Bypass ExpressRoute Gateway for data forwarding |
|ipsecPolicies | [IpsecPolicy](#ipsecpolicy) | false | The IPSec Policies to be considered by this connection. |
|localNetworkGateway2 | [LocalNetworkGateway](#localnetworkgateway) | false | The reference to local network gateway resource. |
|peer | [SubResource](#subresource) | false | The reference to peerings resource. |
|resourceGuid | String | false | The resource GUID property of the VirtualNetworkGatewayConnection resource. |
|routingWeight | Integer | false | The routing weight. |
|sharedKey | String | false | The IPSec shared key. |
|tunnelConnectionStatus | [TunnelConnectionHealth](#tunnelconnectionhealth) | false | Collection of all tunnels' connection health status. |
|usePolicyBasedTrafficSelectors | Boolean | false | Enable policy-based traffic selectors. |
|virtualNetworkGateway1 | [VirtualNetworkGateway](#virtualnetworkgateway) | true | The reference to virtual network gateway resource. |
|virtualNetworkGateway2 | [VirtualNetworkGateway](#virtualnetworkgateway) | false | The reference to virtual network gateway resource. |

## ConnectionProtocol

Gateway connection protocol. Possible values are: 'IKEv2', 'IKEv1'.

```puppet
$azure_connection_protocol = {
}
```

| Name        | Type           | Required       | Description       |
| ------------- | ------------- | ------------- | ------------- |

## IpsecPolicy

An IPSec Policy configuration for a virtual network gateway connection

```puppet
$azure_ipsec_policy = {
  dhGroup => "dhGroup",
  ikeEncryption => "ikeEncryption",
  ikeIntegrity => "ikeIntegrity",
  ipsecEncryption => "ipsecEncryption",
  ipsecIntegrity => "ipsecIntegrity",
  pfsGroup => "pfsGroup",
  saDataSizeKilobytes => "1234",
  saLifeTimeSeconds => "1234",
}
```

| Name        | Type           | Required       | Description       |
| ------------- | ------------- | ------------- | ------------- |
|dhGroup | String | true | The DH Groups used in IKE Phase 1 for initial SA. |
|ikeEncryption | String | true | The IKE encryption algorithm (IKE phase 2). |
|ikeIntegrity | String | true | The IKE integrity algorithm (IKE phase 2). |
|ipsecEncryption | String | true | The IPSec encryption algorithm (IKE phase 1). |
|ipsecIntegrity | String | true | The IPSec integrity algorithm (IKE phase 1). |
|pfsGroup | String | true | The Pfs Groups used in IKE Phase 2 for new child SA. |
|saDataSizeKilobytes | Integer | true | The IPSec Security Association (also called Quick Mode or Phase 2 SA) payload size in KB for a site to site VPN tunnel. |
|saLifeTimeSeconds | Integer | true | The IPSec Security Association (also called Quick Mode or Phase 2 SA) lifetime in seconds for a site to site VPN tunnel. |

## LocalNetworkGateway

A common class for general resource information

```puppet
$azure_local_network_gateway = {
  etag => "etag (optional)",
  id => "id (optional)",
  location => "location (optional)",
  properties => $azure_local_network_gateway_properties_format
  tags => "tags (optional)",
}
```

| Name        | Type           | Required       | Description       |
| ------------- | ------------- | ------------- | ------------- |
|etag | String | false | A unique read-only string that changes whenever the resource is updated. |
|id | String | false | Resource ID. |
|location | String | false | Resource location. |
|properties | [LocalNetworkGatewayPropertiesFormat](#localnetworkgatewaypropertiesformat) | true | Properties of the local network gateway. |
|tags | Hash | false | Resource tags. |

## LocalNetworkGatewayPropertiesFormat

LocalNetworkGateway properties

```puppet
$azure_local_network_gateway_properties_format = {
  bgpSettings => $azure_bgp_settings
  gatewayIpAddress => "gatewayIpAddress (optional)",
  localNetworkAddressSpace => $azure_address_space
  resourceGuid => "resourceGuid (optional)",
}
```

| Name        | Type           | Required       | Description       |
| ------------- | ------------- | ------------- | ------------- |
|bgpSettings | [BgpSettings](#bgpsettings) | false | Local network gateway's BGP speaker settings. |
|gatewayIpAddress | String | false | IP address of local network gateway. |
|localNetworkAddressSpace | [AddressSpace](#addressspace) | false | Local network site address space. |
|resourceGuid | String | false | The resource GUID property of the LocalNetworkGateway resource. |

## BgpSettings

BGP settings details

```puppet
$azure_bgp_settings = {
  asn => "1234 (optional)",
  bgpPeeringAddress => "bgpPeeringAddress (optional)",
  peerWeight => "1234 (optional)",
}
```

| Name        | Type           | Required       | Description       |
| ------------- | ------------- | ------------- | ------------- |
|asn | Integer | false | The BGP speaker's ASN. |
|bgpPeeringAddress | String | false | The BGP peering address and BGP identifier of this BGP speaker. |
|peerWeight | Integer | false | The weight added to routes learned from this BGP speaker. |

## AddressSpace

AddressSpace contains an array of IP address ranges that can be used by subnets of the virtual network.

```puppet
$azure_address_space = {
  addressPrefixes => "addressPrefixes (optional)",
}
```

| Name        | Type           | Required       | Description       |
| ------------- | ------------- | ------------- | ------------- |
|addressPrefixes | Array | false | A list of address blocks reserved for this virtual network in CIDR notation. |

## SubResource

Reference to another subresource.

```puppet
$azure_sub_resource = {
  id => "id (optional)",
}
```

| Name        | Type           | Required       | Description       |
| ------------- | ------------- | ------------- | ------------- |
|id | String | false | Resource ID. |

## TunnelConnectionHealth

VirtualNetworkGatewayConnection properties

```puppet
$azure_tunnel_connection_health = {
}
```

| Name        | Type           | Required       | Description       |
| ------------- | ------------- | ------------- | ------------- |

## VirtualNetworkGateway

A common class for general resource information

```puppet
$azure_virtual_network_gateway = {
  etag => "etag (optional)",
  id => "id (optional)",
  location => "location (optional)",
  properties => $azure_virtual_network_gateway_properties_format
  tags => "tags (optional)",
}
```

| Name        | Type           | Required       | Description       |
| ------------- | ------------- | ------------- | ------------- |
|etag | String | false | Gets a unique read-only string that changes whenever the resource is updated. |
|id | String | false | Resource ID. |
|location | String | false | Resource location. |
|properties | [VirtualNetworkGatewayPropertiesFormat](#virtualnetworkgatewaypropertiesformat) | true | Properties of the virtual network gateway. |
|tags | Hash | false | Resource tags. |

## VirtualNetworkGatewayPropertiesFormat

VirtualNetworkGateway properties

```puppet
$azure_virtual_network_gateway_properties_format = {
  activeActive => "activeActive (optional)",
  bgpSettings => $azure_bgp_settings
  enableBgp => "enableBgp (optional)",
  gatewayDefaultSite => $azure_sub_resource
  gatewayType => "gatewayType (optional)",
  ipConfigurations => $azure_virtual_network_gateway_ip_configuration
  resourceGuid => "resourceGuid (optional)",
  sku => $azure_virtual_network_gateway_sku
  vpnClientConfiguration => $azure_vpn_client_configuration
  vpnType => "vpnType (optional)",
}
```

| Name        | Type           | Required       | Description       |
| ------------- | ------------- | ------------- | ------------- |
|activeActive | Boolean | false | ActiveActive flag |
|bgpSettings | [BgpSettings](#bgpsettings) | false | Virtual network gateway's BGP speaker settings. |
|enableBgp | Boolean | false | Whether BGP is enabled for this virtual network gateway or not. |
|gatewayDefaultSite | [SubResource](#subresource) | false | The reference of the LocalNetworkGateway resource which represents local network site having default routes. Assign Null value in case of removing existing default site setting. |
|gatewayType | String | false | The type of this virtual network gateway. Possible values are: 'Vpn' and 'ExpressRoute'. |
|ipConfigurations | [VirtualNetworkGatewayIPConfiguration](#virtualnetworkgatewayipconfiguration) | false | IP configurations for virtual network gateway. |
|resourceGuid | String | false | The resource GUID property of the VirtualNetworkGateway resource. |
|sku | [VirtualNetworkGatewaySku](#virtualnetworkgatewaysku) | false | The reference of the VirtualNetworkGatewaySku resource which represents the SKU selected for Virtual network gateway. |
|vpnClientConfiguration | [VpnClientConfiguration](#vpnclientconfiguration) | false | The reference of the VpnClientConfiguration resource which represents the P2S VpnClient configurations. |
|vpnType | String | false | The type of this virtual network gateway. Possible values are: 'PolicyBased' and 'RouteBased'. |

## VirtualNetworkGatewayIPConfiguration

IP configuration for virtual network gateway

```puppet
$azure_virtual_network_gateway_ip_configuration = {
  etag => "etag (optional)",
  id => "id (optional)",
  name => "name (optional)",
  properties => $azure_virtual_network_gateway_ip_configuration_properties_format
}
```

| Name        | Type           | Required       | Description       |
| ------------- | ------------- | ------------- | ------------- |
|etag | String | false | A unique read-only string that changes whenever the resource is updated. |
|id | String | false | Resource ID. |
|name | String | false | The name of the resource that is unique within a resource group. This name can be used to access the resource. |
|properties | [VirtualNetworkGatewayIPConfigurationPropertiesFormat](#virtualnetworkgatewayipconfigurationpropertiesformat) | false | Properties of the virtual network gateway ip configuration. |

## VirtualNetworkGatewayIPConfigurationPropertiesFormat

Properties of VirtualNetworkGatewayIPConfiguration

```puppet
$azure_virtual_network_gateway_ip_configuration_properties_format = {
  privateIPAllocationMethod => "privateIPAllocationMethod (optional)",
  publicIPAddress => $azure_sub_resource
  subnet => $azure_sub_resource
}
```

| Name        | Type           | Required       | Description       |
| ------------- | ------------- | ------------- | ------------- |
|privateIPAllocationMethod | String | false | The private IP allocation method. Possible values are: 'Static' and 'Dynamic'. |
|publicIPAddress | [SubResource](#subresource) | false | The reference of the public IP resource. |
|subnet | [SubResource](#subresource) | false | The reference of the subnet resource. |

## VirtualNetworkGatewaySku

VirtualNetworkGatewaySku details

```puppet
$azure_virtual_network_gateway_sku = {
  capacity => "1234 (optional)",
  name => "name (optional)",
  tier => "tier (optional)",
}
```

| Name        | Type           | Required       | Description       |
| ------------- | ------------- | ------------- | ------------- |
|capacity | Integer | false | The capacity. |
|name | String | false | Gateway SKU name. |
|tier | String | false | Gateway SKU tier. |

## VpnClientConfiguration

VpnClientConfiguration for P2S client.

```puppet
$azure_vpn_client_configuration = {
  radiusServerAddress => "radiusServerAddress (optional)",
  radiusServerSecret => "radiusServerSecret (optional)",
  vpnClientAddressPool => $azure_address_space
  vpnClientIpsecPolicies => $azure_ipsec_policy
  vpnClientProtocols => "vpnClientProtocols (optional)",
  vpnClientRevokedCertificates => $azure_vpn_client_revoked_certificate
  vpnClientRootCertificates => $azure_vpn_client_root_certificate
}
```

| Name        | Type           | Required       | Description       |
| ------------- | ------------- | ------------- | ------------- |
|radiusServerAddress | String | false | The radius server address property of the VirtualNetworkGateway resource for vpn client connection. |
|radiusServerSecret | String | false | The radius secret property of the VirtualNetworkGateway resource for vpn client connection. |
|vpnClientAddressPool | [AddressSpace](#addressspace) | false | The reference of the address space resource which represents Address space for P2S VpnClient. |
|vpnClientIpsecPolicies | [IpsecPolicy](#ipsecpolicy) | false | VpnClientIpsecPolicies for virtual network gateway P2S client. |
|vpnClientProtocols | Array | false | VpnClientProtocols for Virtual network gateway. |
|vpnClientRevokedCertificates | [VpnClientRevokedCertificate](#vpnclientrevokedcertificate) | false | VpnClientRevokedCertificate for Virtual network gateway. |
|vpnClientRootCertificates | [VpnClientRootCertificate](#vpnclientrootcertificate) | false | VpnClientRootCertificate for virtual network gateway. |

## VpnClientRevokedCertificate

VPN client revoked certificate of virtual network gateway.

```puppet
$azure_vpn_client_revoked_certificate = {
  etag => "etag (optional)",
  id => "id (optional)",
  name => "name (optional)",
  properties => $azure_vpn_client_revoked_certificate_properties_format
}
```

| Name        | Type           | Required       | Description       |
| ------------- | ------------- | ------------- | ------------- |
|etag | String | false | A unique read-only string that changes whenever the resource is updated. |
|id | String | false | Resource ID. |
|name | String | false | The name of the resource that is unique within a resource group. This name can be used to access the resource. |
|properties | [VpnClientRevokedCertificatePropertiesFormat](#vpnclientrevokedcertificatepropertiesformat) | false | Properties of the vpn client revoked certificate. |

## VpnClientRevokedCertificatePropertiesFormat

Properties of the revoked VPN client certificate of virtual network gateway.

```puppet
$azure_vpn_client_revoked_certificate_properties_format = {
  thumbprint => "thumbprint (optional)",
}
```

| Name        | Type           | Required       | Description       |
| ------------- | ------------- | ------------- | ------------- |
|thumbprint | String | false | The revoked VPN client certificate thumbprint. |

## VpnClientRootCertificate

VPN client root certificate of virtual network gateway

```puppet
$azure_vpn_client_root_certificate = {
  etag => "etag (optional)",
  id => "id (optional)",
  name => "name (optional)",
  properties => $azure_vpn_client_root_certificate_properties_format
}
```

| Name        | Type           | Required       | Description       |
| ------------- | ------------- | ------------- | ------------- |
|etag | String | false | A unique read-only string that changes whenever the resource is updated. |
|id | String | false | Resource ID. |
|name | String | false | The name of the resource that is unique within a resource group. This name can be used to access the resource. |
|properties | [VpnClientRootCertificatePropertiesFormat](#vpnclientrootcertificatepropertiesformat) | true | Properties of the vpn client root certificate. |

## VpnClientRootCertificatePropertiesFormat

Properties of SSL certificates of application gateway

```puppet
$azure_vpn_client_root_certificate_properties_format = {
  publicCertData => "publicCertData",
}
```

| Name        | Type           | Required       | Description       |
| ------------- | ------------- | ------------- | ------------- |
|publicCertData | String | true | The certificate public data. |

## CRUD operations

Here is a list of endpoints that we use to create, read, update and delete the VirtualNetworkGatewayConnection

| Operation | Path | Verb | Description | OperationID |
| ------------- | ------------- | ------------- | ------------- | ------------- |
|Create|`/subscriptions/%{subscription_id}/resourceGroups/%{resource_group_name}/providers/Microsoft.Network/connections/%{virtual_network_gateway_connection_name}`|Put|Creates or updates a virtual network gateway connection in the specified resource group.|VirtualNetworkGatewayConnections_CreateOrUpdate|
|List - list all|``||||
|List - get one|`/subscriptions/%{subscription_id}/resourceGroups/%{resource_group_name}/providers/Microsoft.Network/connections/%{virtual_network_gateway_connection_name}`|Get|Gets the specified virtual network gateway connection by resource group.|VirtualNetworkGatewayConnections_Get|
|List - get list using params|`/subscriptions/%{subscription_id}/resourceGroups/%{resource_group_name}/providers/Microsoft.Network/connections`|Get|The List VirtualNetworkGatewayConnections operation retrieves all the virtual network gateways connections created.|VirtualNetworkGatewayConnections_List|
|Update|`/subscriptions/%{subscription_id}/resourceGroups/%{resource_group_name}/providers/Microsoft.Network/connections/%{virtual_network_gateway_connection_name}`|Put|Creates or updates a virtual network gateway connection in the specified resource group.|VirtualNetworkGatewayConnections_CreateOrUpdate|
|Delete|`/subscriptions/%{subscription_id}/resourceGroups/%{resource_group_name}/providers/Microsoft.Network/connections/%{virtual_network_gateway_connection_name}`|Delete|Deletes the specified virtual network Gateway connection.|VirtualNetworkGatewayConnections_Delete|


text:
---
{
  "title": "Pay to EndPoint",
  "description": "A detailed explanation of how Pay to EndPoint can be used to hide transaction metadata in a collaborative CoinJoin. This is the Wasabi documentation, an archive of knowledge about the open-source, non-custodial and privacy-focused Bitcoin wallet for desktop."
}
---

# Pay To EndPoint

[[toc]]

---

> Satoshi's Vision + CoinJoin + Bulletproofs = Sad Blockchain Analysts

I attended a brainstorming event on Bitcoin privacy.
In this article, I will assume all participants of this meeting would like to remain anonymous, unless she or he explicitly asks me to properly credit her or him.
To keep this post coherent, I will refer to specific participants as Jessie, James and Meowth in an interchangeable, (not pseudonymous) way, hence Team Rocket.

My goal today is to put out one of the ideas of this meeting in a digestible way.
I assume some other Team Rocket member will do the same, in their own way.
But first, let me walk you through some basic concepts.
But before that, I’d like to credit some attendees and link to their articles on the event.

#### Non-Anonymous Attendees

- [Matthew Haywood](https://twitter.com/wintercooled)
- [Adam Back](https://twitter.com/adam3us)
- [Grubles](https://twitter.com/notgrubles)
- [Danger Shony](https://twitter.com/dangershony)
- [Tim Ruffing](https://twitter.com/real_or_random)
- [Adam Gibson](https://twitter.com/waxwing__)

## The Basics

### P2IP

> Bitcoin 0.1 supported Bitcoin addresses just like today, but it also had a “pay to IP address” feature.
>When you used this, you’d connect to the IP and get the sender’s full public key, and then send to that.
>[…] Pay-to-IP was removed around 0.5, I think. — [Theymos](https://reddit.com/u/theymos), 2016

Satoshi built a feature into Bitcoin, called P2IP.
This has terrible security and privacy drawbacks, so it was removed.
Little we knew, when we came up with our Pay To EndPoint (P2EP) scheme, that we reinvented this feature, but in a secure and a privacy enhancing way.
In our scheme the EndPoint can not only be an IP address, but also a domain, preferably a Tor onion.

When Jessie realized what we have done, she jokingly noted: _"We perfected Satoshi's vision!"_

### Bulletproofs

Bulletproofs (BP) is a zero-knowledge proof system, often cited together with Confidential Transactions.
 However we use BP for a different purpose.
We utilize it to prevent a UTXO spoofing attack, that I will expand upon later.
However James argues, we do not even need to deal with the implementation complexity of Bulletproofs, since a simple cut and choose defense might be sufficient.

It is also worth noting, that our scheme does not require any soft or hard fork.

### CoinJoin

My regular readers are probably sick of me repeatedly explaining CoinJoin, (CJ) so feel free to skip this two sentences here.
CJ stands for joining multiple users’ inputs into one transaction.
It is also worth pointing out that most CJ scheme uses equal sized outputs.
If it does not, then it provides zero privacy. But that is just an oversimplification, I often say.
No matter how unintuitive it may seem, there are CJs with unequal sized outputs those still provide privacy, because the adversary have to recognize this was a CJ transaction in the first place.
I will expand upon it later.

### Blockchain Heuristics

Blockchain Heuristics are the methods Blockchain Analytics uses to deanonymize you.
These companies are founded on these heuristics. The scheme Team Rocket is proposing breaks some of these heuristics to the point that today’s Blockchain Analysis would be instantly broken, even if you do not use our scheme, since it would be really hard to figure out if you used P2EP or not.

#### Heuristic 1: All inputs are co-owned.

This means, if you are joining together more than one input, then it must all come from you.
You could say that CJ breaks this assumption instantly, but (equal sized) CJ is easily identifiable, so that is easy to handle.

#### Mixing Heuristics: Subset-Sum Analysis.

This heuristics can be applied to CJ transactions, (and all mixing technique in general,) where the output sizes are not equal.
For example if James and Meowth joins their 1 BTC and 2 BTC coins together and the outputs would be 2 BTC and 1 BTC, it is easy to delink that transaction.
Note, it is only possible if CJ transaction was identified.

#### Heuristics Meta: What Doesn’t Exist Doesn’t Exist.

It is evident, but it is worth pointing out.
If Blockchain Analysis sees a transaction on the Blockchain that theoretically can be interpreted in many different ways, but in practice only one way of interpretation is implemented at that point in time, then that interpretation is what Blockchain Analysis assumes.
This is why it is really easy to be anonymous with Bitcoin today.
Just get familiar with some Blockchain Analysis heuristics, break them manually and they will interpret your transaction in the wrong way with 99% accuracy, because you are the only person who builds such transaction in the world and they are not aware of them.
To denounce the evils of Blockchain Analytics!

To go to town with heuristics, see Adam Gibson’s [Building on Bitcoin talk](https://www.youtube.com/watch?v=XORDEX-RrAI&feature=youtu.be&t=23359), Kristov Atlas’s [CoinJoin Sudoku](https://www.coinjoinsudoku.com), where he broke Blockchain info’s now discontinued SharedCoin feature and [Nick Jonas’s 2016 talk](https://www.youtube.com/watch?v=HScK4pkDNds) in a Zurich Meetup.

### Clusterfuck Wallet

The last concept I would like to get you familiar with is something I have written about.
It is the [clusterfuck wallet](https://medium.com/@nopara73/new-bitcoin-anonymity-technique-the-clusterfuck-wallet-d48aa1787324).
This is either the craziest idea I ever had or the most brilliant one
medium.com

> I remember reading a while ago Mircea Popescu’s ridiculous article, where he states something like Bitcoin is perfectly anonymous.
>His reasoning went like this: Blockchain analysis cannot make any reasonable conclusion, because a Bitcoin transaction can be interpreted in many different ways. — Ádám Ficsor

The goal of this wallet was to make Blockchain Heuristics as unreliable as it can get, by take advantage of Heuristics Meta: What Doesn’t Exist Doesn’t Exist.
The idea is: make them to exist. With that creating chaos, or “clusterfuck” if you like.
Turns out P2EP is a great tool to bring many exotic transactions into existence, not only the one the Team Moon Rocket is proposing here.

## The Protocol

### Part 1: End-To-End Connection Establishment

The idea is to make the recipient participate in the transaction, which would be great for many different reasons, I will come back to later.
For now, we had a practical problem to solve.
How do we establish a connection between the sender and the recipient in a P2P way?

Jessie’s idea was to extend the [BIP21 Bitcoin URI Scheme](/using-wasabi/BIPs.md#bip-21-uri-scheme) with an endpoint.
Somehow like this:
`bitcoin:175tWpb8K1S7NmH4Zx6rewF9WQrcZv245W?endpoint=http://3g2upl4pq6kufc4m.onion`

In this case, first the sender would try to figure out if the recipient is online.
If it is, then make a joint transaction together.
If it is not, then make the transaction in the traditional way.
The problem is [BIP 21](/using-wasabi/BIPs.md#bip-21-uri-scheme) is not fully adopted and it would be confusing to some user to copypaste a [BIP 21](/using-wasabi/BIPs.md#bip-21-uri-scheme) uri instead of just an address.
Team Rocket still debates if this is acceptable or not.

**James has provided an arguably infeasible solution to this issue. What if we would generate the endpoint from the given Bitcoin address?**

In this case, UX would stay the same: the receiver gives a Bitcoin address to the sender.
However in the background the sender would try to establish a connection to the location corresponding to the Bitcoin address given and if that does not work, the sender sends the transaction normally.

The impracticality of James’s idea is that we would need either our own anonymity network to achieve this, like a fork of Tor or we would have to convince an existing anonymity network to incorporate our scheme.
Nevertheless, the idea is novel and worth keeping in mind.

### Part 2: Sender-Receiver CoinJoin

It may seem like I do not even have to say more, because negotiating a transaction where the sender and the recipient both participates is not that hard, especially that they usually trust each other, right?
Wrong.
Their was a point where Team Rocket gave up on this scheme, because they thought they encountered an unsolvable issue.

What if a random sender connects to a publicly known receiver, start negotiating the transaction and ends up not signing in the end? So the sender would learn a UTXO of the receiver and run away with that sensitive information, without following up.

In the most desperate moment Meowth had an idea: What if the receiver would feed the sender with fake UTXOs for a while, forcing the sender to sign a final transaction every time, but only the final, transaction with the receiver’s real UTXO would be broadcasted at the end?

In this case, if the sender is malicious and does not follow up, the only thing he can learn is “there is a small chance of probability that this utxo is the receiver’s.”
We called this to **The Serial Method**.
James quickly came up with **The Parallel Method**, where many UTXOs would be sent in parallel and one of them is the real one.
It reduces the number of rounds, however it tells the attacker that one of the UTXO is the real UTXO.

And finally Jessie came up with something elegant: **The Bulletproofs Method**.
But I will not pretend that I fully understand it and I will leave it to others to expand upon the exact way of doing this.
The bottom line is, both the Serial and the Parallel method leaks a small amount of information, the Bulletproofs method does not leak any, and it does not need monitoring the Blockchain to select fake UTXOs.

### Extensions

Before I explain the novelty of Part 2: Sender-Receiver CoinJoin protocol, I would like to point out that, Part 1: End-To-End Connection Establishment protocol has a side effect, namely it can be used to facilitate many things, I described in the Clusterfuck Wallet post.
One thing this can easily facilitate is merge avoidance.
If the sender would want to join multiple coins together, it could ask for multiple Bitcoin addresses from the receiver and send the coins one by one.
It could also establish a future cooperation with the receiver.
For example, the receiver, if online could participate in later transactions of the sender, thus breaking Heuristics 1 and Meta.
But really, if this P2EP would get adopted, the limit of the number of strange schemes, all breaking Blockchain Analysis assumptions is just the developer’s imagination.

## The Novelty of Sender-Receiver Scheme

If Blockchain Analysis looks at the transaction above, it will deduct:

- The sender paid 6 BTC to the receiver.
- The largest coin of the sender was 5 BTC.
- The change is 2 BTC.

And some other things.
However if it the receiver participated in the transaction above, then this transaction could be interpreted in multiple ways, **breaking some blockchain analysis heuristics, not only the transaction level, but globally!** Remember Heuristics Meta?
What does not exist, that does not exist.
Thus if some people start using strange schemes, that affects how Blockchain analysis interprets other transactions, too.

#### Side effect 1:
In exchange for the privacy benefit, the sender has to pay more fees than a normal transaction.
It is a con for the sender, but a pro for the receiver, since the receiver does not have to consolidate its coin later.
Also it is a pro from a global point of view, due to its UTXO consolidation effects, I will detail a few lines below.

#### Side effect 2:
Harder to coordinate, since the receiver must be online and the payment will take a few seconds longer.
But it is arguable, that if everyone would be able to use this scheme all the time, then that would be the new pattern, thus the new reliable Heuristics for Blockchain analysis.
It is crucial that not everyone use this pattern all the time. Its practical limitations achieves its goal.

#### Side effect 3:
Receiver must have a hot wallet.
This is problematic in a buyer to merchant payment, since merchants usually receive coins to cold wallets.
While my above comment still applies here, notice that this notion makes our problem of UTXO spying a corner case?
That attack only applied if “the receiver is publicly known.”
In peer to peer, person to person payments, the attacker would have to somehow first acquire an endpoint to a receiver.

#### Side effect 4:
Wallets must be deencrypted.
Since many wallets only send-time decrypt the private keys, it would need to decrypt receive-time, too.
There are multiple ways to implement this.
It could be implemented in address generation time, which in our case it is also an endpoint generation time.
But if it is strictly implemented at receive time, the UTXO spying defense may not be needed at all.

##### Side effect 5:
It helps with UTXO bloat.
Let us assume the above transaction is a Sender-Receiver transaction.
Then, if the receiver would not participate, that would mostly result in a one input, two output transaction.
This also means, the receiver would not have to consolidate its UTXOs later, which is a huge win for privacy.

## Conclusion

Overall, I feel this scheme can help a lot in Bitcoin’s privacy globally, especially if it is mixed with other clusterfuck wallet techniques.
However, it must be said, there are some practical problems with it on the receiver’s side. Nevertheless it is simple to implement on the sender side and it affects globally everyone’s privacy, even those who do not use this scheme.

In the end… at the very least.
Crazy ideas always worth writing down.
Sometimes they end up changing things.

repository: PulpCattel/WasabiDoc

repo_name: PulpCattel/WasabiDoc


text:
# Design Proposal for Anomaly Detection

## Motivation
Provide support for anomaly detection
## Goals
* Quickly identify and plot anomalies in time series data.
* Create interactive visualizations.
* Provide a helpful example notebook.

## Non-Goals
* Automatic visualizations (at the click of a button), such as GUI's. Proposed [here](https://github.com/data-describe/data-describe/blob/master/docs/designs/UI.md).
* Creating notebooks for specific use cases i.e. sensor data, accelerometers, manufacturing.
## UI or API
* 2-D plot of time series data with confidence band.
* Use markers/colors to highlight anomalies.
* Interactive plots for selecting time window.

## Design
There are multiple design aspects to be considered when creating this functionality.
1. Supervised: Partitioning the data into train and test sets to calculate the confidence bands and error rates.
2. Unsupervised: Fitting a model on the entire data, i.e. HDBSCAN, isolation forests, and autoencoder
3. Statistical methods: i.e. standard deviation from rolling mean, SH-ESD, and Western Electric rules
```python
import data_describe as dd
dd.anomaly_detection(df, estimator=None, mode="ts")
# estimator: Fitted model. If estimator=None, a default model will be fitted
# mode: Specific use cases such as "ts", "classification", "regression", "wer" (western electric rules), etc.

# Returns
# Time series plot with anomalies marked and prediction intervals.
# Dataframe with all the marked records that are anomalies.
```
[Medium Article](https://towardsdatascience.com/anomaly-detection-with-time-series-forecasting-c34c6d04b24a): Contains relevant time series plots and analysis.
## Alternatives Considered
[R Package](https://github.com/twitter/AnomalyDetection): Open source R package developed by twitter for anomaly detection

[FB Prophet](https://towardsdatascience.com/anomaly-detection-time-series-4c661f6f165f): Facebook time series model. Prophet can be used as an estimator in the anomaly detection widget, but would require additional dependencies.

gha_language: Python

gha_license_id: NOASSERTION


text:
---
layout: post
title: January 6, 2020 - Transferring Larvae to Ethanol Part 5
---

### Lab Work

__Transferring Larvae Samples__

I continued to transfer the oyster larvae samples from Laura's project to 70% ethanol solution. I completed transferring the rest of the samples, which consist of numbers: 125, 153A, 28, 30, 33, 47, 35B, 70, 48, 23, 138B, 127, 94, 131A, 105A, 101, 112B, 147B, 71B, 20B, 145B, 4, 40B, 84, 10, 91, 110, 117B, 73B, 148B, 32, 103B, 93, 111B, 69, 68B, 51, 136A, 8, 1, 143A, 3, 62B, 72, 97, 132A, 128, 146B, 13, 18B, 5B, 90B, 98, 45, 17B, 78B, 151B, 75A, 106, 65B, 82, 116B, 92, 129, and 64.

![transfer_samples1](https://github.com/H-Ra/h-ra.github.io/blob/master/images/transfer_samples1.JPG?raw=true)

![transfer_samples2](https://github.com/H-Ra/h-ra.github.io/blob/master/images/transfer_samples2.JPG?raw=true)

![completed_tubes3](https://github.com/H-Ra/h-ra.github.io/blob/master/images/completed_tubes3.JPG?raw=true)

Samples that were very soft include samples: 28, 30, 33, 47, 35B, 48, 23, 20B, 4, 40B, 32, 8, 3, 18B, 5B, 45, and 17B.

Samples that were somewhat soft include samples: 153A, 94, 145B, 10, 91, 110, 148B, 69, 68B, 51, 1, 97, 13, 78B, and 64.

---

### Next Steps

I will be working with Laura to learn more scientific and analytical skills using the O. angasi samples that she collected.


text:
## 11 December 2019

### [0.0.6](https://github.com/artdecocode/pompeii/compare/v0.0.5...v0.0.6)

- [package] Publish `build`.

## 29 March 2019

### [0.0.3](https://github.com/artdecocode/pompeii/compare/v0.0.2...v0.0.3)

- [package] Upgrade to _ÀLaMode_.

## 5 June 2018

### 1.0.0

- Create `pompeii` with [`mnp`][https://mnpjs.org]
- [repository]: `src`, `test`

repository: artdecocode/pompeii

gha_language: JavaScript

repo_name: artdecocode/pompeii


text:
UPGRADE
=======

## Upgrade FROM 0.2 to 0.3

* The method `increase` of `Version` is renamed to `getNextIncreaseOf`.

* The `VersionValidator` has been renamed to `ContinuesVersionsValidator`.

* The `ContinuesVersionsValidator` has api changed to separate the
  validation from providing suggestions.

  Before:

  ```php
  VersionsValidator::isVersionContinues($versions, Version::fromString('v0.2.0'), $possibleVersions);
  ```

  After:

  ```php
  $validator = new ContinuesVersionsValidator(...$existingVersions); // Expects the versions as a variadic arguments

  $result = ContinuesVersionsValidator::isContinues(Version::fromString('v0.2.0'));
  $possibleVersions = $validator->getPossibleVersions(); // Must be called after isContinues(), otherwise empty
  ```

  **Note:** The suggested versions did not take existing versions into account.

  Now, instead if a newer major or minor version already exists it only allows
  a patch release for the bounded minor version. If both 1.1 and 2.0 exist then
  1.2 is no longer suggested, nor considered an acceptable increment.

  Otherwise all possible suggestions are accepted.

repository: rollerworks/version

gha_language: PHP

repo_name: rollerworks/version


text:
# Contributing

## Project setup

- make sure node 8 and yarn are installed
- run `yarn install` to install dependencies

## Hacking

- run `yarn test-node` to execute tests in nodejs. `test` runs tests in both the browser and nodejs and `test-browsers` runs tests in the browser only (requires Firefox)
- run `yarn lint` to check your code against the rules. Please make sure it passes before contributing

### Running tests with a PostgreSQL instance

ChluIPFS has an optional SQL-backed datastore. This is automatically tested with SQLite but also works with
postgresql. To test it, create a `chlu-test` empty database in postgresql then set the following environment
variables:

- `CHLU_POSTGRESQL_USER` to the postgresql user
- `CHLU_POSTGRESQL_PASSWORD` to the postgresql password
- `CHLU_POSTGRESQL_DB` to the DB you set up (`chlu-test`)

The tests will connect to localhost at the default postgresql port. If you have issues try deleting all tables in the
database.

repository: ChluNetwork/chlu-ipfs-support

repo_name: ChluNetwork/chlu-ipfs-support


text:
Kolor - Vim Theme
=================

Colorful Vim color scheme with 256 color terminal support.

Designed to have high text readability and optimal visibility of every element, and to be eye comfortable as well.

![][screenshot]

## Installation

Just copy `kolor.vim` to `~/.vim/colors` (on Windows `<your-vim-dir>\vimfiles\colors`). Or for global accessibility, `/usr/share/vim/vimfiles/colors`.

Then add the line `colorscheme kolor` in your vimrc file, and restart Vim.

If you use Pathogen or Vundle, the directory structure is compatible.

For Arch Linux users, there is also a [PKGBUILD][pkgbuild] available in the AUR, created by [graysky][graysky2].

## Options

You can put the following lines before `colorscheme kolor` in your vimrc.

    let g:kolor_italic=1                    " Enable italic. Default: 1
    let g:kolor_bold=1                      " Enable bold. Default: 1
    let g:kolor_underlined=0                " Enable underline for 'Underlined'. Default: 0
    let g:kolor_alternative_matchparen=0    " Gray 'MatchParen' color. Default: 0

## Donations

If you want to make a donation, rather, consider to help the Uganda children on Vim's website.

This color scheme is dedicated to them.

## Self-promotion

If you like `kolor.vim` follow the GitHub [repository][repository], and don't forget to vote for it on Vim.org! ([vimscript #4339][script]).

[script]: http://www.vim.org/scripts/script.php?script_id=4339
[repository]: https://github.com/zeis/vim-kolor
[screenshot]: https://lh5.googleusercontent.com/-z7CGCLXhTNQ/UM-4XPy52fI/AAAAAAAAAHo/iCFTSqaoapA/s852/kolor-screenshot.jpg
[pkgbuild]: https://aur.archlinux.org/packages/vim-kolor/
[graysky2]: https://github.com/graysky2

## Support & Development

Feel free to contribute, develop ports or report any problems. I'll be happy to give my support.

extension: markdown


text:
---
title: "React NextJS app using Prisma and Postgres (Part 1)"
date: "2023-07-20"
---

Part 1 of a React tutorial on how to create a CRUD webapp using Next.js 13, Prisma and Postgres.

<!-- end -->

# Installation

I used create-next-app, which sets up everything automatically for me. To create the project, run:

```bash
npx create-next-app@latest
```

On installation, I had to make some decisions. I chose as follows:

```
1. What is your project named? my-crud-app
2. Would you like to use TypeScript? Yes
3. Would you like to use ESLint? Yes
4. Would you like to use Tailwind CSS? No
5. Would you like to use `src/` directory? No
6. Would you like to use App Router? (recommended) Yes
7. Would you like to customize the default import alias? No
```

TypeScript and ESLint are essential for me so that was a no-brainer. I've tried to like Tailwind but I just can't. I thought I would prefer a src folder but after trying it for a bit I don't really see the point so I decided to omit it. No big deal either way though - go with whatever you prefer. App Router is essential for me as I'm learning Next.js in 2023 and this is the future so it's another no-brainer.

I chose not to customise the default import alias but you might want to depending on how you like to structure your projects. The default import alias allows you to import modules without using relative paths. So if I create a `./components` folder and a `./utils` folder at the root level of the project, then I can import modules from those folders using `@components` or `@utils` rather than using realtive paths with ends up with a lot of `../../../` in the imports.

After the prompts, create-next-app will create a folder named `my-crud-app` and install the required dependencies so I cd into that folder and type `code .` to open the project in VSCode. Then I switch back to the terminal (I can't get used to using the one inside VSCode - I perfer a separate terminal app so I can ALT-TAB between them easily) and type the following to run the app locally:

```
npm run dev
```

At this point I can visit http://localhost:3000 to view the application.

# Deployment

I'm a fan of [Continuous Delivery](/posts/on-continuous-delivery) (CD) so I want to push to production whenever I commit code, right from the start. So before writing any code, let's set-up Vercel so we can see the app running live in production from the get go.

First, I need to set-up the repo in GitHub so I goto `github.com` in a browser and create a new empty repo called `my-crud-app` without initialising it with a README or .gitignore as the project on my local machine already has them.

Then, I do the following in the terminal in the root folder of `my-crud-app`:

```
git branch -M main
git remote add origin git@github.com:pdrummond/my-crud-app.git
git push -u origin main
```

Now that git is set-up I can go over to Vercel's dashboard, add a new project and select my newly created repo in the `Import Git Repository` section by clicking the Import button. Vercel automatically recognises this is a NextJS project so I don't have to do anything else. I can just click on the `Deploy` button and without a few minutes my project is deployed and I can visit it at `https://my-crud-app.vercel.app`.

At this point I would normally set-up a domain as well but that's beyond the scope of this article so we'll just stick with the one vercel provides for us.

# Storage

Before I can start coding I need a database as this is a CRUD app after all. I'll be using Vercel's Postgres support and Prisma which is a fancy ORM that makes it easier to fetch and mutate data than using raw SQL queries.

Setting up Vercel's Postgres support couldn't be eaiser:

1. Goto to the storage tab of the project at vercel.com
2. Choose the option to create a `Postgres Serveless SQL` database
3. Select `Postgres Serverless SQL` then `Continue`
4. There is an option to change the name and region but I stick just stick to the defaults.
5. There is an option to configure the database including environments and envvars but again, I stick to the defaults.

Now the database is created I can go back to the terminal on my local machine, in the rood of my project and type:

```bash
npm i -g vercel@latest
```

This installs the vercel CLI. Next, we need to link our project to Vercel:

```
vercel link
```

I just accept the defaults here - it looks like this:

```
Vercel CLI 31.0.4
? Set up “~/src/my-crud-app”? [Y/n] y
? Which scope should contain your project? pdrummond
? Found project “pdrummond/my-crud-app. Link to it? [Y/n] y
✅  Linked to pdrummond/my-crud-app (created .vercel)
```

Next, I need to bring down the environment variables so the app knows how to connect to the Vercel db:

```
vercel env pull .env
```

_*NOTE:* Vercel recommends you pull the env vars down to a `.env.development.local` file but Prisma expects an `.env` file so that's what we'll use for now to keep things simple_

With this done, I can now focus on setting up Prisma.

Create a folder called `prisma` in the root of the project then inside the prisma folder create a file called `schema.prisma` with the following content:

```prisma
// schema.prisma

generator client {
  provider = "prisma-client-js"
}

datasource db {
  provider = "postgresql"
  url = env("POSTGRES_PRISMA_URL") // uses connection pooling
  directUrl = env("POSTGRES_URL_NON_POOLING") // uses a direct connection
  shadowDatabaseUrl = env("POSTGRES_URL_NON_POOLING") // used for migrations
}

model Post {
  id            String     @default(cuid()) @id
  title         String
  content       String?
  author        User?      @relation(fields: [authorId], references: [id])
  authorId      String?
}

model User {
  id            String      @default(cuid()) @id
  name          String?
  email         String?     @unique
  createdAt     DateTime    @default(now()) @map(name: "created_at")
  updatedAt     DateTime    @updatedAt @map(name: "updated_at")
  posts         Post[]
  @@map(name: "users")
}
```

Then I run the following command:

```
npx prisma db push
```

This tells prisma to read the schema and create the tables in the database. I could also use Prisma Migrate here to keep track of all changes to the database as I make changes but I'm going to keep it simple for now. For more info on the differences between `prisma db push` and `prisma migrate dev`, check out [this](https://stackoverflow.com/questions/68539836/difference-between-prisma-db-push-and-prisma-migrate-dev/68540791#68540791) Q&A on StackOverflow.

Now I can run Prisma Studio as follows:

```
npx prisma studio
```

This opens up a UI in the browser that lets me add a test user and a few posts by that user.

And with that, everything is set-up so we can finally start writing some actual code in Part 2, coming soon.


text:
# DRK: DynamoRIO as a Linux Kernel Module

Peter Feiner November 17, 2013

## INTRODUCTION

DRK is DynamoRIO as a loadable Linux Kernel module. When DRK is loaded, all
kernel-mode execution (system calls, interrupt & exception handlers, kernel
threads, etc.) happens under the purview of DynamoRIO whereas user-mode execution is
untouched - the inverse of normal DynamoRIO, which instruments a user-mode
process and doesn't touch kernel-mode execution.

## PREREQUISITES

Although in principle DRK should work with any Linux kernel, in practice you're
best off using Ubuntu's 2.6.32-33-generic, which shipped with Ubuntu 10.04. The
coupling arises from DRK's use of a few non-exported kernel interfaces to look
at symbols and get the kernel's memory layout. With some hacking of
kernel_linux/kernel_interface.c, you should be able to get DRK to work with
other kernel versions.

You'll also need Python and SCons.

## BUILDING

Unlike DynamoRIO, DRK doesn't use cmake for configuration and building.
Instead, it has a simple makefile (core/drk.mk) and a static configuration
header (core/configure.h). To configure DRK, you'll have to edit configure.h
and drk.mk.

To build, run these commands:

```
    cd core/
    make -f drk.mk
```

## CLIENTS

Instrumentation clients are built as separate kernel modules. Several example
clients are included in core/kernel_linux/clients. See
core/kernel_linux/modules/Makefile for how the clients are included in the
build.

## RUNNING

The core/drk.py script loads the DRK and client kernel modules and starts
instrumentation. The script reads DynamoRIO options, including the
specification of clients to load, from the dr_options file. For example, to
start DRK with the instruction counting client, instrcount:

```
    cd core/
    echo "-code_api -client_lib instrcount;1;" > dr_options
    sudo ./drk.py --run-locally
```

When the last command returns, you're running from the code cache!

To stop instrumentation, run

```
    cd core/kernel_linux
    sudo ./controller exit
```

When sudo returns, you're no longer running from the code cache.

## STATS

You can dump DynamoRIO's stats and kstats as follows:

```
    cd core/kernel_linux
    sudo ./controller stats
    sudo ./controller kstats
```

## TODO

There's a bunch of housekeeping that could be done on the DRK code:

- Use CMake to generate drk.mk and configure.h. Get rid of SCons.

- Move clients from core/kernel_linux/clients to clients/.

- Make drk.py work from any directory.

- Move drk.py to tools/.

- Licensing is a bit of a mess. All of the code should be BSD.

There are some less trivial cleanup tasks that could be done:

- Merge dynamorio, dynamorio_controller, and dr_kernel_utils into a single
      kernel module; frankly, I can't remember why I separated it into three
      separate modules in the first place.

- Replace stats & kstats ioctls with sysfs files that can just be read.

- dr_init does not show up in /proc/kallsyms, so clients have to be
      modified to use drinit instead. It would be nice to use drinit so we
      don't have to modify existing clients, but this isn't important because
      it's such a trivial change.

There's a bunch of DynamoRIO stuff that just doesn't work. Making these things
work in DRK will take quite a bit of engineering effort. Making some of these
things work _well_ will probably take some research effort :-) Here's a
list of some major things that don't work:

- Running out of space in the code cache. DRK will crash if it tries to
      reset the cache entirely when it runs out of space. Use these options to
      trigger the crash:
        -loglevel 0 -logmask 0x20 -bb_ibl_targets -disable_traces -kstats
        -code_api -no_finite_bb_cache -vm_size 6mb -reset_at_vmm_free_limit 1mb
        -report_reset_vmm_threshold 1000 -skip_out_of_vm_reserve_curiosity

- vmareas is basically a big nop in DRK. Hence DRK has no sort of code
      cache consistency in face of virtual address changes or native code
      changes. Fortunately, Linux doesn't change its code at runtime under
      normal circumstances :-) However, when it does (e.g., ftrace, kprobes,
      CPU hot plugging), you're going to have a bad time.

- 32-bit kernels.

- Shared code caches. DRK uses CPU-private caches to make interrupt
      handling simpler. See the ASPLOS paper for details.

- Traces. As far as I can remember, they work, but they don't help with
      performance at all.

- Changes to the kernel's GS segment. DRK uses Linux's CPU-private data for
      its own CPU-private data. At every entry into kernel-mode, the first
      code cache instruction is SWAP_GS; any native SWAP_GS instructions are
      suppressed. If the kernel were to change its GS usage at all, then DRK
      would break. One solution would be to emulate the kernel's use of GS,
      like userspace DynamoRIO does since the private loader patches.

There are probably a bunch of other DynamoRIO features & code paths that don't
work at all.

## FURTHER READING

My master's thesis explains the why & how of DRK:

> Peter Feiner, Angela Demke Brown, and Ashvin Goel.
> "Comprehensive Kernel Instrumentation via Dynamic Binary Translation."
> International Conference on Architectural Support for Programming Languages
>     and Operating Systems (ASPLOS-12), March 2012, London, UK.
> http://www.cs.toronto.edu/~peter/feiner_asplos_2012.pdf

repository: DynamoRIO/drk

branch_name: refs/heads/drk

gha_language: C

repo_name: DynamoRIO/drk


text:
# 用于动静脉血管分割的像素网络

![](http://qiniu.crsblog.cn/20200314140034.jpg)

注：红色动脉，绿色静脉

## 以下是原始像素网络的说明

#### PixelNet: Representation _of_ the pixels, _by_ the pixels, and _for_ the pixels.

We explore design principles for general pixel-level prediction problems, from low-level edge detection to mid-level surface normal estimation to high-level semantic segmentation. Convolutional predictors, such as the fully-convolutional network (FCN), have achieved remarkable success by exploiting the spatial redundancy of neighboring pixels through convolutional processing. Though computationally efficient, we point out that such approaches are not statistically efficient during learning precisely because spatial redundancy limits the information learned from neighboring pixels. We demonstrate that stratified sampling of pixels allows one to:

1. add diversity during batch updates, speeding up learning;

2. explore complex nonlinear predictors, improving accuracy;

3. efficiently train state-of-the-art models *tabula rasa* (i.e., _from scratch_) for diverse pixel-labeling tasks.

Our single architecture produces state-of-the-art results for semantic segmentation on PASCAL-Context dataset, surface normal estimation on NYUDv2 depth dataset, and edge detection on BSDS. We also demonstrate self-supervised representation learning via geometry. With even few data points, we achieve results better than previous approaches for unsupervised/self-supervised representation learning. More details are available on our [project page](http://www.cs.cmu.edu/~aayushb/pixelNet/).

If you found these codes useful for your research, please consider citing -

```make
@article{pixelnet,
  title={PixelNet: {R}epresentation of the pixels, by the pixels, and for the pixels},
  author={Bansal, Aayush and Chen, Xinlei, and  Russell, Bryan and Gupta, Abhinav and Ramanan, Deva},
  Journal={arXiv preprint arXiv:1702.06506},
  year={2017}
}
```

## How to use these codes?

Anyone can freely use our codes for what-so-ever purpose they want to use. Here we give a detailed instruction to set them up and use for different applications. We will also provide the state-of-the-art models that we have trained.

The codes can be downloaded using the following command:

```make
git clone --recursive https://github.com/aayushbansal/PixelNet.git
cd PixelNet
```

Our codebase is built around [caffe](http://caffe.berkeleyvision.org/). We have included a pointer to caffe as a submodule.

```make
ls tools/caffe
```

Our required layers are available within this submodule. To install Caffe, please follow the instructions on their [project page](http://caffe.berkeleyvision.org/).

## Models

We give a overview of the trained models in models/ directory.

```make
ls models
```

## Experiments

We provide scripts to train/test models in experiments/ directory.

```make
ls experiments
```

## Python Codes

The amazing [Xinlei Chen](https://www.cs.cmu.edu/~xinleic) wrote a version of code in [Python](https://github.com/endernewton/PixelNet) as well and wrote the GPU implementation of sampling layer. Check it out. We have not tested that extensively. It would be great if you could help us test that :)


text:
# AISHELL4 Speaker Diarization

## Usage
In speaker diarization process, we will only use the first channel as the SAD model and the Speaker-Embedding-Extrator do not support the multi-channel wav.

> You can find all the process in `sd/run.sh` and the comments in it.

**Here is the main stage:**
1. You should first prepare the key file `wav.scp` for our tools, the scripts `local/make_aishell4_test.sh` is just a sample, you can use any language (like perl, python or shell) as you like. **If you find some naming problems when use, you can replace this step by your own script to prepare the files**.
2. You should change your own data path and `kaldi-root` first in `run.sh` and `path.sh`.  The script `local/do_segmentations.sh` is to get the SAD result for the future work. You will find the segments file under the `$sad_result_dir`.
3. When use the `VBx` tools for the diarization, you should convert the segments to the `.lab`. Use `scripts/segment_to_lab.sh` to change the file format
4. The speaker diarization code needs two stage the speaker-embedding extract and the speaker-embedding cluster. Our baseline use the `VBx` tools to extract speaker-embeddings. The feature-extractor is inside, you don't have to prepare the feature before. **Note our scripts run in a SGE systems sor we sub the extract-embedding jobs to the queue.pl, if you do not have it, try to extract the speaker embeddings one by one**. Besides, we recommand you add a `exit 1` after the stage 3 to waiting for the extracting process finished.
5. For the speaker-embedding cluster, you can use the `run_cluster.sh` and the code will make the rttm for each audio in the wav.scp.

## Model Download

You need download the model from the [path](https://data-tx.oss-cn-hangzhou.aliyuncs.com/AISHELL-4-Code/sd-part.zip), you should mv the `exp` to the `sd/` and the `ResNet101_16kHz` to the `VBx/models`.

## Reference
1. [kaldi-sad-model](http://kaldi-asr.org/models/m12)
2. [VBx](https://github.com/BUTSpeechFIT/VBx)


text:
# Erin Goncer | Portfolio

#### A bio and list of works by EG for Epicodus, 03.18.2016

#### By **Erin Goncer**

## Description

This is my most up-to-date portfolio since beginning Epicodus. This page includes a bio of myself along with a list works from the past year and the first week of Epicodus.

## Setup/Installation Requirements

* Clone this repository
* Open file

## Support and Contact Details

For any questions, comments, or inquiries, please contact Erin Goncer.

## Technologies Used

HTML, CSS, Bootstrap

### License

*This software is licensed under the MIT license.*

Copyright (c) 2016 **Erin Goncer**


text:
ReactSharp
==========

ReactSharp is a library that provides async and [functional reactive programming]-like primitives.
It can serve as the basis for a user interface toolkit, or any other library that has a model on
which clients will listen and to which they will react.

* [API docs](http://samskivert.github.com/ReactSharp/apidocs/annotated.html) are available.

Core Concepts
-------------

ReactSharp provides three main abstractions:

* Future/Promise - asynchronous computations which eventually compute a value, along with
  mechanisms to chain and combine async computations.
* Value - a reactive value which reports to observers when it changes.
* Signal - a source of events which can be filtered, transformed, etc. Like a poor man's
  Observable, but works nicely with Value & Future.

The Java library from whence this came provides more reactive datatypes (Set, List, Map, Queue).
Perhaps ReactSharp will grow to include those as well some day.

Origins
-------

ReactSharp was ported from the [Java react] library (same author for both libraries) and made as
C-sharpy as possible given the limitated C# experience of a Java programmer forced to work in the
Unity salt mines.

Distribution
------------

ReactSharp is released under the New BSD License. The most recent version of the library is
available at http://github.com/samskivert/ReactSharp

Contact
-------

Questions, comments, and other communications should be directed to the [OOO Libraries] Google
Group.

[functional reactive programming]: http://en.wikipedia.org/wiki/Functional_reactive_programming
[Java react]: https://github.com/threerings/react
[OOO Libraries]: http://groups.google.com/group/ooo-libs

repository: tconkling/ReactSharp

repo_name: tconkling/ReactSharp


text:
# Birthday

![Prompt](./img/Prompt_1.png)

![Results](./img/Results_1.png)

A small project with specific goals:
- build on top of two libraries (FLTK and Boost)
- use a Makefile and learn Makefile best practices
- write tests
- abstract own gui files to static library

This project will be borrowing heavily from Chapters 12-16 of Bjarne Stroustrup's Programming Principles and Practice.

repository: Chrinkus/birthday

repo_name: Chrinkus/birthday