text:
# Pretraining BERT
:label:`sec_bert-pretraining`

With the BERT model implemented in :numref:`sec_bert`
and the pretraining examples generated from the WikiText-2 dataset in :numref:`sec_bert-dataset`, we will pretrain BERT on the WikiText-2 dataset in this section.

```{.python .input}
from d2l import mxnet as d2l
from mxnet import autograd, gluon, init, np, npx

npx.set_np()
```

```{.python .input}
#@tab pytorch
from d2l import torch as d2l
import torch
from torch import nn
```

To start, we load the WikiText-2 dataset as minibatches
of pretraining examples for masked language modeling and next sentence prediction.
The batch size is 512 and the maximum length of a BERT input sequence is 64.
Note that in the original BERT model, the maximum length is 512.

```{.python .input}
#@tab all
batch_size, max_len = 512, 64
train_iter, vocab = d2l.load_data_wiki(batch_size, max_len)
```

## Pretraining BERT

The original BERT has two versions of different model sizes :cite:`Devlin.Chang.Lee.ea.2018`.
The base model ($\text{BERT}_{\text{BASE}}$) uses 12 layers (transformer encoder blocks)
with 768 hidden units (hidden size) and 12 self-attention heads.
The large model ($\text{BERT}_{\text{LARGE}}$) uses 24 layers
with 1024 hidden units and 16 self-attention heads.
Notably, the former has 110 million parameters while the latter has 340 million parameters.
For demonstration with ease,
we define a small BERT, using 2 layers, 128 hidden units, and 2 self-attention heads.

```{.python .input}
net = d2l.BERTModel(len(vocab), num_hiddens=128, ffn_num_hiddens=256,
                    num_heads=2, num_layers=2, dropout=0.2)
devices = d2l.try_all_gpus()
net.initialize(init.Xavier(), ctx=devices)
loss = gluon.loss.SoftmaxCELoss()
```

```{.python .input}
#@tab pytorch
net = d2l.BERTModel(len(vocab), num_hiddens=128, norm_shape=[128],
                    ffn_num_input=128, ffn_num_hiddens=256, num_heads=2,
                    num_layers=2, dropout=0.2, key_size=128, query_size=128,
                    value_size=128, hid_in_features=128, mlm_in_features=128,
                    nsp_in_features=128)
devices = d2l.try_all_gpus()
loss = nn.CrossEntropyLoss()
```

Before defining the training loop,
we define a helper function `_get_batch_loss_bert`.
Given the shard of training examples,
this function computes the loss for both the masked language modeling and next sentence prediction tasks.
Note that the final loss of BERT pretraining
is just the sum of both the masked language modeling loss
and the next sentence prediction loss.

```{.python .input}
#@save
def _get_batch_loss_bert(net, loss, vocab_size, tokens_X_shards,
                         segments_X_shards, valid_lens_x_shards,
                         pred_positions_X_shards, mlm_weights_X_shards,
                         mlm_Y_shards, nsp_y_shards):
    mlm_ls, nsp_ls, ls = [], [], []
    for (tokens_X_shard, segments_X_shard, valid_lens_x_shard,
         pred_positions_X_shard, mlm_weights_X_shard, mlm_Y_shard,
         nsp_y_shard) in zip(
        tokens_X_shards, segments_X_shards, valid_lens_x_shards,
        pred_positions_X_shards, mlm_weights_X_shards, mlm_Y_shards,
        nsp_y_shards):
        # Forward pass
        _, mlm_Y_hat, nsp_Y_hat = net(
            tokens_X_shard, segments_X_shard, valid_lens_x_shard.reshape(-1),
            pred_positions_X_shard)
        # Compute masked language model loss
        mlm_l = loss(
            mlm_Y_hat.reshape((-1, vocab_size)), mlm_Y_shard.reshape(-1),
            mlm_weights_X_shard.reshape((-1, 1)))
        mlm_l = mlm_l.sum() / (mlm_weights_X_shard.sum() + 1e-8)
        # Compute next sentence prediction loss
        nsp_l = loss(nsp_Y_hat, nsp_y_shard)
        nsp_l = nsp_l.mean()
        mlm_ls.append(mlm_l)
        nsp_ls.append(nsp_l)
        ls.append(mlm_l + nsp_l)
        npx.waitall()
    return mlm_ls, nsp_ls, ls
```

```{.python .input}
#@tab pytorch
#@save
def _get_batch_loss_bert(net, loss, vocab_size, tokens_X,
                         segments_X, valid_lens_x,
                         pred_positions_X, mlm_weights_X,
                         mlm_Y, nsp_y):
    # Forward pass
    _, mlm_Y_hat, nsp_Y_hat = net(tokens_X, segments_X,
                                  valid_lens_x.reshape(-1),
                                  pred_positions_X)
    # Compute masked language model loss
    mlm_l = loss(mlm_Y_hat.reshape(-1, vocab_size), mlm_Y.reshape(-1)) *\
    mlm_weights_X.reshape(-1, 1)
    mlm_l = mlm_l.sum() / (mlm_weights_X.sum() + 1e-8)
    # Compute next sentence prediction loss
    nsp_l = loss(nsp_Y_hat, nsp_y)
    l = mlm_l + nsp_l
    return mlm_l, nsp_l, l
```

Invoking the two aforementioned helper functions,
the following `train_bert` function
defines the procedure to pretrain BERT (`net`) on the WikiText-2 (`train_iter`) dataset.
Training BERT can take very long.
Instead of specifying the number of epochs for training
as in the `train_ch13` function (see :numref:`sec_image_augmentation`),
the input `num_steps` of the following function
specifies the number of iteration steps for training.

```{.python .input}
def train_bert(train_iter, net, loss, vocab_size, devices, num_steps):
    trainer = gluon.Trainer(net.collect_params(), 'adam',
                            {'learning_rate': 1e-3})
    step, timer = 0, d2l.Timer()
    animator = d2l.Animator(xlabel='step', ylabel='loss',
                            xlim=[1, num_steps], legend=['mlm', 'nsp'])
    # Sum of masked language modeling losses, sum of next sentence prediction
    # losses, no. of sentence pairs, count
    metric = d2l.Accumulator(4)
    num_steps_reached = False
    while step < num_steps and not num_steps_reached:
        for batch in train_iter:
            (tokens_X_shards, segments_X_shards, valid_lens_x_shards,
             pred_positions_X_shards, mlm_weights_X_shards,
             mlm_Y_shards, nsp_y_shards) = [gluon.utils.split_and_load(
                elem, devices, even_split=False) for elem in batch]
            timer.start()
            with autograd.record():
                mlm_ls, nsp_ls, ls = _get_batch_loss_bert(
                    net, loss, vocab_size, tokens_X_shards, segments_X_shards,
                    valid_lens_x_shards, pred_positions_X_shards,
                    mlm_weights_X_shards, mlm_Y_shards, nsp_y_shards)
            for l in ls:
                l.backward()
            trainer.step(1)
            mlm_l_mean = sum([float(l) for l in mlm_ls]) / len(mlm_ls)
            nsp_l_mean = sum([float(l) for l in nsp_ls]) / len(nsp_ls)
            metric.add(mlm_l_mean, nsp_l_mean, batch[0].shape[0], 1)
            timer.stop()
            animator.add(step + 1,
                         (metric[0] / metric[3], metric[1] / metric[3]))
            step += 1
            if step == num_steps:
                num_steps_reached = True
                break

    print(f'MLM loss {metric[0] / metric[3]:.3f}, '
          f'NSP loss {metric[1] / metric[3]:.3f}')
    print(f'{metric[2] / timer.sum():.1f} sentence pairs/sec on '
          f'{str(devices)}')
```

```{.python .input}
#@tab pytorch
def train_bert(train_iter, net, loss, vocab_size, devices, num_steps):
    net = nn.DataParallel(net, device_ids=devices).to(devices[0])
    trainer = torch.optim.Adam(net.parameters(), lr=1e-3)
    step, timer = 0, d2l.Timer()
    animator = d2l.Animator(xlabel='step', ylabel='loss',
                            xlim=[1, num_steps], legend=['mlm', 'nsp'])
    # Sum of masked language modeling losses, sum of next sentence prediction
    # losses, no. of sentence pairs, count
    metric = d2l.Accumulator(4)
    num_steps_reached = False
    while step < num_steps and not num_steps_reached:
        for tokens_X, segments_X, valid_lens_x, pred_positions_X,\
            mlm_weights_X, mlm_Y, nsp_y in train_iter:
            tokens_X = tokens_X.to(devices[0])
            segments_X = segments_X.to(devices[0])
            valid_lens_x = valid_lens_x.to(devices[0])
            pred_positions_X = pred_positions_X.to(devices[0])
            mlm_weights_X = mlm_weights_X.to(devices[0])
            mlm_Y, nsp_y = mlm_Y.to(devices[0]), nsp_y.to(devices[0])
            trainer.zero_grad()
            timer.start()
            mlm_l, nsp_l, l = _get_batch_loss_bert(
                net, loss, vocab_size, tokens_X, segments_X, valid_lens_x,
                pred_positions_X, mlm_weights_X, mlm_Y, nsp_y)
            l.backward()
            trainer.step()
            metric.add(mlm_l, nsp_l, tokens_X.shape[0], 1)
            timer.stop()
            animator.add(step + 1,
                         (metric[0] / metric[3], metric[1] / metric[3]))
            step += 1
            if step == num_steps:
                num_steps_reached = True
                break

    print(f'MLM loss {metric[0] / metric[3]:.3f}, '
          f'NSP loss {metric[1] / metric[3]:.3f}')
    print(f'{metric[2] / timer.sum():.1f} sentence pairs/sec on '
          f'{str(devices)}')
```

We can plot both the masked language modeling loss and the next sentence prediction loss
during BERT pretraining.

```{.python .input}
#@tab all
train_bert(train_iter, net, loss, len(vocab), devices, 50)
```

## Representing Text with BERT

After pretraining BERT,
we can use it to represent single text, text pairs, or any token in them.
The following function returns the BERT (`net`) representations for all tokens
in `tokens_a` and `tokens_b`.

```{.python .input}
def get_bert_encoding(net, tokens_a, tokens_b=None):
    tokens, segments = d2l.get_tokens_and_segments(tokens_a, tokens_b)
    token_ids = np.expand_dims(np.array(vocab[tokens], ctx=devices[0]),
                               axis=0)
    segments = np.expand_dims(np.array(segments, ctx=devices[0]), axis=0)
    valid_len = np.expand_dims(np.array(len(tokens), ctx=devices[0]), axis=0)
    encoded_X, _, _ = net(token_ids, segments, valid_len)
    return encoded_X
```

```{.python .input}
#@tab pytorch
def get_bert_encoding(net, tokens_a, tokens_b=None):
    tokens, segments = d2l.get_tokens_and_segments(tokens_a, tokens_b)
    token_ids = torch.tensor(vocab[tokens], device=devices[0]).unsqueeze(0)
    segments = torch.tensor(segments, device=devices[0]).unsqueeze(0)
    valid_len = torch.tensor(len(tokens), device=devices[0]).unsqueeze(0)
    encoded_X, _, _ = net(token_ids, segments, valid_len)
    return encoded_X
```

Consider the sentence "a crane is flying".
Recall the input representation of BERT as discussed in :numref:`subsec_bert_input_rep`.
After inserting special tokens “<cls>” (used for classification)
and “<sep>” (used for separation),
the BERT input sequence has a length of six.
Since zero is the index of the “<cls>” token,
`encoded_text[:, 0, :]` is the BERT representation of the entire input sentence.
To evaluate the polysemy token "crane",
we also print out the first three elements of the BERT representation of the token.

```{.python .input}
#@tab all
tokens_a = ['a', 'crane', 'is', 'flying']
encoded_text = get_bert_encoding(net, tokens_a)
# Tokens: '<cls>', 'a', 'crane', 'is', 'flying', '<sep>'
encoded_text_cls = encoded_text[:, 0, :]
encoded_text_crane = encoded_text[:, 2, :]
encoded_text.shape, encoded_text_cls.shape, encoded_text_crane[0][:3]
```

Now consider a sentence pair
"a crane driver came" and "he just left".
Similarly, `encoded_pair[:, 0, :]` is the encoded result of the entire sentence pair from the pretrained BERT.
Note that the first three elements of the polysemy token "crane" are different from those when the context is different.
This supports that BERT representations are context-sensitive.

```{.python .input}
#@tab all
tokens_a, tokens_b = ['a', 'crane', 'driver', 'came'], ['he', 'just', 'left']
encoded_pair = get_bert_encoding(net, tokens_a, tokens_b)
# Tokens: '<cls>', 'a', 'crane', 'driver', 'came', '<sep>', 'he', 'just',
# 'left', '<sep>'
encoded_pair_cls = encoded_pair[:, 0, :]
encoded_pair_crane = encoded_pair[:, 2, :]
encoded_pair.shape, encoded_pair_cls.shape, encoded_pair_crane[0][:3]
```

In :numref:`chap_nlp_app`, we will fine-tune a pretrained BERT model
for downstream natural language processing applications.

## Summary

* The original BERT has two versions, where the base model has 110 million parameters and the large model has 340 million parameters.
* After pretraining BERT, we can use it to represent single text, text pairs, or any token in them.
* In the experiment, the same token has different BERT representation when their contexts are different. This supports that BERT representations are context-sensitive.

## Exercises

1. In the experiment, we can see that the masked language modeling loss is significantly higher than the next sentence prediction loss. Why?
2. Set the maximum length of a BERT input sequence to be 512 (same as the original BERT model). Use the configurations of the original BERT model such as $\text{BERT}_{\text{LARGE}}$. Do you encounter any error when running this section? Why?

:begin_tab:`mxnet`
[Discussions](https://discuss.d2l.ai/t/390)
:end_tab:

:begin_tab:`pytorch`
[Discussions](https://discuss.d2l.ai/t/1497)
:end_tab:


text:
---
id: 71
title: Java Memory Management Improvement Proposal
date: 2008-10-30T07:25:45+00:00
author: Rob Di Marco
layout: post
guid: http://www.innovationontherun.com/?p=71
permalink: /java-memory-management-improvement-proposal/
categories:
  - Java
  - Software Development
tags:
  - java memory
---
For the umpteenth time, I had to deal with the dreaded Java [OutOfMemoryError](http://java.sun.com/javase/6/docs/api/java/lang/OutOfMemoryError.html), this time when trying to run [Fisheye](https://www.atlassian.com/software/fisheye/) Subversion browser.  My problem deals with a [known issue](http://jira.atlassian.com/browse/FE-624) in Fisheye where a background indexing task sometimes runs causes an OOM error.  Of course, an OOM does not just effect the indexing task, it can also impact other operations unrelated to the task with the memory leak.  While diagnosing this problem, it got me thinking about how this could be better managed.

## The Problem

 The JVM defines a single heap for all objects created across all threads in the java process.  A consequence of this is that if one processing thread has a memory leak, any thread on the JVM may start suffering from OOM errors.  However, not all threads are equally important to my application.  I may not mind too much if a batch processing thread fails with an OOM but I may care very much if the lack of memory causes all of my Tomcat threads to no longer be able to process web requests.

## Proposed Solution

What I would like is to be able to segment my heap so that I can dedicate portions of the heap (either by percentage of the total heap or by absolute number of bytes) to a specific set of work.  That way if an OOM occurs, I can contain its impact to a certain set of threads while other threads continue processing.  I would see this being configured as JVM runtime -X arguments, something like -XHeapSegment:Name=MyMemHeap,Size=25%,Thread=<ThreadNameRegex>

I could see the partitioning being relative (by percentage) or absolute (total bytes allocated).  I could also see linking it to ThreadGroups as opposed to just Threads.

## Some Questions

  * Do others see value in this proposal?
  * I am not an expert at JVM internals.  Is there a fundamental reason this would not work?
  * Any other suggestions?
  * Should we allow this segmentation to be defined at compile-time as well through annotations?

## Notes

  * For the purposes of the problem I defined, I do not have a requirement that the young generations also be segmented.  However, if it is impractical to keep a single set of young generations and segmented older generations, I can deal with having segmentation in the young generations as well.
  * Obviously, this would in no way address all OutOfMemoryErrors.  We can still look forward to running out of PermGenSpace

branch_name: refs/heads/master

extension: md

license_type: permissive


text:
---
layout: default
---

# 1996-11-13 - Re: Information [for new PGP user]

## Header Data

From: snow<EMAIL_ADDRESS>To<EMAIL_ADDRESS>(Dale Thorn)

Message Hash: 5fccad943d61f784951fd211ce6cad7dfe6c4c284d9102160267c86a019cb2f6

Message ID<EMAIL_ADDRESS>Reply To<EMAIL_ADDRESS>UTC Datetime: 1996-11-13 18:24:13 UTC

Raw Date: Wed, 13 Nov 1996 10:24:13 -0800 (PST)

## Raw message

```
{% raw  %}From: snow<EMAIL_ADDRESS>Date: Wed, 13 Nov 1996 10:24:13 -0800 (PST)
To<EMAIL_ADDRESS>(Dale Thorn)
Subject: Re: Information [for new PGP user]
In-Reply-To<EMAIL_ADDRESS>Message-ID<EMAIL_ADDRESS>MIME-Version: 1.0
Content-Type: text/plain

> Paul Bradley wrote:
> > > >I'll bet you can't find 10 out of 1,000 users who have read the total source,
> > > may not function as expected.
> > compile on my system with the borland makefile than comes with PGP.
>Yet another success (NOT!) story for PGP.  I wonder how many people on this list
>would be willing to bet something *really* important to them on the security of PGP?

     I'd have more trouble trusting the Loose Nut on the other end than the
software.

     Security is more than the software, it is picking proper Pass Phrases
(which I am not too sure about), it is keeping the keys where it is less
than easy to get at, it is making sure the machine isn't compromised etc.

     Also, I ask you once again, Could you please format your email to
under 80 columns? You might have something valuable to say, but I can't
stand to read any of your posts more than 2 or 3 lines because of the
way the lines break.

     Thank you.

Petro, Christopher C.
<EMAIL_ADDRESS><prefered for any non-list stuff>
<EMAIL_ADDRESS>

{% endraw %}
```

## Thread

+ Return to [November 1996](/archive/1996/11)

+ Return to "[Dale Thorn <dthorn@gte.net>](/authors/dale_thorn_dthorn_at_gte_net_)"
+ Return to "[Paul Bradley <paul@fatmans.demon.co.uk>](/authors/paul_bradley_paul_at_fatmans_demon_co_uk_)"
+ Return to "[snow <snow@smoke.suba.com>](/authors/snow_snow_at_smoke_suba_com_)"
+ Return to<EMAIL_ADDRESS><William.H.Geiger.III@mailhub.amaranth.com>](/authors/whgiii_at_amaranth_com_william_h_geiger_iii_at_mailhub_amaranth_com_)"
+ Return to "["William H. Geiger III" <whgiii@amaranth.com>](/authors/william_h_geiger_iii_whgiii_at_amaranth_com_)"

+ 1996-11-09 (Sat, 9 Nov 1996 01:31:44 -0800 (PST)) - [Re: Information [for new PGP user]](/archive/1996/11/1778c0e4e0857bbfd69d12cb1cb57c8375e77c4bffdb16d1375c10b3a901f90b) - _Paul Bradley<EMAIL_ADDRESS>  + 1996-11-09 (Sat, 9 Nov 1996 10:04:11 -0800 (PST)) - [Re: Information [for new PGP user]](/archive/1996/11/6c362d5832c4bdd0c2133ac14e5ce5c026d9497ebfd05c321e1d8b6fab4e3100) - _Dale Thorn<EMAIL_ADDRESS>    + 1996-11-09 (Sat, 9 Nov 1996 14:42:37 -0800 (PST)) - [Re: Information [for new PGP user]](/archive/1996/11/7ceac7a367c57fa7c4e69eb9b9bf603d37bc40dfc91046022077b3784e269bab) -<EMAIL_ADDRESS><EMAIL_ADDRESS>      + 1996-11-10 (Sat, 9 Nov 1996 18:52:05 -0800 (PST)) - [Re: Information [for new PGP user]](/archive/1996/11/4cba0500d232732461d1cd58bab11d65356627ed30d71b535ec51f9d7b35c938) - _Dale Thorn<EMAIL_ADDRESS>        + 1996-11-10 (Sun, 10 Nov 1996 09:16:16 -0800 (PST)) - [Apology to Dale Thorn](/archive/1996/11/eb7e93515cf949525131fdf6572ef4ef869d3d1d7e0d2eb05419052200a0edb6) - _"William H. Geiger III"<EMAIL_ADDRESS>          + 1996-11-10 (Sun, 10 Nov 1996 09:49:31 -0800 (PST)) - [Re: Apology to Dale Thorn](/archive/1996/11/61c30ff6afb44414d6b7b2a76ad5ed795f189fb69e59d66760c9689fe77f3ae2) - _Dale Thorn<EMAIL_ADDRESS>    + 1996-11-13 (Wed, 13 Nov 1996 10:24:13 -0800 (PST)) - Re: Information [for new PGP user] - _snow<EMAIL_ADDRESS>

license: MIT

gha_license_id: MIT


text:
---
layout: post
title: Camera Theremin
permalink: /posts/CameraTheremin
---

This post details how I made the [CameraTheremin](https://aneeshdurg.github.io/CameraTheremin/). I hope you find it useful.

## Table of Contents

* [The motivation](#the-motivation)
* [Getting webcam input with getUserMedia](#getting-webcam-input-with-getusermedia)
* [Image processing using canvas](#image-processing-using-canvas)
  * [Proximity detection Method 1: Thresholding](#proximity-detection-method-1)
  * [Proximity detection Method 2: Binary background subtraction](#proximity-detection-method-2)
* [Audio generation](#audio-generation)
* [Conclusion](#conclusion)

## The motivation

I've always thought that one of the coolest instruments in the world is the Theremin. What other instrument can you play without touching it? However, Theremins cost around $300 and as a non-musician who just wants to try it out for the sake of novelty, that's a bit pricey.
That being said, I still wanted to try one and decided to recreate the experience in my browser using JavaScript and getUserMedia+Tone.js (my initial approach was to use python and openCV+pyo, but I later abandoned that for a variety of reasons, most notably the fact that it required a user to install python and all the required packages, as opposed to opening a URL).

So, what do we really need to recreate a theremin-like experience? We need:

* Something to control the pitch
* Something to control the volume.

I felt like proximity to the camera would be a rather novel and fun experience to play with, so I decided to use that to control the pitch, while volume would be controlled by the horizontal movement.

## Getting webcam input with getUserMedia

Getting webcam input with getUserMedia is a snap. All it took was the following lines:

```javascript
var video = document.getElementById("vid");
var constraints = {
    video: {
        mandatory: {
	        maxWidth: 320,
            maxHeight: 180
        }
    }
}
navigator.getUserMedia(constraints, startStream, function(){});
function startStream(stream){
    video.src = URL.createObjectURL(stream);
    video.play();
}
```

The call to getUserMedia takes 3 arguments, the constraints, a success callback, and a error callback.

You can see here that I've not actually defined a error callback, but instead left an empty function in its place.

On successfully obtaining the webcam stream, this is sent to the startStream function which puts my webcam stream into an HTML video object.

Now it's time to start processing the obtained frames.

## Image processing using canvas

In order to modify and edit the image data, I had to first draw the obtained frame from the video to a canvas element. To do this, I used added the line `requestAnimationFrame(draw)` at the end of my `startStream` function which tells the browser to call the function `draw` before refreshing the display.

The function `draw` does the following:

1. A function which converts the video to canvas data
2. Function calls which handle the image processing
3. Put the image data back into the canvas
4. call `requestAnimationFrame(draw)` so that the next frame can be processed.

To accomplish (1), I have the following function:

```javascript
function readFrame(){
    try{
        context.save();
        context.scale(-1, 1);
        context.drawImage(video, -width, 0, width, height);
        context.restore();
    } catch(e){
        console.log(e);
        return null;
    }
    return context.getImageData(0, 0, width, height);
}
```
By using `context.scale(-1, 1)`, I am able to mirror the video input so that the displayed feed seems more natural to the user. This returns an array of the image data. The return data is a 1-d array of RGBA values and thus to access the pixel at coordinates (x, y), I must access the following indices:

```javascript
    var redIndex = 4*(y*width+x);
    var greenIndex = redIndex+1;
    var blueIndex = redIndex+2;
    var alphaIndex = redIndex+3;
```
Conversely, given an index which satisfies the property `index%4 == 0` (i.e. the index is the starting of a pixel) I can obtain the (x, y) coordinates as follows:

```javascript
    var x = index%width;
    var y = index/width;
```

Moving onto (2), in order to detect proximity, I first separated the foreground from the background and then used the area of the foreground as a measure of distance. This works because as objects are moved closer to or further from the camera, they take more and less area respectively as shown below with my finger as the object:

### Closer

![closer to the camera]({{ '/static/images/2016-5-22-Camera-Theremin/closer.png' | relative_url }})

### Further

![further from the camera]({{ '/static/images/2016-5-22-Camera-Theremin/further.png' | relative_url }})

In order to separate the foreground from the background, I created two methods, which you can switch between, depending on which is more suitable for you lighting and surrounding conditions.

In both methods, the obtained frame was first converted to gray scale:

```javascript
function bgr2gray(data){
    for(var i = ys; i<ye; i++){
        for(var j = xs; j<xe; j++){
	    var k = 4*(i*width+j);
	    var lumin = 0.21*data[k]+0.72*data[k+1]+0.07*data[k+2];
	    data[k] = lumin;
	    data[k+1] = lumin;
	    data[k+2] = lumin;
        }
    }
}
```
then blurred to improve accuracy by convolving a hardcoded blur kernel with the image data.

You may notice the fact that I've used the limits ys, ye, xs, xe in my for loops as opposed to 0, height, 0, width. This is due to my crop function, which I'll talk more about later. As the name implies, it crops the image to specified boundaries, only processing selected parts of the image.

### Proximity detection Method 1

#### Thresholding

This method is rather simple and uses the following function:

```javascript
function threshold(data, thresh, a, b){
    for(var i = ys; i<ye; i++){
        for(var j = xs; j<xe; j++){
	     var k = i*4*width+4*j
             if(data[k]>thresh){
                 data[k] = a;
                 data[k+1] = a;
                 data[k+2] = a;
             }
             else{
                data[k] = b;
                data[k+1] = b;
                data[k+2] = b;
	        }
        }
    }
}
```

`threshold` takes 4 arguments, the image data, a threshold value, and two replacement values (representing color intensity) to create the binary image.

This function assumes that if the intensity of a certain point is greater than a certain threshold, it must be part of the background, and thus transforms the grayscale image into a binary image.

### Proximity detection Method 2

#### Binary background subtraction

This method requires me to first save an initial frame containing what I consider to be the background. Note that this will include regions which would otherwise be detected to be the foreground by the thresholder, but that is irrelevant to my usage.

Once I have my initial frame, I can subtract that from subsequent frames to obtain a binary image containing only the difference between the current frame and the initial frame. This is achieved with the following code:

```javascript
function binsub(data, thresh, a, b){
    var len = data.length;
    var xavg = 0;
    var counter = 0;
    for(var i = ys; i<ye; i++){
        for(var j = xs; j<xe; j++){
            var k = i*4*width+4*j
            var replace = true;
            if(Math.abs(data[k]-initialf[k])>thresh){
              replace = false;
            }
            if(replace){
                data[k] = a;
                data[k+1] = a;
                data[k+2] = a;
            }
            else{
                data[k] = b;
                data[k+1] = b;
                data[k+2] = b;
                xavg+=(k/4)%width;
                counter++;
            }
        }
    }
    xavg/=counter;
    if(counter<2000){
        xavg = 0;
    }
    document.getElementById("xavg").innerHTML = xavg;
    return xavg;
}
```
`binsub` takes 4 arguments, the image data, a threshold value, and two replacement values (representing color intensity) to create the binary image.

This function is very similar to `threshold` from Method 1, but instead of testing the value of a pixel itself, it tests the absolute value of the difference between the intensity at a point, and the intensity at the corresponding point in the initial frame (`initialf`).

Note that `binsub` also returns `xavg` which represents the average x-coordinate of all "foreground" pixels. This is useful to change the volume.

This is absent from Method 1, as the thresholeded image will contain other objects which skew the x-coordinate and doesn't allow for accurate control over volume.

Regardless of whether I used Method 1 or 2, I now need to extract the proximity from the binary image and use this to generate an audio output. To do this I ensured that the replacement values `a, b` in both `binsub` and `threshold` are set to 0, 255 respectively. This sets foreground pixels to white and background pixels to black. I then made a function called `countWhite` which is as follows:

```javascript
function countWhite(data){
    var ret = 0;
    for(var i = ys; i<ye; i++){
        for(var j = xs; j<xe; j++){
            var k = i*4*width+4*j
            if(data[k]==255){
                ret+=1;
            }
        }
    }
    return ret;
}
```

This counts the total area of all white pixels. The resulting value is then scaled between the max and min values (which are set during a calibration sequence, more details can be found in the readme of my [github repository](https://github.com/aneeshdurg/CameraTheremin/)) to obtain a value to send to Tone.js as shown below:

```javascript
scaled = scale(white, cont, xpos);
//sends frequency to synth
if(min<0||max<0||white<0)
    //sends a volume of -100 to indicate an error
    scaled = playSynth(0, -100);
else
    scaled = playSynth(scaled, xpos);
--------------------------------------------------
function scale(num, mode, vol){
    if(min<0||max<0||num<0){
        return num;
    }
    var high = max-min;
    var sf = 220/high;
    var ret = (sf*(num-min)+220)*1000;
    var ret = Math.floor(ret);
    var ret = ret/1000;
    if(mode){
        if(ret<=150)
            ret=0;
        else if(ret<=233.082)
            ret=220;
        else if(ret<=(246.942+261.626)/2)
            ret = 246.942;
        else if(ret<=277.183)
            ret = 261.626;
        else if(ret<=311.127)
            ret = 293.665;
        else if(ret<=(349.228+329.628)/2)
            ret = 329.628;
        else if(ret<=369.994)
            ret = 349.228;
        else if(ret<=415.305)
            ret = 391.995;
        else
            ret = 440;
    }
    return ret;
}
```
Now that my detection is done, it's time to move on to (3), restoring the canvas data. Of course, this can be simply done with `context.putImageData(frame, 0, 0)`, but I also wanted to add some form of visual indicator so that the user knows what note is being played. I decided to do this by adding a certain color to the original frame. To do this, I added the following two lines before the call to `putImageData`:

```javascript
frame = readFrame();
increasedColor(frame.data, scaled);
```
Since `requestAnimationFrame` has not been called in between my initial call to `readFrame` and my call to `readFrame` here, the same frame will be returned, and I can edit it with `increasedColor` before drawing it.

```javascript
function increasedColor(data, val){
    var temp = Math.abs(val-220);
        temp/=11;
    temp+=7;
    var f = 0.3;
    r = Math.sin(f*temp+0)*127+128;
    g = Math.sin(f*temp+2)*127+128;
    b = Math.sin(f*temp+4)*127+128;

    var len = data.length;
    if(temp!=0)
        bgr2gray(data);
    for(var i =0; i<len; i+=4){
        data[i] += r;
        data[i+1] += g;
        data[i+2] += b;
        if(data[i]>255)
            data[i]=255;
        if(data[i+1]>255)
            data[i+1] = 255;
        if(data[i+2]>255)
            data[i+2] = 255;
        if(playBack&&((i/4)%width>xs&&(i/4)%width<xe)){
            data[i]   = 255-data[i];
            data[i+1] = 255-data[i+1];
            data[i+2] = 255-data[i+2];
        }
    }
}
```

## Audio generation

Now that the detection is done, I used Tone.js to play out notes of a certain frequency. This was done with the following function:

```javascript
function playSynth(freq, vol){
    if(vol==-100){
        return;
    }
    if(!playing){
        synth.triggerAttack(oldfreq);
        playing = true;
    }
    if(!playBack){
        document.getElementById("freq").innerHTML = "Frequency: "+freq+" Hz";
        if(freq>0&&freq!=oldfreq){
            if(!cont)
                synth.frequency.rampTo(freq, 0.05);
            else
                synth.setNote(freq);
        }
        if(dobinsub&&volEnable){
            vol -= 80;
            vol/=12;
            if (vol<0){
                vol = 0;
            }
            if(vol>15)
                vol = 15;
                synth.volume.rampTo(vol, 0.07);
                vol = Math.floor(vol*100)/100;
                document.getElementById("vol").innerHTML = 'Volume: '+vol+' db';
            }
        }
        else{
            vol = 10;
            synth.volume.value = 10;
            document.getElementById("vol").innerHTML = 'Volume: 10 db';
        }
        if(record){
            recording.push([freq, vol]);
        }
    }
    else{
        if(tracking>=recording.length){
            startPlayBack();
            return;
        }
        var stepSize = width/recording.length;
        xs = Math.floor(tracking*stepSize);
        freq = recording[tracking][0];
        vol = recording[tracking][1];
        synth.frequency.rampTo(freq, 0.05);
        synth.volume.rampTo(vol, 0.07);
        document.getElementById("freq").innerHTML = "Frequency: "+freq+" Hz";
        document.getElementById("vol").innerHTML = "Volume: "+vol+" db";
        tracking++;
    }
    oldfreq = freq;
    return freq;
}
```

Most of this code is fairly self-explanatory and you can also see the recording/playback functionality as well. One interesting thing is that play back changes the cropped regions as it uses those values to display a progress animation while it plays. The original crop bounds are restored at the end of playback.

## Conclusion

This was a fairly fun project to work on, though I'd say it's still not done. I want to improve certain things such as the speed of the detection, the types of sounds that can be generated
(e.g. make the Theremin sound like another instrument, or have it play chords), and the ability to change volume in both methods among others.

There are also some known issues such as the occasional crash due to overuse of memory in firefox (doesn't happen on chrome) which need to be worked out.

You can find the code for the project [here](https://github.com/aneeshdurg/CameraTheremin/) and try it out [here](https://aneeshdurg.me/CameraTheremin). Feel free to fork the repository and play around with the code!


text:
---
layout: post
title: Two Ways of Creating Object Literals in Python
---
First, before diving into Python, let's talk about JavaScript. Here's a refresher of JavaScript's object model from the [MDN](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Working_with_Objects).

> JavaScript is designed on a simple object-based paradigm. An object is a collection of properties, and a property is an association between a name (or key) and a value.

Because of this simplicity, JavaScript allows for object literals. You don't need a constructor to define an object! This is especially useful for one-off objects.

{% highlight javascript %}
    let car = {
      model: "Skyline",
      manufacturer: "Nissan",
      year: "1996",
      race: function() {
        console.log("Vroom vroom!");
      }
    };
{% endhighlight %}

Here's two approaches to doing the same in Python.

First, from [Programming Ideas with Jake](https://programmingideaswithjake.wordpress.com/2016/05/07/object-literals-in-python/) we can create an Object class and give it a constructor that puts every attribute-value pair into \_\_dict\_\_. Recall that \_\_dict\_\_ is "a dictionary or other mapping object used to store an object’s (writable) attributes."

{%highlight python %}
    class Object:
       def __init__(self, **attributes):
          self.__dict__.update(attributes)

    car = Object(
        model = "Skyline",
        manufacturer = "Nissan",
        year = "1996",
        race = lambda: print("Vroom vroom!")
    )
{% endhighlight %}

So far so good right? There's a few issues that I see. What if *race* were a much more complex function? I would like to have an inline function. Python's lambdas have [caveats](https://www.artima.com/weblogs/viewpost.jsp?thread=147358); things like multi-line lambdas are not allowed. So, instead, we can use class variables and class methods.

{% highlight python %}
    class Car:
        model = "Skyline"
        manufacturer = "Nissan"
        year = "1996"

        @classmethod
        def race(cls):
            print("Vroom vroom!")
            print("Skrrrrt!")
{% endhighlight %}

We can access and use attributes as we would a typical Python object.

{% highlight python %}
      Car.model = "Mustang"
      Car.manufacturer = "Ford"
      Car.race()
{% endhighlight %}

Why is this possible? Because, in Python, classes themselves are objects!

gha_language: SCSS


text:
*v0.5.3 Release Notes:*
- *Fixed a "bug" in the `TokenMatcher`. Spaczz expects token matches returned in order of ascending match start, then descending match length. However, spaCy's `Matcher` does not return matches in this order by default. Added a sort in the `TokenMatcher` to ensure this.*

*v0.5.2 Release Notes:*
- *Minor updates to pre-commits and noxfile.*

*v0.5.1 Release Notes:*
- *Minor updates to allowed dependency versions and CI.*
- *Switched back to using typing types instead of generic types because spaCy v3 uses Pydantic and Pydantic does not support generic types in Python < 3.9. I don't know if this would actually cause any issues but I am playing it safe. Potentially more changes for spaczz to play nicely with Pydantic to follow.*

*v0.5.0 Release Notes:*
- *Support for spaCy v3.*
- *If using spaCy v3, the `SpaczzRuler` optional arguments no longer need to be prepended with `"spaczz_"`. This will still work in most cases offering some backwards compatibility. However, optional arguments prepended with `"spaczz_"` will not work with spaCy v3's new `spacy.load` and `nlp.add_pipe` config driven APIs. It is therefore recommended that users move away from using the prepended versions if using spaCy v3. It should be noted however that the prepended arguments are still necessary if using spaczz with spaCy v2.*
- *`Matcher.pipe` methods are now deprecated in accordance with spaCy v3.*
- *`spaczz_span` custom attribute is deprecated in favor of `spaczz_ent`. They both have the same functionality but the `spaczz_ent` name makes more sense.*

*v0.4.2 Release Notes:*
- *Fixed a bug where `TokenMatcher` callbacks did nothing.*
- *Fixed a bug where `spaczz_token_defaults` in the `SpaczzRuler` did nothing.*
- *Fixed a bug where defaults would not be added to their respective matchers when loading from bytes/disk in the `SpaczzRuler`.*
- *Fixed some inconsistencies in the `SpaczzRuler` which will be particularly noticeable with ent_ids. See the "Known Issues" section below for more details.*
- *Small tweaks to spaczz custom attributes.*
- *Available fuzzy matching functions have changed in RapidFuzz and have changed in spaczz accordingly.*
- *Preparing for spaCy v3 updates.*

*v0.4.1 Release Notes:*
- *Spaczz's phrase searching algorithm has been further optimized so both the `FuzzyMatcher` and `SimilarityMatcher` should run considerably faster.*
- *The `FuzzyMatcher` and `SimilarityMatcher` now include a `thresh` parameter that defaults to `100`. When matching, if `flex > 0` and the match ratio is >= `thresh` during the initial scan of the document, no optimization will be attempted. By default perfect matches don't need to be run through match optimization.*
- *PEP585 code updates.*

*v0.4.0 Release Notes:*
- *Spaczz now includes a `TokenMatcher` that provides token pattern support like spaCy's `Matcher`. It provides all the same functionality as spaCy's `Matcher` but adds fuzzy and fuzzy-regex support. However, it adds additional overhead to it's spaCy counterpart so it should only be used as needed for fuzzy matching purposes.*
- *Spaczz's custom attributes have been reworked and now initialize within spaczz's root `__init__`. These are set via spaczz pipeline components (currently just the `SpaczzRuler`) The only downside is that I had to remove the `attr` parameter from the `SpaczzRuler` to enable this.*
- *The `flex` parameter available to fuzzy and similarity phrase matching now accepts the strings `"max"`: `len(pattern)` and `"min"`: `0`.*
- *The `flex` parameter now defaults to `max(len(pattern) - 1, 0)` instead of `len(query)` as this generally makes more sense. Single-token patterns shouldn't have their boundaries extended during optimization by default.*
- *`min_r1` for the fuzzy phrase matcher is now `50`, this is still low but not so low that it filters almost nothing out in the initial document scan.*
- *Bug fixes to phrase searching that could cause index errors in spaCy `Span` objects.*

*v0.3.1 Release Notes:*
- *spaczz now includes an experimental `SimilarityMatcher` that attempts to match search terms based on vector similarity. It requires a a spaCy model with word vectors (e.x. spaCy's medium and large English models) to function properly. See the documentation below for usage details.*

*v0.3.0 Release Notes:*
- *The `FuzzyMatcher` and `RegexMatcher` now return fuzzy ratio and fuzzy count details respectively. The behavior of these two matchers is still the same except they now return lists of tuples of length 4 (match id, start, end, fuzzy details).*
    - *This change could be breaking in instances where these tuples are unpacked in the traditional spaCy fashion (match id, start, end). Simply include the fuzzy details or a placeholder during unpacking to fix.*
- *The SpaczzRuler now writes fuzzy ratio and fuzzy count details for fuzzy/regex matches respectively as custom `Span` attributes. These are `spaczz_ent_ratio` and `spaczz_ent_counts` respectively. They return `None` by default.*
    - *The `spaczz_ent` portion of these attributes is controlled by the `attr` parameter and can be changed if needed. However, the `_ent_ratio` and `_ent_counts` extensions are hard-coded.*
    - *If, in the rare case, the same match is made via a fuzzy pattern and regex pattern, the span will have both extensions set with their repsective values.*
- *Fixed a bug where the `attr` parameter in the `SpaczzRuler` did not actually change the name of the custom span attribute.*

*v0.2.0 Release Notes:*
- *Fuzzy matching is now performed with [RapidFuzz](https://github.com/maxbachmann/rapidfuzz) instead of [FuzzyWuzzy](https://github.com/seatgeek/fuzzywuzzy).*
    - *RapidFuzz is higher performance with a more liberal license.*
- *The spaczz ruler now automatically sets a custom, boolean, `Span` attribute on all entities it adds.*
    - *This is set by the `attr` parameter during `SpaczzRuler` instantiation and defaults to: "spaczz_ent".*
    - *For example: an entity set by the spaczz ruler will have `ent._.spaczz_ent` set to `True`.*
- *Spaczz ruler patterns now support optional "id" values like spaCy's entity ruler. See [this spaCy documentation](https://spacy.io/usage/rule-based-matching#entityruler-ent-ids) for usage details.*
- *Automated Windows testing is now part of the build process.*


text:
---
layout: post
title: "Rust Basics (2)"
date: 2021-02-21
desc: "Function and control flow"
keywords: "rust,easy"
categories: [Rust]
tags: [Rust]
icon: fa-rust
---

The goal of this post is to learn how to **define a function** and **control instructions flow**.

## Function

A good example of a simple function is in the `hello_world` project described in the [previous post](https://guillaumefavelier.github.io/rust/2021/02/13/rust_basics_1.html):

```rust
fn main() {
    println!("Hello, world!");
}
```

Notice the `fn` keyword to start the definition of the function. But in this example,
`main()` does not accept arguments nor return anything.

Maybe a more interesting example would be a function that generates a random integer
in a range:

```rust
fn get_secret_number(nmin: i32, nmax: i32) -> i32 {
    use rand::Rng;
    // gen_range(low, high) is inclusive of low but exclusive of high
    return rand::thread_rng().gen_range(nmin, nmax + 1);
}
```
> `rand` is an external package so the `[dependencies]` section of the `Cargo.toml` configuration file needs
> to be updated like so:
> ```
> ...
> [dependencies]
> rand = "0.6.0"
> ```

This function accepts two arguments called `nmin` and `nmax` of type `i32` (signed integer that takes 32 bit of space)
and its return type is specified after the arrow `-> i32`.

In the above example, the result is returned explicitly with `return` but in Rust, the return value of
the function is synonymous with the value of the final expression. Meaning that `get_secret_number()`
can be modified as follows:

```rust
fn get_secret_number(nmin: i32, nmax: i32) -> i32 {
    use rand::Rng;
    rand::thread_rng().gen_range(nmin, nmax + 1)
}
```

Please note that with this syntax, the semicolon `;` is removed to prevent the expression to become a
**statement** (which *does not* return a value).

Here is a complete example:

```rust
fn get_secret_number(nmin: i32, nmax: i32) -> i32 {
    use rand::Rng;
    rand::thread_rng().gen_range(nmin, nmax + 1)
}

fn main() {
    let secret_number = get_secret_number(1, 100); // function call here!
    println!("The secret number is {}.", secret_number);
}
```

---

## Control Flow

In Rust, *branching* is mainly done with the keyword `if`, the block of code that follows
is executed only if the described condition is evaluated to `true`:

```rust
fn strip_line(line: &mut String) {
    // enter the code block only if the last character of the String is '\n'
    if line.ends_with("\n") {
        line.pop(); // remove this last character '\n'
        // enter the code block only if the last character of the String is '\r'
        if line.ends_with("\r") {
          line.pop(); // and remove this last character '\r'
        }
    }
}
```

Multiple instruction branches can be chained together with the keywords `else if` and `else`:

```rust
fn main() {
    let secret_number = get_secret_number(1, 100);
    let guess = 50;
    if guess < secret_number {
        println!("The secret number is greater.");
    }
    else if guess > secret_number {
        println!("The secret number is lower.");
    }
    else {
        println!("Congratulations! The secret number {} is found.", guess);
    }
}
```

A particular Rust mechanic can simplify the code above. It's called **pattern matching**
and it is based on the **match** keyword. Please note that each matching arm is evaluated
and *all possible values must be covered*:

```rust
fn main() {
    let secret_number = get_secret_number(1, 100);
    let guess = 50;
    // the cmp() function returns an Ordering enumeration
    match guess.cmp(&secret_number) {
        std::cmp::Ordering::Less => println!("The secret number is greater."),
        std::cmp::Ordering::Greater => println!("The secret number is lower."),
        std::cmp::Ordering::Equal => println!("Congratulations! The secret number {} is found.", guess),
    };
}
```
> An `enum`, in its simplest form, holds a set of mutually exclusive values:
> ```rust
> enum Ordering {
>     Less,
>     Greater,
>     Equal,
> }
> ```

In Rust, it's possible to *repeat instructions* by using the keywords `for`, `while` or `loop`.

The `for`-loop is very useful to iterate through a `Range` of values:

```rust
fn main() {
    println!("Start countdown...");
    // rev() reverse the given range
    for number in (1..10).rev() {
        println!("{}...", number);
    }
    println("Let's go!");
}
```

Or through the items of a `Collection`:

```rust
fn main() {
    let a = [10, 20, 30, 40, 50];
    for (index, element) in a.iter().enumerate() {
        println!("The item at {} is {}.", index, element);
    }
}
```

The `while`-loop repeats the block of code as long as the specified condition holds `true`:

```rust
fn main() {
    let mut n: u32 = 10;
    while n > 0 {
      n -= 1;
    }
    println!("Done!");
}
```

And finally, there is the `loop` keyword to create an infinite loop:

```rust
fn is_ready() -> bool {
    use rand::Rng;
    return rand::thread_rng().gen::<bool>()
}

fn main() {
    loop {
      println!("Work in progress...");
      if is_ready() {
        break; // leave the loop when the condition is met
      }
    }
}
```

---

## Reference

Material for this post is available on [GitHub](https://github.com/GuillaumeFavelier/blog_rust_basics_2).

More infos on the [Rust documentation](https://doc.rust-lang.org).

*Next post available soon.*


text:
# RepoFish

search github repos for python functions and generate standard data structures for them. Find functions in online places. Under Development!

### Install

      pip install repofish

### Generate Lib

      cd scripts
      python generate_database.py

repository: vsoch/repofish

gha_language: Python

repo_name: vsoch/repofish


text:
---
id: overview
title: Overview
sidebar_label: Overview
---

> **This section is still a work in progress.**

## Celerity Execution Timeline

The following animation illustrates the basics of how programs are executed
internally by the Celerity runtime system:

<video autoplay loop width="100%">
  <source src="assets/celerity_overview.mp4" type="video/mp4">

    Your browser doesn't support HTML5 video.
    Here is a link to the video instead.

</video>

Everything starts with the _Prepass_, which is similar to normal program
execution. However, whenever a relevant Celerity queue operation -- such as
`submit` -- is encountered, this pass does not
actually fully execute the related _kernel_ code (marked in color in the
video). Instead, this code, together with some dependency and scheduling
metainformation, is recorded in the Celerity **task graph**.

Concurrently with the Prepass, a _Scheduler_ thread constructs a more
detailed **command graph** from existing task graph nodes. This command graph
includes individual instructions for each node in the system, and also
encodes all necessary data transfers to maintain the consistent view of data
which would be expected if the program were to be executed on a single node.

On each _Worker_ node the command graph is asynchronously consumed and
executed, which finally leads to actually running the computations described
by each kernel in the input program, distributed among the GPUs in the
cluster.

In order to make the entire process more clear, the animation is simplified
in a few important aspects:

- The task graph shown is strictly sequential -- in practice independent
  tasks can exist at various points in the timeline.
- For this illustration, there are only two nodes in the system. The command
  graph would of course increase in size with additional nodes.
- While in the animation workers proceed seemingly sequentially over data send
  and receive operations, all of these are in fact processed asynchronously
  in the actual implementation.

Despite these simplifications, the animation captures the basic execution
principle of the Celerity runtime, and hopefully also makes the reasons for
some of the pitfalls described in the [Common Pitfalls](pitfalls.md) section
more clear.

branch_name: refs/heads/main


text:
# BVNDIT drop sultry red 'Dumb' MV!

Published at: **2019-11-05T09:28:00+00:00**

Author: **germainej**

Original: [allkpop](https://www.allkpop.com/video/2019/11/bvndit-drop-sultry-red-dumb-mv)

BVNDIT have dropped their sultry red music video for "Dumb"!"Dumb" is an upbeat Latin pop song about sending messages to someone who doesn't seem to get the signals, and it's the title track of BVNDIT's first mini album 'BE!'. The MV features the BVNDIT members against a red backdrop and sheer curtains.Watch BVNDIT's "Dumb" MV above, and let us know what you think in the comments below.

license: Unlicense

repository: dsadlollioygjmzvcxv/tdc-exported

repo_name: dsadlollioygjmzvcxv/tdc-exported


text:
# wi-nf

The `wi-nf` pipeline aligns, calls variants, and performs analysis from wild isolate sequence data. The output of the `wi-nf` pipeline can be uploaded to google storage as a new release for the CeNDR website.

[TOC]

## Usage

```

     ▄         ▄  ▄▄▄▄▄▄▄▄▄▄▄                         ▄▄        ▄  ▄▄▄▄▄▄▄▄▄▄▄
    ▐░▌       ▐░▌▐░░░░░░░░░░░▌                       ▐░░▌      ▐░▌▐░░░░░░░░░░░▌
    ▐░▌       ▐░▌ ▀▀▀▀█░█▀▀▀▀                        ▐░▌░▌     ▐░▌▐░█▀▀▀▀▀▀▀▀▀
    ▐░▌       ▐░▌     ▐░▌                            ▐░▌▐░▌    ▐░▌▐░▌
    ▐░▌   ▄   ▐░▌     ▐░▌           ▄▄▄▄▄▄▄▄▄▄▄      ▐░▌ ▐░▌   ▐░▌▐░█▄▄▄▄▄▄▄▄▄
    ▐░▌  ▐░▌  ▐░▌     ▐░▌          ▐░░░░░░░░░░░▌     ▐░▌  ▐░▌  ▐░▌▐░░░░░░░░░░░▌
    ▐░▌ ▐░▌░▌ ▐░▌     ▐░▌           ▀▀▀▀▀▀▀▀▀▀▀      ▐░▌   ▐░▌ ▐░▌▐░█▀▀▀▀▀▀▀▀▀
    ▐░▌▐░▌ ▐░▌▐░▌     ▐░▌                            ▐░▌    ▐░▌▐░▌▐░▌
    ▐░▌░▌   ▐░▐░▌ ▄▄▄▄█░█▄▄▄▄                        ▐░▌     ▐░▐░▌▐░▌
    ▐░░▌     ▐░░▌▐░░░░░░░░░░░▌                       ▐░▌      ▐░░▌▐░▌
     ▀▀       ▀▀  ▀▀▀▀▀▀▀▀▀▀▀                         ▀        ▀▀  ▀

    parameters              description                    Set/Default
    ==========              ===========                    =======

    --debug                 Set to 'true' to test          ${params.debug}
    --cores                 Regular job cores              ${params.cores}
    --out                   Directory to output results    ${params.out}
    --fqs                   fastq file (see help)          ${params.fqs}
    --fq_file_prefix        fastq prefix                   ${params.fq_file_prefix}
    --reference             Reference Genome               ${params.reference}
    --annotation_reference  SnpEff annotation              ${params.annotation_reference}
    --bamdir                Location for bams              ${params.bamdir}
    --tmpdir                A temporary directory          ${params.tmpdir}
    --email                 Email to be sent results       ${params.email}

    HELP: http://andersenlab.org/dry-guide/pipeline-wi/

```

# Pipeline Overview

![Pipeline-overview](img/wi-nf.png)

# Usage

## Docker File

[andersenlab/wi-nf](https://hub.docker.com/r/andersenlab/wi-nf/) is the docker image used within the wi-nf pipeline. If Quest ever supports singularity, it can be converted to a singularity image and used with Nextflow.

## pyenv environments

The pipeline uses two python environments due to clashing dependencies. The two environments are:

* `vcf-kit` - A python 2.7.14 environment with vcf-kit and cyvcf installed.
* `multiqc` - A python 3.6.0 environment with multiqc installed.

 If you are not using the docker container you must install these virtual environments by running the `setup_pyenv.sh` script:

```
bash setup_pyenv.sh
```

If you require one of these environments for a process within the pipeline, add the following as the first line of your script:

```
source init_pyenv.sh && pyenv activate <environment name>
```

## Profiles and Running the Pipeline

The `nextflow.config` file included with this pipeline contains three profiles. These set up the environment for testing local development, testing on Quest, and running the pipeline on Quest.

* `local` - Used for local development. Uses the docker container.
* `quest_debug` - Runs a small subset of available test data. Should complete within a couple of hours. For testing/diagnosing issues on Quest.
* `quest` - Runs the entire dataset.

### Running the pipeline locally

When running locally, the pipeline will run using the `andersenlab/wi-nf` docker image. You must have docker installed.

```
nextflow run main.nf -profile local -resume
```

### Debugging the pipeline on Quest

When running on Quest, you should first run the quest debug profile. The Quest debug profile will use a test dataset and sample sheet which runs much faster and will encounter errors much sooner should they need to be fixed. If the debug dataset runs to completion it is likely that the full dataset will as well.

```
nextflow run main.nf -profile quest_debug -resume
```

### Running the pipeline on Quest

The pipeline can be run on Quest using the following command:

```
nextflow run main.nf -profile quest -resume
```

# Configuration

Most configuration is handled using the `-profile` flag and `nextflow.config`; If you want to fine tune things you can use the options below.

## --cores

The number of cores to use during alignments and variant calling.

## --out

A directory in which to output results. By default it will be `WI-YYYY-MM-DD` where YYYY-MM-DD is todays date.

## --fqs (FASTQs)

When running the `wi-nf` pipeline you must provide a sample sheet that tells it where fastqs are and which samples group into isotypes. By default, this is the sample sheet in the base of the wi-nf repo (`SM_sample_sheet.tsv`), but can be specified using `--fqs` if an alternative is required. The sample sheet provides information on the isotype, fastq_pairs, library, location of fastqs, and sequencing folder.

More information on the sample sheet and adding new sequence data in the [Sample sheet](#sample_sheet) section.

## --fqs_file_prefix

A prefix path for FASTQs defined in a sample sheet. The sample sheet designed for usage on Quest (`SM_sample_sheet.tsv`) uses absolute paths so no FASTQ prefix is necessary. Additionally, there is no need to set this option as it is set for you when using the `-profile` flag. This option is only necessary (maybe) with a custom dataset where you are not using absolute paths to reference FASTQs.

## --reference

A fasta reference indexed with BWA. On Quest, the reference is available here:

```
/projects/b1059/data/genomes/c_elegans/WS245/WS245.fa.gz
```

## --tmpdir

A directory for storing temporary data.

# Sample Sheet

The sample sheet defines which FASTQs belong to which isotypes, set FASTQ IDs, library, and more. The sample sheet is constructed using the `scripts/construct_SM_sheet.sh` script. When new sequence data is added, this needs to be modified to add information about the new FASTQs. Updates to this script should be committed to the repo.

!!! Important
    You have to assign isotypes using the concordance script before they can be processed using wi-nf.

When you run the `scripts/construct_SM_sheet.sh`, it will output the `SM_Sample_sheet.tsv` file in the base of the repo. You should carefully examine the diff of this file using Git to ensure it is modifying the sample sheet correctly. __Errors can be disasterous__. Note that the output uses absolute paths to FASTQ files.

__Sample sheet structure__

```
AB1 BGI1-RET2-AB1   RET2    /projects/b1059/data/fastq/WI/dna/processed/original_wi_set/BGI1-RET2-AB1-trim-1P.fq.gz /projects/b1059/data/fastq/WI/dna/processed/original_wi_set/BGI1-RET2-AB1-trim-2P.fq.gz original_wi_set
AB1 BGI2-RET2-AB1   RET2    /projects/b1059/data/fastq/WI/dna/processed/original_wi_set/BGI2-RET2-AB1-trim-1P.fq.gz /projects/b1059/data/fastq/WI/dna/processed/original_wi_set/BGI2-RET2-AB1-trim-2P.fq.gz original_wi_set
AB1 BGI3-RET2b-AB1  RET2b   /projects/b1059/data/fastq/WI/dna/processed/original_wi_set/BGI3-RET2b-AB1-trim-1P.fq.gz    /projects/b1059/data/fastq/WI/dna/processed/original_wi_set/BGI3-RET2b-AB1-trim-2P.fq.gz    original_wi_set
```

Notice that the file does not include a header. The table with corresponding columns looks like this.

| isotype   | fastq_pair_id   | library   | fastq-1-path   | fastq-2-path   | sequencing_folder |
|:----|:---------------|:------|:-----------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------|:----------------|
| AB1 | BGI1-RET2-AB1  | RET2  | /projects/b1059/data/fastq/WI/dna/processed/original_wi_set/BGI1-RET2-AB1-trim-1P.fq.gz  | /projects/b1059/data/fastq/WI/dna/processed/original_wi_set/BGI1-RET2-AB1-trim-2P.fq.gz  | original_wi_set |
| AB1 | BGI2-RET2-AB1  | RET2  | /projects/b1059/data/fastq/WI/dna/processed/original_wi_set/BGI2-RET2-AB1-trim-1P.fq.gz  | /projects/b1059/data/fastq/WI/dna/processed/original_wi_set/BGI2-RET2-AB1-trim-2P.fq.gz  | original_wi_set |
| AB1 | BGI3-RET2b-AB1 | RET2b | /projects/b1059/data/fastq/WI/dna/processed/original_wi_set/BGI3-RET2b-AB1-trim-1P.fq.gz | /projects/b1059/data/fastq/WI/dna/processed/original_wi_set/BGI3-RET2b-AB1-trim-2P.fq.gz | original_wi_set |

The columns are detailed below:

* __isotype__ - The name of the isotype. It is used to group FASTQ-pairs into BAMs which are treated as individuals.
* __fastq_pair_id__ - This must be unique identifier for all individual FASTQ pairs. There is (unfortunately) no standard for this.
* __library__ - A name identifying the DNA library. If the FASTQ-pairs for a strain were sequenced using different library preps they should be assigned different library names. Likewise, if they were the same DNA library they should have the same library name. Keep in mind that within an isotype the library names for each strain must be independent.
* __fastq-1-path__ - The __absolute__ path of the first fastq.
* __fastq-2-path__ - The __absolute__ path of the second fastq.

# Output

The output from the pipeline is structured to enable it to easily be integrated with CeNDR. The final output directory looks like this:

```
├── log.txt
├── alignment
│   ├── isotype_bam_idxstats.tsv
│   ├── isotype_bam_stats.tsv
│   ├── isotype_coverage.full.tsv
│   └── isotype_coverage.tsv
├── cegwas
│   ├── kinship.Rda
│   └── snps.Rda
├── isotype
│   ├── tsv
│   │   ├── <isotype>.(date).tsv.gz
│   │   └── <isotype>.(date).tsv.gz
│   └── vcf
│       ├── <isotype>.(date).vcf.gz
│       └── <isotype>.(date).vcf.gz.tbi
├── phenotype
│   ├── MT_content.tsv
│   ├── kmers.tsv
│   └── telseq.tsv
├── popgen
│   ├── WI.(date).tajima.bed.gz
│   ├── WI.(date).tajima.bed.gz.tbi
│   └── trees
│       ├── I.pdf
│       ├── I.png
│       ├── I.tree
│       ├── ...
│       ├── genome.pdf
│       ├── genome.png
│       └── genome.tree
├── report
│   ├── multiqc.html
│   └── multiqc_data
│       ├── multiqc_bcftools_stats.json
│       ├── multiqc_data.json
│       ├── multiqc_fastqc.json
│       ├── multiqc_general_stats.json
│       ├── multiqc_picard_AlignmentSummaryMetrics.json
│       ├── multiqc_picard_dups.json
│       ├── multiqc_picard_insertSize.json
│       ├── multiqc_samtools_idxstats.json
│       ├── multiqc_samtools_stats.json
│       ├── multiqc_snpeff.json
│       └── multiqc_sources.json
├── tracks
│   ├── (date).HIGH.bed.gz
│   ├── (date).HIGH.bed.gz.tbi
│   ├── (date).LOW.bed.gz
│   ├── (date).LOW.bed.gz.tbi
│   ├── (date).MODERATE.bed.gz
│   ├── (date).MODERATE.bed.gz.tbi
│   ├── (date).MODIFIER.bed.gz
│   ├── (date).MODIFIER.bed.gz.tbi
│   ├── phastcons.bed.gz
│   ├── phastcons.bed.gz.tbi
│   ├── phylop.bed.gz
│   └── phylop.bed.gz.tbi
└── variation
    ├── WI.(date).soft-filter.vcf.gz
    ├── WI.(date).soft-filter.vcf.gz.csi
    ├── WI.(date).soft-filter.vcf.gz.tbi
    ├── WI.(date).soft-filter.stats.txt
    ├── WI.(date).hard-filter.vcf.gz
    ├── WI.(date).hard-filter.vcf.gz.csi
    ├── WI.(date).hard-filter.vcf.gz.tbi
    ├── WI.(date).hard-filter.stats.txt
    ├── WI.(date).hard-filter.genotypes.tsv
    ├── WI.(date).hard-filter.genotypes.frequency.tsv
    ├── WI.(date).impute.vcf.gz
    ├── WI.(date).impute.vcf.gz.csi
    ├── WI.(date).impute.vcf.gz.tbi
    ├── WI.(date).impute.stats.txt
    ├── sitelist.tsv.gz
    └── sitelist.tsv.gz.tbi
```

### log.txt

A summary of the nextflow run.

### alignment/

Alignment statistics by isotype

* __isotype_bam_idxstats.tsv__ - A summary of mapped and unmapped reads by sample.
* __isotype_bam_stats.tsv__ - BAM summary at the sample level
* __isotype_coverage.full.tsv__ - Coverage at the sample level
* __isotype_coverage.tsv__ - Simple coverage at the sample level.

### cegwas/

* __kinship.Rda__ - A kinship matrix constructed from `WI.(date).impute.vcf.gz`
* __snps.Rda__ - A mapping snp set generated from `WI.(date).hard-filter.vcf.gz`

### isotype

This directory contains files that integrate with the genome browser on CeDNR.

#### isotype/tsv/

This directory contains tsv's that are used to show where variants are in CeNDR.

#### isotype/vcf/

This direcoty contains the VCFs of isotypes.

#### phenotype/

* __MT_content.tsv__ - Mitochondrial content (MtDNA cov / Nuclear cov).
* __kmers.tsv__ - 6-mers for each isotype.
* __telseq.tsv__ - Telomere length for each isotype ~ split out by read group.

#### popgen/

* __WI.(date).tajima.bed.gz__ - Tajima's D bedfile for use on the report viewer of CeNDR.
* __WI.(date).tajima.bed.gz.tbi__ - Tajima's D index.
* __trees/__ - Phylogenetic trees for each chromosome and the entire genome.
    * I.(pdf|png|tree)
    * ...
    * genome.(pdf|png|tree)

#### report/

* multiqc.html - A comprehensive report of the sequencing run.
* __multiqc\_data/__
    * multiqc_bcftools_stats.json
    * multiqc_data.json
    * multiqc_fastqc.json
    * multiqc_general_stats.json
    * multiqc_picard_AlignmentSummaryMetrics.json
    * multiqc_picard_dups.json
    * multiqc_picard_insertSize.json
    * multiqc_samtools_idxstats.json
    * multiqc_samtools_stats.json
    * multiqc_snpeff.json
    * multiqc_sources.json

#### track/

* __(date).LOW.bed.gz(+.tbi)__ - LOW effect mutations bed track and index.
* __(date).MODERATE.bed.gz(+.tbi)__ - MODERATE effect mutations bed track and index.
* __(date).HIGH.bed.gz(+.tbi)__ - HIGH effect mutations bed track and index.
* __(date).MODIFIER.bed.gz(+.tbi)__ - MODERATE effect mutations bed track and index.
* __phastcons.bed.gz(+.tbi)__ - Phastcons bed track and index.
* __phylop.bed.gz(+.tbi)__ - PhyloP bed track and index

#### Variation/

* __WI.(date).soft-filter.vcf.gz(+.csi|+.tbi)__ -
* __WI.(date).soft-filter.stats.txt__ -
* __WI.(date).hard-filter.vcf.gz(+.csi|+.tbi)__ -
* __WI.(date).hard-filter.stats.txt__ -
* __WI.(date).hard-filter.genotypes.tsv__ -
* __WI.(date).hard-filter.genotypes.frequency.tsv__ -
* __WI.(date).impute.vcf.gz(+.csi|+.tbi)__ -
* __WI.(date).impute.stats.txt__ -
* __sitelist.tsv.gz(+.tbi)__ - Union of all sites identified in original SNV variant calling round.


text:
---
hrs_structure:
  division: '2'
  volume: '11'
  title: '27'
  chapter: '490'
  section: '490:7-509'
type: hrs_section
tags:
  - Business
menu:
  hrs:
    identifier: HRS_0490-0007-0509
    parent: HRS0490
    name: '490:7-509 Adequate compliance with commercial contract'
weight: 78075
title: Adequate compliance with commercial contract
full_title: '490:7-509 Adequate compliance with commercial contract'
---
**§490:7-509 Adequate compliance with commercial contract.** Whether a document of title is adequate to fulfill the obligations of a contract for sale, a contract for lease, or the conditions of a letter of credit is determined by article 2, 2A, or 5\. [L 2004, c 163, pt of §1]

repository: SamMade/BlueAndKrabby

repo_name: SamMade/BlueAndKrabby


text:
---
id: computer-networks-welcome
title: Welcome to CN
course: computer-networks
lecture: welcome
---

# Welcome

## Howdy Friends

**NB**: These are the new CN notes. If you want to look at the old CN notes, you can find them [here](../computer-networks-old/welcome).

Each document in "Lecture Notes" corresponds to a module in Canvas. Within each document, the headings correspond to the videos within that module.

## Textbook Information

The following textbooks have been recommended for this course:

- [Computer Networking: A Top-Down Approach](https://amzn.to/3bTL2o3)
- [Computer Networks](https://amzn.to/3gjF0km)

## Download These Notes

Some students have asked for PDF versions of the notes for a simpler, more portable
studying experience. The following PDFs are available for download.

- [Computer Networks - Complete](https://payhip.com/b/tCvzd 'The complete set of CN lecture notes, covering content from all twelve lectures.')


text:
---
title: Snakes and sticks and skins
date: '2018-12-17 06:00:00'
image: stick_snake_.jpg
tags: snake fear phobia ophidiophobia ophiophobia amygdala anxiety stress reactivity
  scorpion
categories: pests
layout: post
---

I have ophidiophobia or ophiophobia (fear of snakes). Not herpetophobia (fear of reptiles) since anything with legs is no longer terrifying. [Lizards (snakes with legs), for example, are fascinating to me](https://reverdecer.annalisagross.com/2018/11/06/lizard-gallery/) especially [when engaging the cats](https://reverdecer.annalisagross.com/2018/11/09/kittens-vs-lizards/).

But snakes freak me out so much that I have to stick a camera between us to keep from turning to flee. Why was this snake in a decorative pool near the San Antonio River?

<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/9YjqDOYRjP0" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

While we discussed it, a kinder passerby moved it out with a stick.

Our amygdala processes this potential threat with a hormones to boost our reactivity. Then our visual processing decides whether it's a snake or stick.

Turns out, same with our ears!

We were hiking when Phillip, several strides ahead, pointed back to my feet and said, "There's a snake skin!" And as I gasped and jumped back I processed the "skin" part.

[![](/images/snake_skin_.jpg){:width="60%"}](/images/snake_skin.jpg)

The skin might be interesting, except it means a snake was there, in a place we often hike. We've seen a dead one on the road and only one, alive, in El Terreno.

gha_language: CSS


text:
---
lang: en
layout: post
title: Interoperate our platforms… what does it concretely mean? How can we make it
  happen?
description: 'Starting to envision the data architecture: the different steps of the
  data and the impacts of different technological approaches'
auteur: Myriam Bouré
img: img/data.jpg
comments: true
date: 2017-02-22 10:17
img_credit: []

---
**February 22nd, 2017. Second meeting.**

Before we start getting our hands dirty on a concrete prototype, we wanted to take the time to build our collective awareness on what it means to interoperate our platforms. What can be the impact? What can be the technological solutions we could use? So that we can all engage with awareness.

Here is how we formulated the intention for that second session :

_“Reach a shared consciousness about the possible solutions to interoperate our platforms and their respective impacts, and start to envision an architecture on how “products” and “orders” data could be described, filled in, shared and accessed.”_

## The different steps of the data

To simplify daily management of products catalogs for farmers and food hubs, we need platforms to be able to share data. For example, a producer working with various food hubs using different platforms should be able to manage his inventory in one platform and then allow the other hubs to access and display it (or part of it) on their own shop front. What is the life cycle of data and what do we need to do at each stage to be able to share it?

**1. Describe.**
To enable our platforms to share data that can be used by other platforms, we need to adopt a common vocabulary, a standard. That doesn’t mean that every platform will need to change its terminology. That means “when you talk about a tomato and I talk about a tomato, are we talking about the same tomato?”. The solution would be to define a meta-ontology encompassing all the specific ontologies used by every actor.

**2. Fill in.**
Data are currently entered in a specific way and format, depending on each platform. No need to make that evolve to ensure interoperability, APIs will make it possible whatever the origin format. It might be interesting however, to consider options to enrich the data through a semantic format. Semantic data makes it easier to connect with external databases (see [wikidata](https://www.wikidata.org/wiki/Wikidata:Main_Page) / [dbpedia](http://wiki.dbpedia.org/)) and make the data smarter.

**3. Share.**
Given the meta-ontology described, every platform can set up an API that will enable to “match” fields used by different platforms and display information from distant databases. This stage has more impact on how platforms and applications are built. The platform will now need to query not only its own database, but also distant databases through the API. Can it be done in a dynamic way? Is a proxy needed to copy the data into the platform database before displaying them?

**4. View.**
When the data has been “called” through the API and is ready to be displayed in the platform’s front end, the last step is to display it so that it can be read by the client in his browser. If the platform wants to display new field they need to add them, but else, no specific work needs to be done on the platform side.

![](/img/webdistribueenglish.jpeg)

## Which data are we talking about and what questions does that raise?

It’s pretty easy to imagine the tomatoes of Charlie’s Grow Organic, who manages his catalog in The Food Assembly, displayed on a buying group shopfront using Open Food France. But what about “orders data”? If I buy tomatoes in my buying group using Open Food France, how can Charlie’s Grow Organic receive the order and process it? Does he receive the order in his Food Assembly interface? Does he receive an email notification and need to go in Open Food France to see and process the order? Does he only receive an email? Does he use a third party interface where he receives orders from any platform and process them there? The last option would probably be the most logical. But then what should be the governance of that third party interface?

So yes, we will start by describing a meta-ontology that we can use as a standard to share information about products. Each platform will need to build an API respecting that standard if they want to be interoperable with the other ones. That will be a first step, and we’ll see then how to move forward. Butlet’s keep in mind what we do that for: to simplify the life of the farmers and build a decentralized and efficient food system.

## About semantic data: the different technological approaches

If the players around the table want to evolve toward semantic data and benefit from the possibilities it opens in terms of connectedness, there are 4 main technological possibilities that should be considered all along the project.

1- An external shared solution where data are filled in and can then be displayed in any platform. For example, the producers catalogs are managed in a third party producer application, all the data are entered there and then can be displayed in all the applications that the producer has authorized.

2- A shared solution deployed internally by each actor. For example a module/plug-in that enables data to be entered in every platform but that is connected to a database that any platform can then access (given permission from the producer)

3- Each platform chooses how the data are filled in and just deploy the API to share them.

4- A third party “pivot” tool called “semantic bus” is implemented and interfaced with every platform, so anytime a data is entered in a platform, it is “processed” by this semantic bus and can be reintegrated in a semantic format and/or shared given the rules agreed upon.

Some of those approaches are based on a shared common, a solution that we decide to deploy together and use, because it benefits us all. Of course in that case we will need to build a collaborative governance around it. No choice need to be made at that stage, we will see when we move forward on the prototype what makes more sense, and the solution chosen can change when we move forward and work on other cases. But I think it does make sense **to ask ourselves at each moment the question**:

> **What do we need to manage in common to make the life of producers easier?**

## Next steps

In our next session we will get our hands dirty, discuss about potential use cases and chose the one we are going to work on for our first prototype :-)

Want to take part in the discussion ? [Contact us](http://datafoodconsortium.org/#contact) or leave a comment !

gha_language: HTML