text:
# Fine-Tuning week of XLSR-Wav2Vec2 on 60 languages 🌍

Welcome to the fine-tuning week! The goal of this week is to have state-of-the-art automatic speech recognition (ASR) models in as many languages as possible. The fine-tuning week ends on Friday, the 26th March at midnight PST time.

Participants are encouraged to fine-tune the pretrained [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) checkpoint on one or more of the 60 languages of [Common Voice dataset](https://commonvoice.mozilla.org/en/datasets).
Furthermore, it is very much appreciated if participants fine-tune XLSR-Wav2Vec2 on a language that is not included in the Common Voice dataset.

All fine-tuned models uploaded until Friday, the 26th March midnight PST, will be taken into account for competition, and the best model per language will be awarded a prize if the best model performs reasonably well.
The testing data to evaluate the models will be the official [Common Voice dataset](https://commonvoice.mozilla.org/en/datasets) *`test data`* of version 6.1. Again, participants are very much encouraged to fine-tune XLSR-Wav2Vec2 on languages that are not found in the Common Voice dataset since those languages are even more likely to be underrepresented in the speech community.
Each model fine-tuned on a language not found in Common Voice, will be evaluated by the Hugging Face team after Friday, the 26th March at midnight PST, and if the model performs reasonably well, the model receives a prize as well.
For more information on which data can be used for training, how the models are evaluated exactly, and what type of data preprocessing can be used, please see ["Training and Evaluation Rules"](#training-and-evaluation-rules).

**Please keep in mind:**
The spirit of the fine-tuning week is to provide state-of-the-art speech recognition in as many languages as possible to the community!
So while we encourage healthy competition between people/groups of the same language so that better results are obtained, it is extremely important that we help each other and share our insights with the whole team/community.
What matters in the end is what has been achieved by the team as a whole during the fine-tuning week.
That being said, we strongly encourage people to share tips & tricks on the forum or Slack, help each other when team members encounter bugs, and work in groups.
To make it easier to share and help, forum threads have been created under the name {language} ASR: Fine-Tuning Wav2Vec2, e.g. here.
It is very much possible that prizes will be given to groups of people instead of individuals. Also, don't hesitate to ask questions, propose improvements to the organization, to the material given to participants, etc...🤗

## Table of Contents

- [Organization of the fine tuning week](#organization-of-the-fine-tuning-week)
- [How to fine tune XLSR Wav2Vec2](#how-to-fine-tune-xlsr-wav2vec2)
	- [Google colab setup](#google-colab-setup)
	- [Local machine](#local-machine)
- [How to upload my trained checkpoint](#how-to-upload-my-trained-checkpoint)
	- [How to create the README](#how-to-create-the-readme)
- [How to evaluate my trained checkpoint](#how-to-evaluate-my-trained-checkpoint)
- [Rules of training and evaluation](#rules-of-training-and-evaluation)
- [Tips and tricks](#tips-and-tricks)
	- [How to combine multiple datasests into one](#how-to-combine-multiple-datasets-into-one)
	- [How to effectively preprocess the data](#how-to-effectively-preprocess-the-data)
	- [How to efficiently preproces the data](#how-to-do-efficiently-load-datasets-with-limited-ram-and-hard-drive-space)
	- [How to do hyperparameter tuning](#how-to-do-hyperparameter-tuning)
	- [How to preprocess and evaluate character based languages](#how-to-preprocess-and-evaluate-character-based-languages)
- [Further reading material](#further-reading-material)
- [FAQ](#faq)

## Organization of the fine tuning week

The week officially starts on 22.03.2021 and ends on 29.03.2021, but you are more than welcome to start fine-tuning models before the start date.
General questions you might have, general problems you encounter, and general tips can be shared directly on the Slack channel (see [this post](https://discuss.huggingface.co/t/open-to-the-community-xlsr-wav2vec2-fine-tuning-week-for-low-resource-languages/4467) on how to be added to Slack).
More language-specific questions or specific bugs should be posted on the [forum](https://discuss.huggingface.co/) (feel free to use already existing language-specific threads, *e.g.* [this one](https://discuss.huggingface.co/t/arabic-asr-fine-tuning-wav2vec2/4608) or open a new one if there is no thread for your language yet) or directly on [github](https://github.com/huggingface/transformers) if you think some code or document needs correction/improvement.
Starting on Monday, the 22.03.2021, the Hugging Face team will try to provide an overview of currently trained models along with their evaluation results.
All the necessary information on:

- How to fine-tune the XLSR model
- How to upload the model
- How to share your evaluation results & training/eval script
- What are the training/evaluation rules

can be found in the sections below. If something is still unclear, feel free to drop a message in the Slack channel.

## How to fine tune XLSR Wav2Vec2

This chapter gives an in-detail explanation of how to fine-tune [Facebook's multi-lingual Wav2vec2](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on any language of the [Common Voice dataset](https://commonvoice.mozilla.org/en/datasets).

Two possible setups can be used to fine-tune Wav2Vec2. The easiest setup is to simply use [google colab](https://colab.research.google.com/). It is possible to train the full model in a *free* google colab, but it is recommended to use google colab pro since it is more stable.

The other option is to run a script locally. While this can be more difficult to set up, it also means that you have more control over the training run and probably access to better GPUs than you would have in a google colab.
For small datasets, it is usually totally sufficient to train your model
in a google colab. For larger and thus more memory-intensive datasets, it is probably
better to fine-tune the model locally.

For each option, we explain in detail how to fine-tune XLSR-Wav2Vec2 in the following.

### Google colab setup

**Note**: Instead of reading the following section, you can simply watch [this](https://www.youtube.com/watch?v=UynYn2C3tI0&ab_channel=PatrickvonPlaten) video, where Patrick explains how to adapt the google colab for your specific language.

**1.**: If you plan on training XLSR-Wav2Vec2 in a google colab, you should first make sure to have a valid gmail account. You can sign up for a gmail account [here](https://accounts.google.com/signup/v2/webcreateaccount?hl=en&flowName=GlifWebSignIn&flowEntry=SignUp).
Having successfully signed up for gmail, you can now sign in to your account to make sure you are logged in when opening new tabs in your browser.

**2.**: Next, head over to the official [Fine-Tune XLSR-Wav2Vec2 with 🤗 Transformes](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/Fine_Tune_XLSR_Wav2Vec2_on_Turkish_ASR_with_%F0%9F%A4%97_Transformers.ipynb) google colab. The first thing you should do is to make a copy of it - click `->File->Save a copy in Drive`. This should save a copy of the google colab in your google drive.

**3.**: Now it is highly recommended to carefully read the google colab without running the cells yet.
You should get an understanding of the model is trained and what you will have to change when training the model in a different language.
Having done so, you can again head over to [Common Voice](https://commonvoice.mozilla.org/en/datasets) and pick a language you want to fine-tune [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on. Make sure you remember the language code (For each language, you can find it under the field "*Version*". It corresponds to **all characters before the first underscore**. *E.g.* for Greek it is *el*, while for Irish it is *ga-IE*.

**4.**: Now you should replace the language code used for the demo of this colab, being *tr* for Turkish with the language code corresponding to the language you just chose in the **second** cell of the google colab. This will load the correct data for your language.

**5.**: It is time to start running the google colab! Make sure that you have selected "GPU" as your runtime environment and you can start running the cells one-by-one. Make sure you attentively read the text between the cells to understand what is happening and to eventually correct the cells to improve the fine-tuning script for your language. Things you might want to improve/change:

 - Data loading. It is very much recommended to use more than just the official training data of the Common Voice dataset. If you find more data on the internet, feel free to use it! Check out the section ["How to combined multiple datasets into one"](#how-to-combine-multiple-datasets-into-one)

- Data Processing. You should adapt the data processing to your specific language. In data processing, you should make the data more uniform so that it will be easier for the model to learn how to classify speech in your data. Here it can be really helpful to be proficient in the language to know what can be done to simplify the language without changing the meaning.
Data processing methods include, but are not limited to:
	- Normalizing your data. Make sure all characters are lower-cased.
	- Remove typographical symbols and punctuation marks. See a list [here](https://en.wikipedia.org/wiki/List_of_typographical_symbols_and_punctuation_marks). Be careful to not remove punctuation marks that can change the meaning of the sentence. *E.g.* you should not remove the single quotation mark `'` in English, as it would change the words `"it's"` to `"its"` which is a different word and has thus a different meaning. For more tips on data processing see ["How to effectively preprocess the data"](#how-to-effectively-preprocess-the-data")

- Hyperparameter Tuning. Depending on the size of the data you should probably change the hyperparameters of the google colab. You can change any parameter you like. For more tips and tricks see ["How to do hyperparameter tuning for my language"](#how-to-do-hyperparameter-tuning-for-my-language)

When running the google colab make sure that you uncomment the cell corresponding to mounting your google drive to the colab. This cell looks as follows:

```python
# from google.colab import drive
# drive.mount('/content/gdrive/')
```

Uncomment it, run it, and follow the instructions to mount your google drive. This way you can be sure that the model parameters and created tokenizer & feature extractor files are saved in **your** google drive.

Also, make sure that you uncomment the cells corresponding to save the preprocessing files and trained model weights to your drive. Otherwise, you might lose a trained model if you google crashes. You should change the name of your model from `wav2vec2-large-xlsr-turkish-demo` to `wav2vec2-large-xlsr-{your_favorite_name}`.

Those cells correspond to:

```python
# processor.save_pretrained("/content/gdrive/MyDrive/wav2vec2-large-xlsr-turkish-demo")
```

and the line:

```python
  output_dir="/content/gdrive/MyDrive/wav2vec2-large-xlsr-turkish-demo",
```

further below (which should already be uncommented).

Having finished the training you should find the following files/folders under the folder `wav2vec2-large-xlsr-{your_favorite_name}` in your google drive:

- `preprocessor_config.json` - the parameters of the feature extractor
- `special_tokens_map.json` - the special token map of the tokenizer
- `tokenizer_config.json` - the parameters of the tokenizer
- `vocab.json` - the vocabulary of the tokenizer
- `checkpoint-{...}/` - the saved checkpoints saved during training. Each checkpoint should contain the files: `config.json`, `optimizer.pt`, `pytorch_model.bin`, `scheduler.pt`, `training_args.bin`. The files `config.json` and `pytorch_model.bin` define your model.

If you are happy with your training results it is time to upload your model!
Download the following files to your local computer: **`preprocessor_config.json`, `special_tokens_map.json`, `tokenizer_config.json`, `vocab.json`, `config.json`, `pytorch_model.bin`**. Those files fully define a XLSR-Wav2Vec2 model checkpoint.

Awesome you have successfully trained a XLSR-Wav2Vec2 model 😎. Now you can jump to the section ["How to upload my trained checkpoint"](#how-to-upload-my-trained-checkpoint)

### Local machine

We have provided `run_common_voice.py` script to run fine-tuning on local machine. The script is similar to the colab but allows you to launch training using command line, save and continue training from previous checkpoints and launch training on multiple GPUs.
For bigger datasets, we recommend to train Wav2Vec2 locally instead of in a google colab.

1. To begin with, we should clone transformers localy and install all the required packages.

First, you need to clone the `transformers` repo with:

```
$ git clone https://github.com/huggingface/transformers.git
```

Second, head over to the `examples/research_projects/wav2vec2` directory, where the `run_common_voice.py` script is located.

```
$ cd transformers/examples/research_projects/wav2vec2
```

Third, install the required packages. The
packages are listed in the `requirements.txt` file and can be installed with

```
$ pip install -r requirements.txt
```

	**Note**: Installing the latest version of `torchaudio` will also upgrade `torch` to it's latest stable version. If you are using specific version of `torch` then make sure
	to use the correct `torchaudio` version compatible with your version of `torch`. By default the `requirements.txt` will install the latest version of `torchaudio`.

2. Next, take a look at the `run_common_voice.py` script to get an understanding of how it works. In short the script does the following:

	- Load the given common voice dataset
	- Create vocab for the language
	- Load the model with given hyperparameters
	- Pre-process the dataset to input into the model
	- Run training
	- Run evaluation

3. The following examples show how you can launch fine-tuning for the common voice dataset.
Here we will run the script on the *Turkish* Common Voice dataset for demonstration purposes.

	 Use the `--nproc_per_node` option to specify the number of GPUs.

	Once the training is finished, the model and checkpoints will be saved under the directory specified by the `--output_dir` argument.

4. The script also allows you to resume training from the last saved checkpoint. To resume training from last saved checkpoint remove the `--overwrite_output_dir` option and run the same command again.  And to continue training from a specific checkpoint, keep the `--overwrite_output_dir`
option and pass the path of the checkpoint as `--model_name_or_path`.

As the script is based on the `Trainer` API, refer to the [Trainer docs](https://huggingface.co/transformers/main_classes/trainer.html) for more information about ``Trainer`` and ``TrainingArguments``.

[OVH cloud](https://www.ovh.com/world/) has generously offered free compute for this sprint. Please refer to [this video](https://www.youtube.com/watch?v=2hlkWAESMk8&ab_channel=Databuzzword) to get started with OVH.

## How to upload my trained checkpoint

To upload your trained checkpoint, you have to create a new model repository on the 🤗 model hub, from this page: https://huggingface.co/new

> You can also follow the more in-depth instructions [here](https://huggingface.co/transformers/model_sharing.html) if needed.

Having created your model repository on the hub, you should clone it locally:

```bash
git lfs install

git clone https://huggingface.co/username/your-model-name
```

Then and add the following files that fully define a XLSR-Wav2Vec2 checkpoint into the repository. You should have added the following files.

- `preprocessor_config.json`
- `special_tokens_map.json`
- `tokenizer_config.json`
- `vocab.json`
- `config.json`
- `pytorch_model.bin`

Having added the above files, you should run the following to push files to your model repository.
```
git add . && git commit -m "Add model files" && git push
```

The next **very important** step is to create the model card. For people to use your fine-tuned
model it is important to understand:

- What kind of model is it?
- What is your model useful for?
- What data was your model trained on?
- How well does your model perform?

All these questions should be answered in a model card which is the first thing people see when
visiting your model on the hub under `https://huggingface.co/{your_username}/{your_modelname}`.

**Note**:
It is extremely important that you add this model card or else we cannot find your model and thus cannot take the model into
account for the final evaluation.

### How to create the readme

The model card is written in markdown (`.md`) and should be added by simply clicking on the "Add model card" button which is found on the top right corner.
You are encouraged to copy-paste the following template into your model card.

**Make sure that** instead of copying the output of the markdown file you copy the **raw** version of the following part.

To get the raw version of this file, simply click on the "`raw`" button on the top right corner of this file next to "`blame`" and copy everything below the marker.
Make sure that you read and consequently remove all #TODO: statements from the model card.

<======================Copy **raw** version from here=========================
---
language: {lang_id} #TODO: replace {lang_id} in your language code here. Make sure the code is one of the *ISO codes* of [this](https://huggingface.co/languages) site.
datasets:
- common_voice #TODO: remove if you did not use the common voice dataset
- TODO: add more datasets if you have used additional datasets. Make sure to use the exact same
dataset name as the one found [here](https://huggingface.co/datasets). If the dataset can not be found in the official datasets, just give it a new name
metrics:
- wer
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
license: apache-2.0
model-index:
- name: {human_readable_name} #TODO: replace {human_readable_name} with a name of your model as it should appear on the leaderboard. It could be something like `Elgeish XLSR Wav2Vec2 Large 53`
  results:
  - task:
      name: Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: Common Voice {lang_id} #TODO: replace {lang_id} in your language code here. Make sure the code is one of the *ISO codes* of [this](https://huggingface.co/languages) site.
      type: common_voice
      args: {lang_id} #TODO: replace {lang_id} in your language code here. Make sure the code is one of the *ISO codes* of [this](https://huggingface.co/languages) site.
    metrics:
       - name: Test WER
         type: wer
         value: {wer_result_on_test} #TODO (IMPORTANT): replace {wer_result_on_test} with the WER error rate you achieved on the common_voice test set. It should be in the format XX.XX (don't add the % sign here). **Please** remember to fill out this value after you evaluated your model, so that your model appears on the leaderboard. If you fill out this model card before evaluating your model, please remember to edit the model card afterward to fill in your value
---

# Wav2Vec2-Large-XLSR-53-{language} #TODO: replace language with your {language}, *e.g.* French

Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on {language} using the [Common Voice](https://huggingface.co/datasets/common_voice), ... and ... dataset{s}. #TODO: replace {language} with your language, *e.g.* French and eventually add more datasets that were used and eventually remove common voice if model was not trained on common voice
When using this model, make sure that your speech input is sampled at 16kHz.

## Usage

The model can be used directly (without a language model) as follows:

```python
import torch
import torchaudio
from datasets import load_dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor

test_dataset = load_dataset("common_voice", "{lang_id}", split="test[:2%]") #TODO: replace {lang_id} in your language code here. Make sure the code is one of the *ISO codes* of [this](https://huggingface.co/languages) site.

processor = Wav2Vec2Processor.from_pretrained("{model_id}") #TODO: replace {model_id} with your model id. The model id consists of {your_username}/{your_modelname}, *e.g.* `elgeish/wav2vec2-large-xlsr-53-arabic`
model = Wav2Vec2ForCTC.from_pretrained("{model_id}") #TODO: replace {model_id} with your model id. The model id consists of {your_username}/{your_modelname}, *e.g.* `elgeish/wav2vec2-large-xlsr-53-arabic`

resampler = torchaudio.transforms.Resample(48_000, 16_000)

# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
	speech_array, sampling_rate = torchaudio.load(batch["path"])
	batch["speech"] = resampler(speech_array).squeeze().numpy()
	return batch

test_dataset = test_dataset.map(speech_file_to_array_fn)
inputs = processor(test_dataset[:2]["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)

with torch.no_grad():
	logits = model(inputs.input_values, attention_mask=inputs.attention_mask).logits

predicted_ids = torch.argmax(logits, dim=-1)

print("Prediction:", processor.batch_decode(predicted_ids))
print("Reference:", test_dataset[:2]["sentence"])
```

## Evaluation

The model can be evaluated as follows on the {language} test data of Common Voice.  # TODO: replace #TODO: replace language with your {language}, *e.g.* French

```python
import torch
import torchaudio
from datasets import load_dataset, load_metric
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import re

test_dataset = load_dataset("common_voice", "{lang_id}", split="test") #TODO: replace {lang_id} in your language code here. Make sure the code is one of the *ISO codes* of [this](https://huggingface.co/languages) site.
wer = load_metric("wer")

processor = Wav2Vec2Processor.from_pretrained("{model_id}") #TODO: replace {model_id} with your model id. The model id consists of {your_username}/{your_modelname}, *e.g.* `elgeish/wav2vec2-large-xlsr-53-arabic`
model = Wav2Vec2ForCTC.from_pretrained("{model_id}") #TODO: replace {model_id} with your model id. The model id consists of {your_username}/{your_modelname}, *e.g.* `elgeish/wav2vec2-large-xlsr-53-arabic`
model.to("cuda")

chars_to_ignore_regex = '[\,\?\.\!\-\;\:\"\“]'  # TODO: adapt this list to include all special characters you removed from the data
resampler = torchaudio.transforms.Resample(48_000, 16_000)

# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
	batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower()
	speech_array, sampling_rate = torchaudio.load(batch["path"])
	batch["speech"] = resampler(speech_array).squeeze().numpy()
	return batch

test_dataset = test_dataset.map(speech_file_to_array_fn)

# Preprocessing the datasets.
# We need to read the aduio files as arrays
def evaluate(batch):
	inputs = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)

	with torch.no_grad():
		logits = model(inputs.input_values.to("cuda"), attention_mask=inputs.attention_mask.to("cuda")).logits

	pred_ids = torch.argmax(logits, dim=-1)
	batch["pred_strings"] = processor.batch_decode(pred_ids)
	return batch

result = test_dataset.map(evaluate, batched=True, batch_size=8)

print("WER: {:2f}".format(100 * wer.compute(predictions=result["pred_strings"], references=result["sentence"])))
```

**Test Result**: XX.XX %  # TODO: write output of print here. IMPORTANT: Please remember to also replace {wer_result_on_test} at the top of with this value here. tags.

## Training

The Common Voice `train`, `validation`, and ... datasets were used for training as well as ... and ...  # TODO: adapt to state all the datasets that were used for training.

The script used for training can be found [here](...) # TODO: fill in a link to your training script here. If you trained your model in a colab, simply fill in the link here. If you trained the model locally, it would be great if you could upload the training script on github and paste the link here.

=======================To here===============================>

Your model in then available under *huggingface.co/{your_username}/{your_chosen_xlsr-large_model_name}* for everybody to use 🎉.

## How to evaluate my trained checkpoint

Having uploaded your model, you should now evaluate your model in a final step. This should be as simple as
copying the evaluation code of your model card into a python script and running it. Make sure to note
the final result on the model card **both** under the YAML tags at the very top **and** below your evaluation code under "Test Results".

## Rules of training and evaluation

In this section, we will quickly go over what data is allowed to be used as training
data, what kind of data preprocessing is allowed be used, and how the model should be evaluated.

To make it very simple regarding the first point: **All data except the official common voice `test` data set can be used as training data**. For models trained in a language that is not included in Common Voice, the author of the model is responsible to
leave a reasonable amount of data for evaluation.

Second, the rules regarding the preprocessing are not that as straight-forward. It is allowed (and recommended) to
normalize the data to only have lower-case characters. It is also allowed (and recommended) to remove typographical
symbols and punctuation marks. A list of such symbols can *e.g.* be fonud [here](https://en.wikipedia.org/wiki/List_of_typographical_symbols_and_punctuation_marks) - however here we already must be careful. We should **not** remove a symbol that
would change the meaning of the words, *e.g.* in English, we should not remove the single quotation mark `'` since it
would change the meaning of the word `"it's"` to `"its"` which would then be incorrect. So the golden rule here is to
not remove any characters that could change the meaning of a word into another word. This is not always obvious and should
be given some consideration. As another example, it is fine to remove the "Hypen-minus" sign "`-`" since it doesn't change the
meaninng of a word to another one. *E.g.* "`fine-tuning`" would be changed to "`finetuning`" which has still the same meaning.

Since those choices are not always obvious when in doubt feel free to ask on Slack or even better post on the forum, as was
done, *e.g.* [here](https://discuss.huggingface.co/t/spanish-asr-fine-tuning-wav2vec2/4586).

## Tips and tricks

This section summarizes a couple of tips and tricks across various topics. It will continously be updated during the week.

### How to combine multiple datasets into one

Check out [this](https://discuss.huggingface.co/t/how-to-combine-local-data-files-with-an-official-dataset/4685) post.

### How to effectively preprocess the data

### How to do efficiently load datasets with limited ram and hard drive space

Check out [this](https://discuss.huggingface.co/t/german-asr-fine-tuning-wav2vec2/4558/8?u=patrickvonplaten) post.

### How to do hyperparameter tuning

### How to preprocess and evaluate character based languages

## Further reading material

It is recommended that take some time to read up on how Wav2vec2 works in theory.
Getting a better understanding of the theory and the inner mechanisms of the model often helps when fine-tuning the model.

**However**, if you don't like reading blog posts/papers, don't worry - it is by no means necessary to go through the theory to fine-tune Wav2Vec2 on your language of choice.

If you are interested in learning more about the model though, here are a couple of resources that are important to better understand Wav2Vec2:

- [Facebook's Wav2Vec2 blog post](https://ai.facebook.com/blog/wav2vec-state-of-the-art-speech-recognition-through-self-supervision/)
- [Official Wav2Vec2 paper](https://arxiv.org/abs/2006.11477)
- [Official XLSR Wav2vec2 paper](https://arxiv.org/pdf/2006.13979.pdf)
- [Hugging Face Blog](https://huggingface.co/blog/fine-tune-xlsr-wav2vec2)
- [How does CTC (Connectionist Temporal Classification) work](https://distill.pub/2017/ctc/)

It helps to have a good understanding of the following points:

- How was XLSR-Wav2Vec2 pretrained? -> Feature vectors were masked and had to be predicted by the model; very similar in spirit to masked language model of BERT.

- What parts of XLSR-Wav2Vec2 are responsible for what? What is the feature extractor part used for? -> extract feature vectors from the 1D raw audio waveform; What is the transformer part doing? -> mapping feature vectors to contextualized feature vectors; ...

- What part of the model needs to be fine-tuned? -> The pretrained model **does not** include a language head to classify the contextualized features to letters. This is randomly initialized when loading the pretrained checkpoint and has to be fine-tuned. Also, note that the authors recommend to **not** further fine-tune the feature extractor.

- What data was used to XLSR-Wav2Vec2? The checkpoint we will use for further fine-tuning was pretrained on **53** languages.

- What languages are considered to be similar by XLSR-Wav2Vec2? In the official [XLSR Wav2Vec2 paper](https://arxiv.org/pdf/2006.13979.pdf), the authors show nicely which languages share a common contextualized latent space. It might be useful for you to extend your training data with data of other languages that are considered to be very similar by the model (or you).

## FAQ

- Can a participant fine-tune models for more than one language?
Yes! A participant can fine-tune models in as many languages she/he likes
- Can a participant use extra data (apart from the common voice data)?
Yes! All data except the official common voice `test data` can be used for training.
If a participant wants to train a model on a language that is not part of Common Voice (which
is very much encouraged!), the participant should make sure that some test data is held out to
make sure the model is not overfitting.
- Can we fine-tune for high-resource languages?
Yes! While we do not really recommend people to fine-tune models in English since there are
already so many fine-tuned speech recognition models in English. However, it is very much
appreciated if participants want to fine-tune models in other "high-resource" languages, such
as French, Spanish, or German. For such cases, one probably needs to train locally and apply
might have to apply tricks such as lazy data loading (check the ["Lazy data loading"](#how-to-do-lazy-data-loading) section for more details).

repository: huggingface/transformers

repo_name: huggingface/transformers


text:
# Node
A graph node for use in an AStar algorithm.

This class implements `IHeapItem<Node>` and `IHashable`.

### [Public members](#public-members)
 + [`parent: Node`](#parent-node)
 + [`gCost: number`](#gcost-number)
 + [`hCost: number`](#hcost-number)
 + [`x: number`](#x-number)
 + [`y: number`](#y-number)
 + [`walkable: boolean`](#walkable-boolean)
 + [`heapIndex: number`](#heapindex-number)
 + [`fCost: number`](#fcost-number)
### [Public methods](#public-methods)
 + [`hash(): string`](#hash-string)
 + [`compareTo(item: Node): number`](#comparetoitem-node-number)

### Public members
#### `parent: Node`
The parent of this node or null when finding a path.

#### `gCost: number`
The distance of this node from the starting node when finding a path.

#### `hCost: number`
The distance of this node from the end node when finding a path.

#### `x: number`
The x coordinate of the node in game coordinates.

#### `y: number`
The y coordinate of the node in game coordinates.

#### `walkable: boolean`
Whether or not the player can walk on this node.

#### `heapIndex: number`
> Inherited from `IHeapItem<Node>`

The index of the item in the heap.

#### `fCost: number`
The sum of the `gCost` and the `hCost`.

### Public methods
#### `hash(): string`
> Inherited from `IHashable`

Returns a concatenation of the `x` and `y` properties.

#### `compareTo(item: Node): number`
> Inherited from `IHeapItem<T>`

Returns `-1` if the instance's `fCost` is higher than that of the `item`'s `fCost`. If the `fCost` is the same, then the `hCost` will be compared in the same manner. Otherwise returns `1`.

license: MIT

repository: Arkzzz/nrelay

branch_name: refs/heads/master

extension: md

gha_license_id: MIT

license_type: permissive

repo_name: Arkzzz/nrelay


text:
git-reset-file(1) -- Reset one file
================================

## SYNOPSIS

`git-reset-file` [<filename>] commit-hash

## DESCRIPTION

  Reset one file to HEAD or certain commit-hash

## OPTIONS

  <filename>

  The name of the file to reset.

  <commit-hash>

  (Optional) Hash of commit to reset the file to. Defaults to HEAD.

## EXAMPLES

  Reset one file to HEAD

	$ git reset-file .htaccess

  or reset one file to certain commit

	$ git reset-file .htaccess dc82b19

## AUTHOR

Written by Sasha Khamkov<EMAIL_ADDRESS>
## REPORTING BUGS

<<https://github.com/tj/git-extras/issues>>

## SEE ALSO

<<https://github.com/tj/git-extras>>

repository: tj/git-extras

gha_language: Shell

repo_name: tj/git-extras


text:
# Adafruit Feather M0 RFM95

[Quickstart Guide](https://developer.helium.com/devices/arduino-quickstart/adafruit-feather-m0-rfm95)
[Adafruit Feather M0 with RFM95 Product Page](https://www.adafruit.com/product/3178)
[Adafruit Feather M0 with RFM95 Datasheets & Files](https://learn.adafruit.com/adafruit-feather-m0-radio-with-lora-radio-module/downloads).

## Required Arduino Libraries

### Important
Make sure you do not have other LMIC arduino libraries installed, otherwise you will experience
build errors. You can find your Arduino libraries for your operating system below, simply delete the directory if you would like to remove a library.

linux: /home/{user}/Arduino/libraries
windows: Documents/Arduino/libraries
mac os: Documents/Arduino/libraries

From the Arduino IDE, open the Library Manager (Sketch->Include Library->Manage Libraries). In the search box, type the library name below and install version `1.5.0+arduino-2`.

[IBM LMIC framework](https://github.com/matthijskooijman/arduino-lmic)

### Required Configuration
This library requires that a config file be setup properly. After you have installed the `IBM LMIC framework` library, navigate to it's directory on your operating system found below. Next, replace the `config.h` file in this directory with this [config.h](https://github.com/helium/longfi-arduino/blob/master/Sparkfun-Pro-RF/longfi-us915/config.h) file.

linux: /home/{user}/Arduino/libraries/IBM_LMIC_framework/src/lmic
windows: Documents/Arduino/libraries/IBM_LMIC_framework/src/lmic
mac os: Documents/Arduino/libraries/IBM_LMIC_framework/src/lmic

## Required Arduino Board Support

1. Install Arduino SAMD Support, instructions [here](https://learn.adafruit.com/adafruit-feather-m0-radio-with-lora-radio-module/using-with-arduino-ide#install-samd-support-6-5).
2. Install Adafruit SAMD Support, instructions [here](https://learn.adafruit.com/adafruit-feather-m0-radio-with-lora-radio-module/using-with-arduino-ide#install-adafruit-samd-6-7).
3. (Windows 7 & 8 Only) Install drivers, instructions [here](https://learn.adafruit.com/adafruit-feather-m0-radio-with-lora-radio-module/using-with-arduino-ide#install-drivers-windows-7-and-8-only-6-11)

repository: EugeneScully/longfi-arduino

repo_name: EugeneScully/longfi-arduino


text:
# Shellbomb

Shellbomb is a not-so-basic tomcat/jboss remote exploitation shell.
Features are:

  - run system commands
  - traverse file system
  - upload files
  - download files with resume

# WARNING

This software is intended to be used for educational purposes and by security professionals only.

The author takes no responsibility whatsoever for any nefarious or unethical or criminal use of this software tool.

# Requirements to build war file:
 - Java 5 minium
 - Gradle

 # Instructions

 TBC

repository: fredcampos/shellbomb

repo_name: fredcampos/shellbomb


text:
---
layout: page
title: -- SI-15 INFORMATION OUTPUT FILTERING
parent: . 3.19 SYSTEM AND INFORMATION INTEGRITY
nav_order: 319150
---

## SI-15 INFORMATION OUTPUT FILTERING

Control: Validate information output from the following software programs and/or applications to ensure that the information is consistent with the expected content: [ _Assignment: organization-defined software programs and/or applications_ ].

Discussion: Certain types of attacks, including SQL injections, produce output results that are unexpected or inconsistent with the output results that would be expected from software programs or applications. Information output filtering focuses on detecting extraneous content, preventing such extraneous content from being displayed, and then alerting monitoring tools that anomalous behavior has been discovered.

Related Controls: SI-3 , SI-4 , SI-11.

Control Enhancements: None.

References: None.


text:
---
title: "ggplot your missing data"
author: 'www.njtierney.com'
date: '2015-11-12'
slug: ggplot-your-missing-data
categories:
- bloglink
tags:
  - rstats
  - njtierneycom
---

This is what the Amelia package produces by default: And let's explore the missing data using my own ggplot function: Let's test it out It's much cleaner, and easier to interpret. As another aside - if anyone knows how I can improve the image quality of the plots, I'm all ears. I've tried changing the dpi, changing it to pdf, png, svg, and even svglite but they all haven't really solved the[... <i class="fas fa-external-link-alt"></i>](https://www.njtierney.com/post/2015/11/12/ggplot-missing-data/)

repository: mkearney/data-scribers

repo_name: mkearney/data-scribers


text:
# ThisApp Methods

The following methods are available to your app when you initialize `ThisApp`:

**Note**: `_` means an HTMLElement wrapped with `this._()`. Everywhere it's used
as a parameter type, the actual HTMLElement can be used instead.

-   **_** ( element: `string` | `array` | `HTMLElement` ) : `_`

    Wraps the HTMLElement or array of objects in an object which provides methods
    to manipulate the items provided. [See available methods here](./_.md)

-   **addToCache** ( element: `_` ) : `ThisApp`

    Adds an element to the cache collection for later reuse

-   **back** ( ) : ThisApp

    Takes the app back one step in history

-   **before** ( event: `string`, callback: `function` ) : `ThisApp`

    Registers a callback to be called before an event happens. If the callback
    returns false, the event is terminated. [See list of available events here](./events.md#before-events)

-   **bindToObject** (  element:  `_`, object:  `object`, callback: `function` ) : `Promise`

    Binds an element to an object. This will bind every model property in the element
    to the corresponding key on the object provided and calls the callback function
    when done with the bounded element has the only parameter.
    [See more about models](./features/models.md)

-   **clearDOMCache** () : `ThisApp`

    Clears all the cached DOM elements.

-   **clearPageCache** ( reload: `boolean` ) : `ThisApp`

    Clears the cache of all models and collections on the current page. If parameter
    is `true`, models and collections on the page would be reloaded afterwards.

-   **collection** ( modelName: `string`, options: `object` ) : `Promise`

    Fetches a model's [collection object](./js-collection.md).

-   **debug** ( mode: `boolean` ) : `ThisApp`

    Sets the app debug mode. If called without a parameter, `true` is assumed.

-   **fillAutocompleteList** ( list: `_` | `string`, data: `object` | `array`, idKey: `string`, filter: `string` ) : `ThisApp`

    Fills an [autocomplete](./others/autocomplete.md) list with the data given.

-   **forward** ( ) : `ThisApp`

    Takes the app forward one step in history

-   **getCached** ( selector: `string` | `_`, type: `string` ) : `ThisApp`

    Fetches a clone of the required elements from the cache collection

-   **home** ( replaceState: `boolean` ) : `ThisApp`

    Returns the app to the home page.

    If parameter is `true`, the current page is lost in history and replaced with
    the home page.

-   **load** ( element: `_`, data: `object`, callback: `function` ) : `Promise`

    Loads any feature element that can be on the page (collection, model, component).

    The callback function is called after the element has been loaded.

-   **loadPage** ( pageIdOrPath: `string`, replaceState: `boolean` ) : `ThisApp`

    Loads the given page.

    If `replaceState` is `true`, the current page is lost in history and replaced with
    the given page.

-   **on** ( event: `string`, selector: `string`, callback: `function` ) : `ThisApp`

    Adds event listeners to elements in the app container.
    [See list of available events here](./events.md#after-events)

-   **onError** ( callback: `function` ) : `ThisApp`

    Accepts a function to call whenever any error occurs in the lifetime of the app.

    The function passed a `string` error message or an object in case of a `catch`.

-   **pageNotFound** ( callback: `function` | `string` ) : `ThisApp`

    Sets up the function to call when a page is not found. Alternatively, a string
    id of page to load could be passed in.

    If `callback` is a function, the function is given the string page id.

-   **promise** ( func: `function` ) : `Promise`

    Creates a promise while ensuring the function's context is ThisApp. The provided
    is provided with parameters `resolve` and `reject` which both are functions to
    be called on success or failed processing within `func` respectively.

-   **reload** ( layoutsToo: `boolean` ) : `ThisApp`

    Reloads the current page. If paramater is `true`, then the layouts are reloaded
    as well.

-   **replaceCached** ( selector: `string`, element: `_` ) : `ThisApp`

    Replaces cached elements (identified by `selector`) with the given element.

-   **request** ( config: `string` | `object` ) : `Promise`

    Sends an AJAX request.

    If parameter is string, it is the url and a `GET` request is sent to the url
    expecting a `json` response.

    The config should extend the JSON object below:

    ```javascript
    {
        "type': "GET", // POST | PATCH | PUT | DELETE
        "url": location.href, // The url to connect to. Default is current url
        "dataType": "json", // The expected response type
        "withCredentials": null, // Boolean. Indicates whether to send request with credientials or not.
        "async": true, // Asyncronous request or not
        "data": {}, // The object or query string data to send with the request. This may also be an instance of FormData
        "headers": {}, // Object of string keys to string values to pass to the request header
        "success": function(){}, // Function to call when a success response is gotten. The response data is passed as a parameter
        "error": function(){} // Function to call when error occurs
    }
    ```

-   **resetAutocomplete** ( id: `string` ) : `ThisApp`

    Reset an autocomplete input element. The parameter should be a single id of the
    element with `this-autocomplete` attribute and a comma-separated list of such ids.

-   **secureAPI** ( func: `function` ) : `ThisApp`

    Function call before all api requests are sent. You can set/overwrite request
    headers and data api here.

    Parameters are `object` headers, `object` data and `object` options.

    `options` override the options set in `request()`

-   **setBaseURL** ( url: `string` ) : `ThisApp`

    Set the base url for the app.

-   **setComponentsPath** ( path: `string`, fileExtension: `string`, minification: `object` ) : `ThisApp`

    Sets the path from where to load components from. Parameter `fileExtension`
    indicates the file extension for components. Default is `html`

    The minification object has the following defaults:

    ```javascript
    {
        "prod": false, // use minified in production
        "dev": false, // use minified in developement (when debug:true)
        "subdir": "" // subdirectory containing minified files
    }
    ```

-   **setCSSPath** ( path: `string`, minification: `object` ) : `ThisApp`

    Sets the path from where to load CSS files from.

    The minification object has the following defaults:

    ```javascript
    {
        "prod": false, // use minified in production
        "dev": false, // use minified in developement (when debug:true)
        "subdir": "" // subdirectory containing minified files
    }
    ```

-   **setDataKey** ( dataKey: `string` ) : `ThisApp`

    Sets the key in the response object which holds the data array. Default is `data`

-   **setDataTransport** ( func: `function` ) : `ThisApp`

    Sets the the transport system to use for data connection.

    The function would be passed an `object` parameter which has the following keys
    in addition to those in `request()`:

    ```javascript
    {
        // ...
        "action": "", // create | read | update | delete | autocomplete | handled-submit
        "elem": _, // Available only when action is read
        "id": "", // Available only when action is update
        "isCollection": true, // or false. Available when loading a collection or a model
        "form": HTMLFormElement, // Available when submitting a form
    }
    ```

-   **setDefaultLayout** ( id: `string` ) : `ThisApp`

    Sets the default layout for the application

-   **setJSPath** ( path: `string`, minification: `object` ) : `ThisApp`

    Sets the path from where to load JavaScript files from.

    The minification object has the following defaults:

    ```javascript
    {
        "prod": false, // use minified in production
        "dev": false, // use minified in developement (when debug:true)
        "subdir": "" // subdirectory containing minified files
    }
    ```

-   **setLayoutsPath** ( path: `string`, fileExtension: `string`, minification: `object` ) : `ThisApp`

    Sets the path from where to load layouts from. Parameter `fileExtension`
    indicates the file extension for layouts. Default is `html`

    The minification object has the following defaults:

    ```javascript
    {
        "prod": false, // use minified in production
        "dev": false, // use minified in developement (when debug:true)
        "subdir": "" // subdirectory containing minified files
    }
    ```

-   **setPagesPath** ( path: `string`, fileExtension: `string`, minification: `object` ) : `ThisApp`

    Sets the path from where to load pages from. Parameter `fileExtension`
    indicates the file extension for pages. Default is `html`

    The minification object has the following defaults:

    ```javascript
    {
        "prod": false, // use minified in production
        "dev": false, // use minified in developement (when debug:true)
        "subdir": "" // subdirectory containing minified files
    }
    ```

-   **setStore** ( func: `function` ) : `ThisApp`

    Sets the client side store/db to use in the app. The default store is localStorage.

    The function must take a `string` value which is the name of the collection and
    return a collection object with the methods as explained on [`store()`](#store)

-   **setTitleContainer** ( selector: `string` ) : `ThisApp`

    Sets the container that would always hold the current page's title as it is
    being loaded.

-   **setTransition** ( transition:  `string` | `function`, options: `object` ) : `ThisApp`

    Set the transition effect to use between pages. The default is 'switch'.
    [See more about transitions](./transitions.md)

-   **setUploader** ( func: `function` ) : `ThisApp`

    Sets the function to call when there are files to be uploaded in a form.

    The function is called with an object parameter:

    ```javascript
    {
        "modelName": "", // the model on which the upload action is being carried out
        "files": [], // ann array of all file elements in the form,
        "data": {}, // a copy of the data gotten from the form
        "id": "#", // the id of the model, if updating one
        "url": "...", // the url of the model
        // This function must be called when upload is done. The form data
        // would be updated with the data object passed into this function.
        // The second parameter should be a function to call in case saving the model
        // fails. Here, the uploaded file may be deleted.
        "done": function( data:  `object`, cancelUpload: `function` )
    }
    ```

-   **start** ( pageId: `string`, freshCopy: `boolean` ) : `ThisApp`

    Starts the app.

    If `pageId` is provided, the page is loaded and set as the home page.
    If `freshCopy` is true, a fresh copy of the page is rendered. Otherwise, a
    copy from history is rendered, if available.

-   **store** ( collectionName: `string` ) : `Store`

    Fetches the store object for the given `collectionName`.

    The store object has the following methods:

    -   **find** ( id: `string` | `int` )
        Returns a single model if id is provided.
        Otherwise, it returns the whole collection object.

    -   **save** ( data: `object`, id: `string` | `int`, overwrite: `booelan` ) -
        Saves the data and returns the id if none was specified. Otherwise, the
        data after being saved is returned.

        If overwrite is `true`, any existing data at the id would be overwritten.

    -   **saveMany** ( data: `object` | `array`, idKey: `string`, overwrite: `boolean` )
        Saves a list data objects to the store.

        If `idKey` is not provided, `id` is assumed.

    -   **remove** ( id: `string` )
        Removes the object from the store and returns the removed object.

    -   **drop** ( )
        Removes the collection entirely from the store.

-   **tryCatch** ( tryFunc: `function`, catchFunc: `function` ) : `ThisApp`

    Calls the given function in a try...catch block and outputs any errors to the console if
    debug mode is enabled.

-   **watch** ( func: `function` ) : `ThisApp`

    The provided function is called is called when models and collections are
    rendered for the server to be watched for changes to these.

    The function receives two parameters:

    -  `object` **config** - The config representing the endpoint to be watched

        ```javascript
        {
            "type": "...", // model or collection
            "url": "...", // the url for the model or collection
            "mid": "#", // the id of the model
            "modelName": "..." // the model name
        }
        ```

    -   `function` **callback** ()
        To be called when a change occurs on the server on a watched element/endpoint.
        This will update both the DOM elements whenever they are rendered and also
        update the store.

        The function expects a single object parameter:

        ```javascript
        {
            "event": "...", // created, updated, deleted
            "id": "..", // the id of the model acted upon. This must be returned for all events
            "data": {} // the created or updated data object
        }
        ```

-   **when** ( event: `string`, selector: `string`, callback: `function` ) : `ThisApp`

    A handy shortcut to method [`on()`](#on) for [features](../../README.md#features).

    Instead of writing

    ```javascript
    on('page.loaded', '[this-type="page"][this-id="default"]', function(){})
    ```

    Write

    ```javascript
    when('page.loaded', 'page#default', function(){})
    ```

    ---
    **Note**:
    1.  This is only possible for features i.e. pages, layouts, models,
        collections and list.
    2.  It would also work for anything that has attributes `this-id` and/or `this-type`
    3.  It can also be used for blanketting features, i.e. without using id
    4.  It also accepts multiple targets separated by commas
    ---

repository: ezra-obiwale/this-js

gha_language: JavaScript

repo_name: ezra-obiwale/this-js


text:
# elfinder-2.x-spring-boot
a web image manager based on spring boot + freemarker + elfinder 2.x + jquery ui

Technologies:
1)Java Development Kit 8 ;
2)Spring Boot 1.5.22 with freemarker ;
3)Jquery-ui 1.12.1 ;
4)Elfinder  2.1.39 ;

Overview of differences and usage with servlet versions:

1.the servlet .xml file split into 2 spring-boot config files.
"ElfinderBeansConfiguration.java": define all the beans the system needs
"ElfinderConnectorServlet.java": the servlet used by the original program.

2.add a file download servlet, turn url from some kind of hash code to clear text, reduce the complexity of file references.
such like from "some-servlet/connector?cmd=file&target=images_L3Bob3RvLzE1MjUzMTM5NDg1MjkuanBn&_t=1591144312&exculsiveFolder=photo" change to "/another-servlet/images/photo/1525313948529.jpg". this allows front-end developers to assemble file URLs as they did in good old times, making it easier for them to accept the module import and operation changes

3.modified the entrance to the volume configuration. we can customize all volume initialization operations above, such as reading configuration information from properties/json/database,and here we define the volume using the configuration "VolumesProperties.java" & "volumes.properties" to define volumes of the web site.

4.if a more advanced volume configuration is required, it is recommended to refer to "StaticFsServiceFactory.java" to customize your scenario, such as setting the right path based on user profiles

5.used an advanced elfinder webjar package to solve the problem of manual downloading of elfinder files that will be named "connector",these can be referred to in the "pom.xml".

6.don't forget to put "mime.types" in the resources directory *_^

Next plan:
The implementation of "elfinder-2.x-servlet" is straightforward, so we made this spring boot demo based on this branch.
I'm going to modify all the servlet entrances to the controller, are there partners interested in this? We can learn and solve the problems we all encounter.


text:
---
date: 2019-10-09
linktitle: "release create"
title: "release create"
weight: 90120
---

Create a new release using a collection of YAML application manifests.

### Usage
```bash
replicated release create --yaml-dir YAML_DIR [flags]
```

* _`YAML_DIR` corresponds to the root directory of the YAML application manifest files._
* _Additional flags returned by `--help` that are not supported in KOTS have been omitted from the list below_

| Flag                 | Type | Description |
|:----------------------|------|-------------|
| `--yaml-dir` | path | The directory containing multiple yamls for a Kots release. (**required**) |
| `--promote` | string |    Channel name to promote this release to (**case sensitive**)|
| `--ensure-channel` |  |    When used with --promote <channel>, will create the channel if it doesn't exist |
| `--release-notes` | string |  When used with --promote <channel>, sets the **markdown** release notes |
| `--required` |  |          When used with --promote <channel>, marks this release as required during upgrades. |
| `--version` string | When used with --promote <channel>, sets the version label for the release in this channel |
| `-h, --help`   |  |          help for admin-console |

### Examples
```bash
replicated release create --yaml-dir ./manifests --promote Unstable --ensure-channel --release-notes "CI Release" --version "1.2.3"
SEQUENCE: 28
Channel cA7ZWXUjq0BkH8KIoMHZubrHrZoCulWo successfully set to release 28
```

```bash
replicated release create --yaml-dir ./manifests
SEQUENCE: 29
```

path: /content/vendor/cli/release create.md

filename: release create.md

gha_language: SCSS


text:
# OSTA Workshop Bioc2021

This repository contains materials for our [OSTA](https://lmweber.org/OSTA-book/) workshop at the [Bioc2021 conference](https://bioc2021.bioconductor.org/).

## Links

- Workshop website: https://lmweber.org/OSTAWorkshopBioc2021/
- Workshop source repository: https://github.com/lmweber/OSTAWorkshopBioc2021
- OSTA website: https://lmweber.org/OSTA-book/
- OSTA source repository: https://github.com/lmweber/OSTA-book

## Overview

Our online textbook [Orchestrating Spatially Resolved Transcriptomics Analysis with Bioconductor (OSTA)](https://lmweber.org/OSTA-book/) describes the steps in a computational analysis pipeline for spatially resolved transcriptomics (ST) data using the Bioconductor framework in R, including examples and workflows with R code and datasets.

OSTA is built around the [SpatialExperiment](https://bioconductor.org/packages/SpatialExperiment) object class and the Bioconductor principle of modularity, which allows users to easily adapt the pipeline to substitute alternative or updated methods for individual steps.

In particular, OSTA and `SpatialExperiment` are compatible with [SingleCellExperiment](https://bioconductor.org/packages/SingleCellExperiment), allowing existing methods and pipelines developed for single-cell RNA sequencing data (such as those described in [OSCA](https://bioconductor.org/books/release/OSCA/)) to be re-used and adapted to the spatial context.

## Prerequisites

- Intermediate level of R programming
- Basic familiarity with Bioconductor object classes, e.g. [SpatialExperiment](https://bioconductor.org/packages/SpatialExperiment) and/or [SingleCellExperiment](https://bioconductor.org/packages/SingleCellExperiment)
- Basic knowledge of spatially resolved transcriptomics (ST) technologies

## Installation

For this workshop, we will use R version 4.1 and a `devel` installation of Bioconductor 3.14. In addition, you will need to install the following package from GitHub:

```sh
remotes::install_github("lmweber/ggspavis", build_vignettes = TRUE)
```

Alternatively, you can use a `release` version of Bioconductor 3.13 along with the following versions of packages installed from GitHub:

```sh
remotes::install_github("drighelli/SpatialExperiment", build_vignettes = TRUE)
remotes::install_github("lmweber/STexampleData", ref = "no_accessors", build_vignettes = TRUE)
remotes::install_github("lmweber/ggspavis", build_vignettes = TRUE)
```

## Docker image

A Docker image containing a complete version of all materials used in the workshop is also available. Note this is a large download.

To run the Docker image:

```sh
docker run -e PASSWORD=abc -p 8787:8787 lmweber/ostaworkshopbioc2021:latest
```

Then navigate to http://localhost:8787/ in your browser, and log in with username `rstudio` and password `abc`.

## Acknowledgments

[OSTA](https://lmweber.org/OSTA-book/) contributors:

- Abby Spangler, *Lieber Institute for Brain Development, Baltimore, MD, USA*
- Madhavi Tippani, *Lieber Institute for Brain Development, Baltimore, MD, USA*
- Leonardo Collado-Torres, *Lieber Institute for Brain Development, Baltimore, MD, USA*
- Stephanie C. Hicks, *Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA*

[SpatialExperiment](https://bioconductor.org/packages/SpatialExperiment):

- Dario Righelli, *Department of Statistical Sciences, University of Padova, Padova, Italy*
- Helena L. Crowell, *Department of Molecular Life Sciences, University of Zurich, Zurich, Switzerland*
- Aaron Lun, *Genentech, South San Francisco, CA, USA*
- Stephanie C. Hicks, *Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA*
- Davide Risso, *Department of Statistical Sciences, University of Padova, Padova, Italy*


text:
---
layout: default
title: Media
---

Presentations

See the Monitorama 2015 (PDX) presentation for the most up to date presentation. This presentation has screencasts of building an alert starting about 15 minutes in. The first 15 minutes explain some of the design principles behind bosun. After the first screen cast of building an alert there are also some example alerts.

See the LISA presentation for slides, video and audio.
		This was the first presentation on Bosun which was given right after release.  Serves as a great introduction to the underlying thoughs on problems with alerting and how Bosun solves them.  Still highly relevant, although since then Bosun has gained a few new features, most notably support for graphite and logstash/elasticsearch.

Screenshots

repository: datto/bosun

gha_language: Go

repo_name: datto/bosun


text:
pagination
==========

Paginate record sets, not tied in directly to a database.

## Usage
1. include the class
2. instantiate a new object pass in the number of items per page and the instance identifier, this is used for the GET parameter such as ?p=2
3. pass the set_total method the total number of records
4. show the records
5. call the page_links method to create the navigation links

 ````php
 include('paginator.php');

 $pages = new Paginator('10','p');
 $pages->set_total('100'); //or a number of records

 //display the records here

 echo $pages->page_links();
 ````
 if using a database you limit the records by placing $pages->get_limit() in your query, this will limit the number of records

 ````php
 select * from table $pages->get_limit()
 ````

by default the page_links method created links starting with ? this can be changed by passing in a parameter to the method:

````php
echo $pages->page_links('&');
````

The method also allows you to pass in extra data such as a series of GET's

````php
echo $pages->page_links('?','&status='.$_GET['status'].'&active='.$_GET['active']);
````

## Database example

````php
//include the class
include('paginator.php');

//create new object pass in number of pages and identifier
$pages = new Paginator('10','p');

//get number of total records
$stmt = $db->query('SELECT id FROM table');
$total = $stmt->rowCount();

//pass number of records to
$pages->set_total($total);

$data = $db->query('SELECT * FROM table '.$pages->get_limit());
foreach($data as $row) {
    //display the records here
}

//create the page links
echo $pages->page_links();
````

## MVC example

using this class in an MVC environment its almost the same, only the database or dataset calls come from the model instead of the page directly.

in the controller:

````php

//create a new object
$pages = new Paginator('10','p');

//set the total records, calling a method to get the number of records from a model
$pages->set_total( $this->_model->get_all_count() );

//calling a method to get the records with the limit set
$data['records'] = $this->_model->get_all( $pages->get_limit() );

//create the nav menu
$data['page_links'] = $pages->page_links();

//then pass this to the view, may be different depending on the system
$this->_view->render('index', $data);
````

repository: dcblogdev/pagination

repo_name: dcblogdev/pagination


text:
# ``CRDT/GSet``

## Topics

### Creating a Set

- ``CRDT/GSet/init(actorId:clock:)``
- ``CRDT/GSet/init(actorId:clock:_:)``

### Inspecting the Set

- ``CRDT/GSet/values``
- ``CRDT/GSet/contains(_:)``
- ``CRDT/GSet/count``

### Growing the Set

- ``CRDT/GSet/insert(_:)``

### Replicating a Set

- ``CRDT/GSet/merged(with:)``
- ``CRDT/GSet/merging(with:)``

### Delta-based Replicating

- ``CRDT/GSet/state``
- ``CRDT/GSet/GSetState``
- ``CRDT/GSet/delta(_:)``
- ``CRDT/GSet/mergeDelta(_:)``
- ``CRDT/GSet/mergingDelta(_:)``
- ``CRDT/GSet/GSetDelta``

### Decoding a Set

- ``CRDT/GSet/init(from:)``

### Debugging and Optimization Methods

- ``CRDT/LWWRegister/sizeInBytes()``

repository: heckj/CRDT

branch_name: refs/heads/main

gha_language: Swift

repo_name: heckj/CRDT


text:
# was-it-used

> Spy on 3rd party modules to tell if they were really used during testing

[![NPM][was-it-used-icon] ][was-it-used-url]

[![Build status][was-it-used-ci-image] ][was-it-used-ci-url]
[![dependencies][was-it-used-dependencies-image] ][was-it-used-dependencies-url]
[![devdependencies][was-it-used-devdependencies-image] ][was-it-used-devdependencies-url]

[was-it-used-icon]: https://nodei.co/npm/was-it-used.png?downloads=true
[was-it-used-url]: https://npmjs.org/package/was-it-used
[was-it-used-ci-image]: https://travis-ci.org/bahmutov/was-it-used.png?branch=master
[was-it-used-ci-url]: https://travis-ci.org/bahmutov/was-it-used
[was-it-used-dependencies-image]: https://david-dm.org/bahmutov/was-it-used.png
[was-it-used-dependencies-url]: https://david-dm.org/bahmutov/was-it-used
[was-it-used-devdependencies-image]: https://david-dm.org/bahmutov/was-it-used/dev-status.png
[was-it-used-devdependencies-url]: https://david-dm.org/bahmutov/was-it-used#info=devDependencies

Read [Was NodeJS module used](http://bahmutov.calepin.co/was-nodejs-module-used.html).

## Example

Figure out which modules are used

```js
var modules = ['./foo', './bar'].map(function (name) {
  return path.resolve(__dirname, name);
});
var wasIt = require('was-it-used');
wasIt(modules); // init
// use modules, even via another file that calls require('foo');
var foo = require('./foo');
foo();
var bar = require('./bar');
// bar is not used!
console.table(wasIt.used); // using console.table npm module
/*
key     value
------  -----
foo.js  true
bar.js  false
*/
```

Implemented using [really-need](https://github.com/bahmutov/really-need) `require` wrapper.

### Small print

Author: Gleb Bahmutov © 2015

* [@bahmutov](https://twitter.com/bahmutov)
* [glebbahmutov.com](http://glebbahmutov.com)
* [blog](http://bahmutov.calepin.co/)

License: MIT - do anything with the code, but don't blame me if it does not work.

Spread the word: tweet, star on github, etc.

Support: if you find any problems with this module, email / tweet /
[open issue](https://github.com/bahmutov/was-it-used/issues) on Github

## MIT License

The MIT License (MIT)

Copyright (c) 2015 Gleb Bahmutov

Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software is furnished to do so,
subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

 IN NO EVENT SHALL THE AUTHORS OR
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

repository: bahmutov/was-it-used

repo_name: bahmutov/was-it-used


text:
![skedit logo](https://i.imgur.com/NlYbBSJ.png)

# skedit
A WIP, cross-platform, lightweight text editor built in Tk

# dependencies

if you want to run a raw, uncompiled instance of skedit, it depends on

* `python3`
* `tcl`
* `tk`
* `python-tk`

however, there are no dependencies to run a `.exe` of skedit or a compiled *nix script.
if you move the executable out of the build folder it comes in, it will not work.

# installation

download `skedit-nix` (on *nix) or `skedit-windows` (on Windows) from the [releases](https://github.com/smhsketch/skedit/releases) tab.

extract the .zip to wherever you want. make sure to keep the executable in the original folder it was in.

move the `skeditFiles` folder to `C:\Program Files\skedit` (Windows) or `/usr/share` (*nix).

execute the executable.

# bindings

`<control-h>` help menu

`<control-e>` exit help menu

`<control-n>` new buffer

`<control-s>` save

`<control-d>` save as

`<control-o>` open file

`<control-t>` go to top of buffer

`<control-x>` remove current line

repository: smhsketch/skedit

gha_language: Python

repo_name: smhsketch/skedit


text:
---
title: Middleware and guards
id: version-2.0.0-beta.3-middlewares
original_id: middlewares
---

Middleware are pieces of reusable code that can be easily attached to resolvers and fields. By using middleware we can extract the commonly used code from our resolvers and then declaratively attach it using a decorator or even registering it globally.

## Creating Middleware

### What is Middleware?

Middleware is a very powerful but somewhat complicated feature. Basically, middleware is a function that takes 2 arguments:

- resolver data - the same as resolvers (`root`, `args`, `context`, `info`)
- the `next` function - used to control the execution of the next middleware and the resolver to which it is attached

We may be familiar with how middleware works in [`express.js`](https://expressjs.com/en/guide/writing-middleware.html) but TypeGraphQL middleware is inspired by [`koa.js`](http://koajs.com/#application). The difference is that the `next` function returns a promise of the value of subsequent middleware and resolver execution from the stack.

This makes it easy to perform actions before or after resolver execution. So things like measuring execution time are simple to implement:

```ts
export const ResolveTime: MiddlewareFn = async ({ info }, next) => {
  const start = Date.now();
  await next();
  const resolveTime = Date.now() - start;
  console.log(`${info.parentType.name}.${info.fieldName} [${resolveTime} ms]`);
};
```

### Intercepting the execution result

Middleware also has the ability to intercept the result of a resolver's execution. It's not only able to e.g. create a log but also replace the result with a new value:

```ts
export const CompetitorInterceptor: MiddlewareFn = async (_, next) => {
  const result = await next();
  if (result === "typegql") {
    return "type-graphql";
  }
  return result;
};
```

It might not seem very useful from the perspective of this library's users but this feature was mainly introduced for plugin systems and 3rd-party library integration. Thanks to this, it's possible to e.g. wrap the returned object with a lazy-relation wrapper that automatically fetches relations from a database on demand under the hood.

### Simple Middleware

If we only want to do something before an action, like log the access to the resolver, we can just place the `return next()` statement at the end of our middleware:

```ts
const LogAccess: MiddlewareFn<TContext> = ({ context, info }, next) => {
  const username: string = context.username || "guest";
  console.log(`Logging access: ${username} -> ${info.parentType.name}.${info.fieldName}`);
  return next();
};
```

### Guards

Middleware can also break the middleware stack by not calling the `next` function. This way, the result returned from the middleware will be used instead of calling the resolver and returning it's result.

We can also throw an error in the middleware if the execution must be terminated and an error returned to the user, e.g. when resolver arguments are incorrect.

This way we can create a guard that blocks access to the resolver and prevents execution or any data return.

```ts
export const CompetitorDetector: MiddlewareFn = async ({ args }, next) => {
  if (args.frameworkName === "type-graphql") {
    return "TypeGraphQL";
  }
  if (args.frameworkName === "typegql") {
    throw new Error("Competitive framework detected!");
  }
  return next();
};
```

### Reusable Middleware

Sometimes middleware has to be configurable, just like we pass a `roles` array to the [`@Authorized()` decorator](./authorization.md). In this case, we should create a simple middleware factory - a function that takes our configuration as a parameter and returns a middleware that uses the provided value.

```ts
export function NumberInterceptor(minValue: number): MiddlewareFn {
  return async (_, next) => {
    const result = await next();
    // Hide values below minValue
    if (typeof result === "number" && result < minValue) {
      return null;
    }
    return result;
  };
}
```

Remember to call this middleware with an argument, e.g. `NumberInterceptor(3.0)`, when attaching it to a resolver!

### Error Interceptors

Middleware can also catch errors that were thrown during execution. This way, they can easily be logged and even filtered for info that can't be returned to the user:

```ts
export const ErrorInterceptor: MiddlewareFn<any> = async ({ context, info }, next) => {
  try {
    return await next();
  } catch (err) {
    // Write error to file log
    fileLog.write(err, context, info);

    // Hide errors from db like printing sql query
    if (someCondition(err)) {
      throw new Error("Unknown error occurred!");
    }

    // Rethrow the error
    throw err;
  }
};
```

### Class-based Middleware

Sometimes our middleware logic can be a bit complicated - it may communicate with a database, write logs to file, etc., so we might want to test it. In that case we create class middleware that is able to benefit from [dependency injection](./dependency-injection.md) and easily mock a file logger or a database repository.

To accomplish this, we implement a `MiddlewareInterface`. Our class must have the `use` method that conforms with the `MiddlewareFn` signature. Below we can see how the previously defined `LogAccess` middleware looks after the transformation:

```ts
export class LogAccess implements MiddlewareInterface<TContext> {
  constructor(private readonly logger: Logger) {}

  async use({ context, info }: ResolverData<TContext>, next: NextFn) {
    const username: string = context.username || "guest";
    this.logger.log(`Logging access: ${username} -> ${info.parentType.name}.${info.fieldName}`);
    return next();
  }
}
```

## How to use

### Attaching Middleware

To attach middleware to a resolver, place the `@UseMiddleware()` decorator above the field or resolver declaration. It accepts an array of middleware that will be called in the provided order. We can also pass them without an array as it supports [rest parameters](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/rest_parameters):

```ts
@Resolver()
export class RecipeResolver {
  @Query()
  @UseMiddleware(ResolveTime, LogAccess)
  randomValue(): number {
    return Math.random();
  }
}
```

We can also attach the middleware to the `ObjectType` fields, the same way as with the [`@Authorized()` decorator](./authorization.md).

```ts
@ObjectType()
export class Recipe {
  @Field()
  title: string;

  @Field(type => [Int])
  @UseMiddleware(LogAccess)
  ratings: number[];
}
```

### Global Middleware

However, for common middleware like measuring resolve time or catching errors, it might be annoying to place a `@UseMiddleware(ResolveTime)` decorator on every field/resolver.

Hence, in TypeGraphQL we can also register a global middleware that will be called for each query, mutation, subscription and field resolver. For this, we use the `globalMiddlewares` property of the `buildSchema` configuration object:

```ts
const schema = await buildSchema({
  resolvers: [RecipeResolver],
  globalMiddlewares: [ErrorInterceptor, ResolveTime],
});
```

### Custom Decorators

If we want to use middlewares with a more descriptive and declarative API, we can also create a custom method decorators. See how to do this in [custom decorators docs](./custom-decorators.md#method-decorators).

## Example

See how different kinds of middlewares work in the [middlewares and custom decorators example](https://github.com/MichalLytek/type-graphql/tree/v2.0.0-beta.3/examples/middlewares-custom-decorators).

repository: MichalLytek/type-graphql

gha_language: TypeScript

repo_name: MichalLytek/type-graphql


text:
zsh-syntax-highlighting / highlighters / pattern
================================================

This is the ***pattern*** highlighter, that highlights user defined patterns.

How to activate it
------------------
To activate it, add it to `ZSH_HIGHLIGHT_HIGHLIGHTERS`:

    ZSH_HIGHLIGHT_HIGHLIGHTERS=( [...] pattern)

How to tweak it
---------------
To use this highlighter, associate patterns with styles in the `ZSH_HIGHLIGHT_PATTERNS` array, for example in `~/.zshrc`:

    # To have commands starting with `rm -rf` in red:
    ZSH_HIGHLIGHT_PATTERNS+=('rm -rf *' 'fg=white,bold,bg=red')

The syntax for declaring styles is [documented here](http://zsh.sourceforge.net/Doc/Release/Zsh-Line-Editor.html#SEC135).

repository: luxe/unilang

gha_language: Fancy

repo_name: luxe/unilang


text:
# robots

[![Build Status](https://github.com/AntoineGagne/robots/actions/workflows/erlang.yml/badge.svg)](https://github.com/AntoineGagne/robots/actions)
[![Hex Pm](http://img.shields.io/hexpm/v/robots.svg?style=flat)](https://hex.pm/packages/robots)
[![Docs](https://img.shields.io/badge/hex-docs-green.svg?style=flat)](https://hexdocs.pm/robots)
[![Releases](https://img.shields.io/github/release/AntoineGagne/robots?color=brightgreen)](https://github.com/AntoineGagne/robots/releases)
[![Coverage](https://coveralls.io/repos/github/AntoineGagne/robots/badge.svg?branch=master)](https://coveralls.io/github/AntoineGagne/robots?branch=master)

A library that parses and validates rules from `robots.txt`.

## Installation

This library is available on [hex.pm](https://hex.pm/packages/robots).
To install this library, simply add the following lines to your
`rebar.config`:

```erlang
{robots, "1.1.1"}
```

## Usage

```erlang
Content = <<"User-Agent: bot\nAllow: /fish">>,
%% This will return an opaque type that contains all the rules and their agents
{ok, RulesIndex} = robots:parse(Content, 200),
true = robots:is_allowed(<<"bot/1.0.0">>, <<"/fish/salmon.html">>, RulesIndex),
true = robots:is_allowed(<<"bot/1.0.0">>, <<"/Fish.asp">>, RulesIndex),
```

## Development

### Running all the tests and linters

You can run all the tests and linters with the `rebar3` alias:

```sh
rebar3 check
```

repository: AntoineGagne/robots

gha_language: Erlang

gha_license_id: NOASSERTION

repo_name: AntoineGagne/robots


text:
# ffs

Python tools for querying a Flexible File Structure as specified in <https://gitlab.com/cardonazlaticlabs/data-policy>

The name of the PyPI package is `flexfs` to avoid naming collisions with the unrelated [`ffs`](https://pypi.org/project/ffs/) project.
The import and CLI name is `ffs`.

## Usage

The entry point is a command line tool called `ffs` with a number of subcommands:

```_main
Usage: ffs [OPTIONS] COMMAND [ARGS]...

  Command line tool for working with a Flexible File Structure.

Options:
  --version      Show the version and exit.
  -v, --verbose  Increase logging verbosity.
  --help         Show this message and exit.

Commands:
  book      Export the FFS metadata into files for mdbook.
  create    Create a new FFS entry.
  export    Read the FFS and its metadata into JSON.
  problems  List problems with the structure of the FFS.
```

### Create

```_create
Usage: ffs create [OPTIONS] NAME [DIRECTORY]

  Create a new FFS entry.

Options:
  -d, --description TEXT          Description of the entry (interactive if not
                                  given)
  -r, --responsible PARSEADDR     Email address of person responsible for this
                                  entry. Several may be given; interactive if
                                  none given.
  -D, --date-resolution [year|y|month|m|day|d]
                                  Smallest unit to show in date prefix of
                                  entry name (default month).
  -t, --today FROMISOFORMAT       ISO-8601 date (YYYY-MM-DD) to assign to
                                  entry (default today)
  -l, --leaf                      Ignore all subdirectories
  -R, --root                      Do not check that parent is an entry which
                                  does not ignore the new entry.
  --help                          Show this message and exit.
```

### Export

```_export
Usage: ffs export [OPTIONS] [ROOT]

  Read the FFS and its metadata into JSON.

Options:
  -s, --sort               Whether to sort keys in outupt.
  -s, --indent INTEGER     Indentation of output: none by default, 0 for
                           newlines, a positive number N for N spaces, a
                           negative number -N for N tabs.
  -l, --flatlines          Un-nest the entries and print one object per line.
                           The 'children' attribute is replaced by an array of
                           string names, and the 'name' attribute now includes
                           the entry's ancestors (/-separated). '--indent'
                           option is ignored.
  -r, --recursion INTEGER  Depth to recurse into entries; negative (default)
                           for infinite. Directories which are not valid
                           entries are not explored.
  --help                   Show this message and exit.
```

### Book

```_book
Usage: ffs book [OPTIONS] [ROOT] TARGET

  Export the FFS metadata into files for mdbook.

Options:
  -t, --title TEXT         Title for generated book, default
                           '{FQDN}:{ROOT_REAL_PATH}'.
  -r, --recursion INTEGER  Depth to recurse into entries; negative (default)
                           for infinite. Directories which are not valid
                           entries are not explored.
  --help                   Show this message and exit.
```

### Problems

```_problems
Usage: ffs problems [OPTIONS] [ROOT]

  List problems with the structure of the FFS.

  Prints a TSV with columns:

  1- comma-separated individuals responsible for entry (or parent if unknown);
  2- path of problem entry, relative to given root; 3- description of the
  problem

Options:
  -c, --check          Exit with an error code at the first problem
  -s, --skip-problems  Do not attempt to traverse below directories with
                       malformed metadata
  --help               Show this message and exit.
```

## Development

A number of `make` recipes are included for convenience of regular development tasks.
In particular, see `make {install-dev,update-spec,fmt,lint,test,readme,book}`.

repository: clbarnes/ffs

repo_name: clbarnes/ffs


text:
# Makefile example

Makefile is a powerful tool to manage your containers and services easier.

It is good to make shortcuts with commands like docker-compose and running container scripts like migrations, provision, configuration.

For example, with Makefile in this folder, you can do the next:

```shell
make up
make ps
make stop
make restart
make bash
```

Or even combine few directives from the command line, this will run your containers, show the list of them and get you inside of your app:

```shell
make up ps bash
```

Please notice that make is not installed as default, try this for ubuntu:

```shell
apt-get update && apt-get install build-essential
```

repository: abagayev/docker-bootstrap-collection

repo_name: abagayev/docker-bootstrap-collection