text:
---
layout: post
title: "NLP Newsletter #6 [EN]: BERTology Primer, fastpages, T5, Data Science Education, PyTorch Notebooks, Slow Science in ML"
author: billy_rick
modified:
comments: true
excerpt: "This issue covers topics that range from extending the Transformer model to slowing publication in ML to a series of ML and NLP books and project releases."
tags: [nlp_newsletter]
image:
  thumb: nlp_newsletter_6.png
---

![](https://cdn-images-1.medium.com/max/1200/1*vWICxAehSy3xOnqGIXtpoQ.png)

\\
Welcome to the sixth issue of the NLP Newsletter. Thanks for all the support and for taking the time to read through the latest in ML and NLP. This issue covers topics that range from extending the Transformer model to slowing publication in ML to a series of ML and NLP book and project launches.

\\
***A few updates about the NLP Newsletter and dair.ai***

\\
We have been translating the newsletter to other languages such as Brazilian Portuguese, Chinese, Arabic, Spanish, among others. Thanks to those folks that have helped with the translations 🤗. You can also contribute [here](https://github.com/dair-ai/dair-ai.github.io/issues/11).

\\
A month ago, we officially launched our new [website](https://dair.ai/). You can look at our [GitHub organization](https://github.com/dair-ai) for more information about dair.ai and projects. If you are interested in seeing how others are already contributing to dair.ai or are interested in contributing to democratizing artificial intelligence research, education, and technologies, check our [issues](https://github.com/dair-ai/dair-ai.github.io/issues) section.

\\
[*Subscribe*](https://dair.ai/newsletter/) *🔖 to the NLP Newsletter to receive future issues in your inbox.*

# Publications 📙

***A Primer in BERTology: What we know about how BERT works***

\\
Transformer-based models have shown to be effective at approaching different types of NLP tasks that range from *sequence labeling* to *question answering*. One of those models called BERT [(Devlin et al. 2019)](https://arxiv.org/abs/1810.04805) is widely used but, like other models that employ deep neural networks, we know very little about their inner workings. A new [paper](https://arxiv.org/abs/2002.12327) titled “**A Primer in BERTology: What we know about how BERT works**” aims to answer some of the questions about why BERT performs well on so many NLP tasks. Some of the topics addressed in the paper include the type of knowledge learned by BERT and where it is represented, and how that knowledge is learned and other methods researchers are using to improve it.

\\
***Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer***

\\
Google AI recently published a [method](https://arxiv.org/abs/1910.10683) that brings together all the lessons learned and improvements from NLP transfer learning models into one unified framework called Text-to-Text Transfer Transformer (T5). This work proposes that most NLP tasks can be formulated in a text-to-text format, suggesting that both the inputs and outputs are texts. The authors claim that this “*framework provides a consistent training objective both for pre-training and fine-tuning*”. T5 is essentially an encoder-decoder Transformer that applies various improvements in particular to the attention components of the model. The model was pre-trained on a newly released dataset called [Colossal Clean Crawled Corpus](https://www.tensorflow.org/datasets/catalog/c4) and achieved SOTA results on NLP tasks such as summarization, question answering, and text classification.

\\
![](https://cdn-images-1.medium.com/max/800/1*T9MXxcDOd2fX6xblbu7VdQ.png)

[*(Raffel et al. 2020)*](https://arxiv.org/abs/1910.10683)

\\
***12-in-1: Multi-Task Vision and Language Representation Learning***

\\
Current research uses independent tasks and datasets to perform vision-and-language research even when the “*visually-grounded language understanding skill*s” required to perform these tasks overlap. A new [paper](https://arxiv.org/abs/1912.02315) (to be presented at CVPR) proposes a large-scale multi-task approach to better model and jointly train vision-and-language tasks to generate a more generic vision-and-language model. The model reduces the parameter size and performs well on tasks like caption-based image retrieval and visual question answering.

\\
![](https://cdn-images-1.medium.com/max/800/1*yyvN4bK0K2iykyJ2-QVBjw.png)

[*(Lu et al. 2020)*](https://arxiv.org/abs/1912.02315)

\\
***BERT Can See Out of the Box: On the Cross-modal Transferability of Text Representations***

\\
reciTAL researchers and collaborators published a paper that aims to answer the question of whether a BERT model can produce representations that generalize to other modalities beyond text such as vision. They propose a model called BERT-gen that leverages mono or multi-modal representations and achieve improved results on visual question generation datasets.

\\
![](https://cdn-images-1.medium.com/max/800/1*2NgR7yBuVLDcEza9UT41dw.png)

[*(Scialom et al. 2020)*](https://arxiv.org/abs/2002.10832)

# Creativity and Society 🎨

***The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence***

\\
Gary Marcus recently published a [paper](https://arxiv.org/abs/2002.06177) where he explains a series of steps that, in his view, we should be taking to build more robust AI systems. Gary’s central idea in this paper is to focus on building hybrid and knowledge-driven systems guided by cognitive models as opposed to focusing on building larger systems that require more data and computation power.

\\
***10 Breakthrough Technologies 2020***

\\
MIT Technology Review published a list of the [10 breakthroughs](https://www.technologyreview.com/lists/technologies/2020/) they have identified that will make a difference in solving problems that could change the way we live and work. The list — in no particular order — includes unhackable internet, hyper-personalized medicine, digital money, anti-aging drugs, AI-discovered molecules, satellite mega-constellations, quantum supremacy, Tiny AI, differential privacy, and climate attribution.

\\
***Time to rethink the publication process in machine learning***

\\
Yoshua Bengio recently [wrote](https://yoshuabengio.org/2020/02/26/time-to-rethink-the-publication-process-in-machine-learning/) on his concerns about the fast-paced cycles of ML publications. The main concern is that due to the velocity of publishing, a lot of papers get published that contain errors and are just incremental, whereas spending more time and ensuring rigour, which is how it used to work many years ago, seems to be vanishing. On top of it all, students are the ones that have to deal with the negative consequences of this pressure and stress. To address the situation, Bengio talks about his actions to help in the process of slowing down research publications for the good of science.

# Tools and Datasets ⚙

***PointerGenerator network implementation in AllenNLP***

\\
Pointer-Generator networks aim to augment sequence-to-sequence attentional models that are used to [improve](https://arxiv.org/abs/1704.04368) [*abstractive summarization*](https://arxiv.org/abs/1704.04368). If you wish to use this technique for abstractive summarization using AllenNLP, Kundan Krishna has developed a [library](https://github.com/kukrishna/pointer-generator-pytorch-allennlp) that allows you to run a pretrained model (provided) or train your own model.

\\
![](https://cdn-images-1.medium.com/max/800/1*Fa4G6BrnJm3NSDr3TDHhfw.jpeg)

\\
***Question answering for different languages***

\\
With the proliferation of Transformer models and their effectiveness for large-scale NLP tasks performed in other languages, there has been an impressive amount of effort to release different types of datasets in different languages. For instance, Sebastian Ruder [shared](https://twitter.com/seb_ruder/status/1231713840502657025?s=20) a list of datasets that can be used for question answering research in different languages: [DuReader](https://www.aclweb.org/anthology/W18-2605/), [KorQuAD](https://arxiv.org/abs/1909.07005), [SberQuAD](https://arxiv.org/abs/1912.09723), [FQuAD](https://arxiv.org/abs/2002.06071), [Arabic-SQuAD](https://arxiv.org/abs/1906.05394), [SQuAD-it](https://github.com/crux82/squad-it), and [Spanish SQuAD](https://arxiv.org/abs/1912.05200v2).

\\
***PyTorch Lightning***

\\
PyTorch Lightning is a [tool](https://towardsdatascience.com/from-pytorch-to-pytorch-lightning-a-gentle-introduction-b371b7caaf09) that allows you to abstract training that could require setting up GPU/TPU training and the use of 16-bit precision. Getting those things to work can become tedious but the great news is that PyTorch Lightning simplifies this process and allows you to train models on multi GPUs and TPUs without the need to change your current PyTorch code.

\\
***Graph Neural Networks in TF2***

\\
A Microsoft Research team releases a [library](https://github.com/microsoft/tf2-gnn) that provides access to implementations of many different graph neural network (GNN) architectures. This library is based on TensorFlow 2 and also provides data-wrangling modules that can directly be used in training/evaluation loops.

\\
***Pre-training SmallBERTa — A tiny model to train on a tiny dataset***

\\
Have you ever wanted to train your own language model from scratch but didn’t have enough resources to do so? If so, then Aditya Malte have you covered with this great [Colab notebook](https://gist.github.com/aditya-malte/2d4f896f471be9c38eb4d723a710768b#file-smallberta_pretraining-ipynb) that teaches you how to train an LM from scratch with a smaller dataset.

# Ethics in AI 🚨

***Why faces don’t always tell the truth about feelings***

\\
For some time now, many researchers and companies have attempted to build AI models that understand and can recognize emotions either in the textual or visual context. A new [article](https://www.nature.com/articles/d41586-020-00507-5) reopens the debate that AI techniques that aim to recognize emotion directly from face images are not doing it right. The main argument, raised by prominent psychologists in the space, is that there is no evidence of universal expressions that can be used for emotion detection from face images alone. It would take a model better understanding of personality traits, body movement, among other things to really get closer to more accurately detecting the emotions displayed by humans.

\\
***Differential Privacy and Federated Learning Explained***

\\
One of the ethical considerations when building AI systems is to ensure privacy. Currently, this can be achieved in two ways, either using differential privacy or federated learning. If you want to know more about these topics, Jordan Harrod provides us a great introduction in this [video](https://www.youtube.com/watch?v=MOcTGM_UteM) which also includes a hands-on practice session with the use of a Colab notebook.

# Articles and Blog posts ✍

***A Deep Dive into the Reformer***

\\
Madison May wrote a [new blog post](https://www.pragmatic.ml/reformer-deep-dive/) that provides a deep dive into [Reformer](https://ai.googleblog.com/2020/01/reformer-efficient-transformer.html), which is a new and improved Transformer-based model recently proposed by Google AI. We also featured Reformer in a [previous issue](https://medium.com/dair-ai/nlp-newsletter-reformer-deepmath-electra-tinybert-for-search-vizseq-open-sourcing-ml-68d5b6eed057) of the newsletter.

\\
***A free blogging platform***

\\
[fastpages](https://fastpages.fast.ai/fastpages/jupyter/2020/02/21/introducing-fastpages.html) allows you to automatically set up a blog using GitHub pages for free. This solution simplifies the process of publishing a blog and it also supports the use of exported word documents and Jupyter notebooks.

\\
***Tips for interviewing at Google***

\\
Pablo Castro, from the Google Brain team, published an [excellent blog post](https://psc-g.github.io/interviews/google/2020/02/25/interviewing-at-google.html) highlighting a list of tips for those interested in interviewing for a job at Google. Topics include advice on how to prepare for the interview, what to expect during the interview, and what happens after the interview.

\\
***Transformers are Graph Neural Networks***

\\
Both graph neural networks (GNNs) and Transformers have shown to be effective at different NLP tasks. To better understand the inner workings behind these approaches and how they relate, Chaitanya Joshi wrote a great [article](https://graphdeeplearning.github.io/post/transformers-are-gnns/) explaining the connection between GNNs and Transformers and different ways these methods can be combined in a sort of hybrid model.

\\
![](https://cdn-images-1.medium.com/max/800/0*u-BkejfKSKcnWOBx.jpg)

*Representing a sentence as a fully-connected word graph —* [*source*](https://graphdeeplearning.github.io/post/transformers-are-gnns/)

\\
***CNNs and Equivariance***

\\
Fabian Fuchs and Ed Wagstaff [discuss](https://fabianfuchsml.github.io/equivariance1of2/) the importance of equivariance and how CNNs enforce it. The concept of equivariance is first defined and then discussed in the context of CNNs with respect to translation.

\\
***Self-supervised learning with images***

\\
Self-supervised has been discussed a lot in previous issues of the NLP Newsletter due to the role it has played in modern techniques for language modeling. This [blog post](https://datasciencecastnet.home.blog/2020/02/22/self-supervised-learning-with-image%e7%bd%91/) by Jonathan Whitaker provides a nice and intuitive explanation of self-supervision in the context of images. If you are really interested in the topic, Amit Chaudhary also wrote an excellent [blog post](https://amitness.com/2020/02/illustrated-self-supervised-learning/) describing the concept in a visual way.

# Education 🎓

***Stanford CS330: Deep Multi-Task and Meta-Learning***

\\
Stanford recently released video recordings, in the form of a YouTube playlist, for their new [course on deep multi-task and meta-learning](https://www.youtube.com/playlist?list=PLoROMvodv4rMC6zfYmnD7UG3LVvwaITY5). Topics include bayesian meta-learning, lifelong learning, a reinforcement learning primer, model-based reinforcement learning, among others.

\\
***PyTorch Notebooks***

\\
dair.ai releases a [series of notebooks](https://github.com/dair-ai/pytorch_notebooks) that aim to get you started with deep neural networks using PyTorch. This is a work in progress and some current topics include how to implement a logistic regression model from scratch and how to program a neural network or recurrent neural network from scratch. Colab notebooks are also available in the GitHub repository.

\\
***The fastai book (draft)***

\\
Jeremy Howard and Sylvain Gugger release a [comprehensive list](https://github.com/fastai/fastbook) of draft notebooks for an upcoming course that introduces deep learning concepts and how to develop different methods using PyTorch and the fastai library.

\\
***Free Data Science courses***

\\
In case you missed it, Kaggle provides a series of [free micro-courses](https://www.kaggle.com/learn/overview) to get you started with your Data Science journey. Some of these courses include machine learning explainability, an intro to machine learning and Python, data visualization, feature engineering, and deep learning, among others.

\\
Here is another excellent [online data science course](https://lewtun.github.io/dslectures/) that provides a syllabus, slides, and notebooks on topics that range from exploratory data analysis to model interpretation to natural language processing.

\\
***8 Creators and Core Contributors Talk About Their Model Training Libraries From PyTorch Ecosystem***

\\
nepture.ai published an [extensive article](https://neptune.ai/blog/model-training-libraries-pytorch-ecosystem?utm_source=twitter&utm_medium=tweet&utm_campaign=blog-model-training-libraries-pytorch-ecosystem) that contains detailed discussions with core creators and contributors about their journey and philosophy of building PyTorch and tools around it.

\\
***Visualizing Adaptive Sparse Attention Models***

\\
Sasha Rush shares an impressive [Colab notebook](http://Visualizing%20Adaptive%20Sparse%20Attention%20Models) that explains and shows the technical details of how to produce sparse softmax outputs and induce sparsity into the attention component of a Transformer model which helps to produce zero probability for irrelevant words in a given context, improving performance and interpretability all at once.

\\
![](https://cdn-images-1.medium.com/max/800/1*7BB322LlVgt1zzk-cviSoA.png)

*Visualizing probability distribution of a softmax output*

# Noteworthy Mentions ⭐

You can access the previous issue of the 🗞 NLP Newsletter [here](https://medium.com/dair-ai/nlp-newsletter-the-annotated-gpt-2-understanding-self-distillation-haiku-ganilla-sparkwiki-b0f47f595c82).

\\
Conor Bell wrote this nice [python script](https://gist.github.com/connorbell/9269401d127f1e507cc9aaf2803067c4) that allows you to view and prepare a dataset that can be used for a StyleGAN model.

\\
Manu Romero [contributes](https://github.com/huggingface/transformers/tree/master/model_cards/mrm8488/bert-spanish-cased-finetuned-pos) a fine-tuned POS model for Spanish. The model is available for use in the Hugging Face Transformer library. It will be interesting to see this effort in other languages.

\\
This [repo](https://github.com/tomohideshibata/BERT-related-papers) contains a long list of carefully curated BERT-related papers that approach different problems such as model compression, domain-specific, multi-model, generation, downstream tasks, etc.

\\
Connor Shorten published a short [15-minute video](https://www.youtube.com/watch?time_continue=79&v=-Bh_7tzyoR4&feature=emb_logo) explaining a new general framework that aims to reduce the effect of “shortcut” features in self-supervised representation learning. This is important because if not done right, the model can fail to learn useful semantic representations and potentially prove ineffective in a transfer learning setting.

\\
Sebastian Ruder published a new issue of the NLP News newsletter that highlights topics and resources that range from an analysis of NLP and ML papers in 2019 to slides for learning about transfer learning and deep learning essentials. Check it out [here](http://newsletter.ruder.io/issues/accelerating-science-memorizing-vs-learning-to-look-things-up-schmidhuber-s-2010s-greek-bert-arc-illustrated-reformer-annotated-gpt-2-olmpics-223195).

\\
[*Subscribe*](https://dair.ai/newsletter/) *🔖 to the NLP Newsletter to receive future issues in your inbox.*


text:
# rbac
a Role and Resource Based Access Control implementation in golang

![ci](https://github.com/jacexh/rbac/workflows/ci/badge.svg)
[![codecov](https://codecov.io/gh/jacexh/rbac/branch/master/graph/badge.svg)](https://codecov.io/gh/jacexh/rbac)
[![PkgGoDev](https://pkg.go.dev/badge/github.com/jacexh/rbac)](https://pkg.go.dev/github.com/jacexh/rbac)
[![Go Report Card](https://goreportcard.com/badge/github.com/jacexh/rbac)](https://goreportcard.com/report/github.com/jacexh/rbac)

## Installation

```
go get github.com/jacexh/rbac
```

## Example

```go
rbac := NewRBAC()
qa := NewSimpleRole("qa")
deployment := NewSimpleResource("deployment")
qa.Grant(deployment, PermissionGet, PermissionUpdate)
rbac.RegisterRole(qa)
rbac.Permit(qa, deployment.ID(), PermissionGet)
```

repository: jacexh/rbac

branch_name: refs/heads/master

extension: md

license_type: permissive

repo_name: jacexh/rbac


text:
# face-detection-node-opencv

Forked from [drejkim](https://github.com/drejkim/face-detection-node-opencv)

Real-time face detection using OpenCV, Node.js, WebSockets and ngrok.

Click [here](http://youtu.be/v2SY0naPBFw) to see it in action.

## Requirements

* [Node.js](http://nodejs.org/)
* [OpenCV 3.4.0](http://opencv.org/)
    * May also work with older versions of OpenCV, but most recently tested with OpenCV 3.4.0
* A webcam, e.g. laptop-integrated webcam, USB webcam

## Installing Node.js packages

* Navigate to the `project_root` directory
* To install the packages: `npm install`

## Running the demo

* Make sure you are still in the `project_root` directory
* To run the server: `node server.js` then you will get the `ngrokURL` in terminal
* To run the demo, open a browser and go to `ngrokURL`

The app should be up and running!

## References
* http://stackoverflow.com/questions/24107378/socket-io-began-to-support-binary-stream-from-1-0-is-there-a-complete-example-e/24124966#24124966Ｖ
* https://gist.github.com/jonleighton/958841
* https://stackoverflow.com/questions/9267899/arraybuffer-to-base64-encoded-string/9458996#9458996ＶＶ

license: MIT

repository: bookjan/face-detection-node-opencv

repo_name: bookjan/face-detection-node-opencv


text:
# txtcalc

Code repository for the [txtcalc experiment](https://txtcalc.brendanm.dev)

Built using Preact

# Installation

```bash
$ npm install
```

# Development

```bash
$ npm run dev
```

# Deployment

Prod build

```bash
$ npm run build
```

Prod server

```bash
$ npm run serve
```

repository: brendanatme/txtcalc

branch_name: refs/heads/main

repo_name: brendanatme/txtcalc


text:
---
title: Proportion of people that feel safe walking alone around the area they live
permalink: /16-1-4/
sdg_goal: 16
layout: indicator
indicator: 16.1.4
indicator_variable: null
graph: null
variable_description: null
variable_notes: null
target_id: '16.1'
has_metadata: false
goal_meta_link: 'http://unstats.un.org/sdgs/files/metadata-compilation/Metadata-Goal-16.pdf'
goal_meta_link_page: 10
indicator_name: Proportion of people that feel safe walking alone around the area they live
target: Significantly reduce all forms of violence and related death rates everywhere.
source_title: null
source_notes: null
published: true
indicator_definition: >-
  Adult U.S. residents who answer "yes" to the following: Is there any area near
  where you live - that is,within a mile - where you would be afraid to walk
  alone at night?
method_of_computation: Percent of total adult U.S. resident population
periodicity: 'Annual survey, conducted by Gallup, in October of each year'
time_period: Snapshot at time of interview
unit_of_measure: Randomly selected adult
disaggregation_categories: None
disaggregation_geography: None
date_of_national_source_publication: November 2015
date_metadata_updated: December 2016
source_agency_staff_name: 'Gallup Inc., Jim Norman (provided by the Bureau of Justice Statistics)'
source_agency_staff_email<EMAIL_ADDRESS>international_and_national_references: >-
  (for 2015 data)
  http://www.gallup.com/poll/186563/women-poor-urbanitesnot-feel-safe-walking-night-near-home.aspx
source_url: >-
  (for 2007-2014 data)
  http://www.gallup.com/poll/179558/not-feel-safe-walking-night-alone.aspx
comments_and_limitations: >-
  These results come from Gallup’s annual Crime survey, conducted annually in
  October. The results are based on telephone interviews with a random sample of
  adults, age 18 +, living in the U.S. The interviews are conducted with
  respondents on landline telephones and cell phones with a minimum quota of 50%
  cell phone and 50% landlines respondents, with additional quota by region.
  Phone numbers are selected using random digit dial methods. Landline
  respondents are chosen at random within each household based on the adult with
  the most recent birthday. Samples are weighted to correct for unequal
  probability of selection, non-response, and double coverage of landline and
  cell users in two sampling frames. All estimates are subject to sampling
  error. Each year, the margin of error is approximately +/- 4 percentage points
  at the 95% confidence level. (For additional detail, see
  http://www.gallup.com/poll/179558/not-safe-walking-night-near-home.

  The National Crime Victimization Survey, conducted by the U.S. Census Bureau
  for the Bureau of Justice Statistics, does not collect this information;
  however, as part of methodological work for the NCVS redesign, the BJS is
  conducting the American Crime Survey. The survey includes an item similar to
  that used by Gallup: Is there any place within a mile of your home where you
  would be afraid to walk alone at night? [ ] Yes   [ ] No. The companion survey
  is a mail survey, self-administered (with a split sample of about 200,000 to
  determine appropriate placement). For more detail, see the OMB clearance
  package:
  https://www.reginfo.gov/public/do/PRAViewDocument?ref_nbr=201502-1121-001. The
  field test is nearing completion, with a report expected in the fall of 2017.
---
##

##

repository: mwdumas/sdg-indicators

branch_name: refs/heads/gh-pages

repo_name: mwdumas/sdg-indicators


text:
#LL1 and SLR parser

See: http://eat-sleep-code.blogspot.in/p/my-coding-projects.html

MIT License, Apoorv Kumar, 2012

	We created a LL1 parser and SLR parser both able to take a grammar as input , parse it and then generate tables associated with it. Also we created a code checker/generator that took care of missing declarations and warned user about invalid data usage. and The first step was creating a tokenizer.

##Steps
	Supplying the code
	And the list of keywords , delimiters and
	operators.
	Our tokenizer prints the list of tokens and generates the tokenized code

	Now we created the parser that parsed the grammar depending upon its type.
	The grammars (set of productions) look something like

	S_dash # E
	E # T D
	D # + T D : epsilon
	T # F G
	G # * F G : epsilon
	F # ( E ) : id

	Then , the tool takes a code as input. After preprocessing is done, it parses the code and checks whether it is a valid code and outputs the LL1 table.

	Finally we created a java based code generator that took complex code and broke it down to basic (one left operand , 2 right operand) form that can be easily converted to assembly   .

	We give this machine dependent code to the code generator and it parses , fixes , issues errors and generates the machine independent
	code along with the token table

repository: apoorv-kumar/compilers

repo_name: apoorv-kumar/compilers


text:
## AWS Lambda Edge Workshops

This repository contains instructions and blueprints for Lambda@Edge workshops

### [Workshop - Customizing Content Delivery with Lambda@Edge](./Workshop1/README.md)

## License

This library is licensed under the Apache 2.0 License.


text:
# Kubebuilder v2 vs v3

This document covers all breaking changes when migrating from v2 to v3.

The details of all changes (breaking or otherwise) can be found in [kubebuilder](https://github.com/kubernetes-sigs/kubebuilder/releases) release notes.

## Kubebuilder

- A plugin design was introduced to the project. For more info see the [Extensible CLI and Scaffolding Plugins][plugins-phase1-design-doc]
- The GO supported version was upgraded from 1.13+ to 1.15+

## Project config versions

- [v3][project-v3]

## `go.kubebuilder.io` plugin versions

- [v3][plugin-v3]

[plugins-phase1-design-doc]: https://github.com/kubernetes-sigs/kubebuilder/blob/master/designs/extensible-cli-and-scaffolding-plugins-phase-1.md
[project-v3]:/migration/project/v2_v3.md
[plugin-v3]:/migration/plugin/v2_v3.md

repository: shizhaoquan/kubebuilder

repo_name: shizhaoquan/kubebuilder


text:
# Biowulf and High-Performance Computing at NIH

This collates resources for using Biowulf and HPC

- [Biowulf Tutorials][biowulf_tutorials]
- [Biowulf User Guide][biowulf_guide]
- [Automount Helix Drives](#automount-helix-drives)
- [Useful Modules](#Useful-Modules)
- [VSCode and Biowulf](#VSCode-and-Biowulf)

## Automount Helix Drives

These are instructions for setting Helix drives to automount on MacOS.
In order to automatically mount helix drives, you should create a script
using Script Editor called `HelixAutoMount.scpt` that looks like this:

```scpt
tell application "Finder"
    mount volume "smb://helixdrive.nih.gov/SFIM_100RUNS" as user name "USERNAME"
    mount volume "smb://helixdrive.nih.gov/NIMH_SFIM" as user name "USERNAME"
    mount volume "smb://helixdrive.nih.gov/USERNAME" as user name "USERNAME"
    mount volume "smb://helixdrive.nih.gov/SFIMLBC" as user name "USERNAME"
    mount volume "smb://helixdrive.nih.gov/SFIM" as user name "USERNAME"
    mount volume "smb://helixdrive.nih.gov/data" as user name "USERNAME"
end tell
```

with `USERNAME` your NIH username.
The `USERNAME` address will mount to `/home/USERNAME` on biowulf.
The `/data` address will mount to `/data/USERNAME`.
All others will go to `/data/DIRNAME` with DIRNAME the directory name.
Then export it as an application in Script Editor.
When you run it, it will mount the directories.
You should be able to access them in MacOS under `/Volumes/` in Terminal.
If you run

```bash
sudo ln -s /Volumes /data
```

then you can share scripts across MacOS and biowulf which rely on data
living in `/data/` rather than `/Volumes/`.
Follow additional instructions [here][helix_mount] for MacOS in order to
tune mount performance.

## Useful Modules

Biowulf has a [module system][module_system] that is sometimes useful for loading common programs that aren't loaded by default. You can use it to load either the newest version on biowulf or an older version for a range of programs. The [linked guide][module_system] has more information.  These might be out-of-date, but if you want to install a newer version there so that others can benefit, just ask around before updating.

## VSCode and Biowulf

VSCode has some nice features for using with Biowulf. Noteably, with the remote development extension, you can open a local copy of VSCode to save and edit scripts with no network lag, but, have scripts and command executions happen on a biowful node. The big advantage of this approach is that, unlike opening a remote script in a local script editor, the actual VSCode system is running on biowulf so, if you get disconnected, you can simply reconnect and everything is still there.

There are a few quirks with this setup. VScode will recognize your self-defined PATH, but not any calls to `module load` That means, unless you hard-code the path to a newer git version, it will use the default, very old git version. To correct this, add `export PATH=/usr/local/git/2.31.1/bin:$PATH` to your `~/.profile` on biowful. Note: 2.31.1 is the most recent git version, when this was written. You can `ls /usr/local/git` to see the current newest version and use that number instead.

[Here is a guide][biowulf_vscode] to connecting directly to an interactive or persistent node on biowulf. The key steps are to:

1. Install the `Remote Developer Extension` in VSCode.
2. Create an SSH key for logging into biowulf (detailed in the biowulf guide)
3. Open your ssh config file on your Mac. Likely  `/Users/$username/.ssh/config` with $username being your user ID. In that file add (again with your user ID):

    ```
    Host cn*
    User $username
    ProxyCommand /usr/bin/ssh -o ForwardAgent=yes<EMAIL_ADDRESS>nc -w 120ms %h %p
    ```
4. Start an spersist node on biowulf and note the node number (i.e cn2728)
5. With the Remote Developer Exension installed, the bottom left corner of VSCode should be a symbol like >< in green. Click it, `Connect to Host` and type in just the node number (i.e. `cn2728`).

## Biowulf and Python
Sometimes biowulf and Python can have issues.
HPC has a fairly comprehensive guide [here][biowulf_python] that you can read.
One fairly common issue is the use of Python environments and biowulf, particularly conda.
There is a naming conflict ("dbus") that can be introduced, and may cause NoMachine to fail intermittently.
The solution is to first try to remove the lines inserted by conda into your `.bashrc`, so that conda does not load by default.
In the case that this does not work, try removing `dbus` with
```
conda uninstall dbus
```
which will remove the `dbus` package.
This can cause issues if you have packages which depend on it.

## Additional HPC resources
In addition to the user guide and tutorials, Dustin Moraczewski in the Data Sharing and Science Team has helpfully created additional biowulf resources.
Check them out [here][https://github.com/dmoracze/HPC_helper_tools].
Several key tools are the ability to store environment in an `spersist` node on the cluster, and the ability to easily run BIDS and fMRIPrep validation.

[biowulf_tutorials]: <https://hpc.nih.gov/training/intro_biowulf/>
[biowulf_guide]: <https://hpc.nih.gov/docs/userguide.html>
[helix_mount]: <https://hpc.nih.gov/docs/helixdrive.html>
[module_system]: <https://hpc.nih.gov/apps/modules.html>
[biowulf_vscode]: <https://hpc.nih.gov/apps/vscode.html>
[biowulf_python: <https://hpc.nih.gov/apps/python.html>


text:
---
uid: UsersManualPivSlots
summary: *content
---

# PIV slots

The PIV (Personal Identity Verification) standard specifies 25 slots. Depending on the
firmware version of the YubiKey, its PIV application will have 5, 25, 26, or 28 slots.
The table below lists all the slots and the firmware version it is first supported.

The first YubiKeys that implemented PIV only supported five of the slots. Starting with
version 4 of the firmware, all 25 slots were supported. With 4.3, Yubico added slot F9
(attestation), which is unique to the YubiKey (it is not part of the standard). For 5.3,
two more non-standard slots were added, but they are not really slots. What was added was
the ability to access metadata on the PIN and PUK, so slots 80 (PIN) and (81) PUK were
identified as where the PIN and PUK information is stored.

Each slot has a name and number. Each slot number is given as a hex value, and all slot
numbers can be represented as a single byte. For example, slot "9A" is the slot with the
number `0x9A`. That is decimal 154, but it is never referred to as "slot 154". If the
number of the slot is given, it will always be the hex value. Furthermore, it will
usually be written without the "0x". That is, when writing hex numbers, the custom is to
write it as `0x9A`. But in PIV documents, it is almost always written as "Slot 9A".

Some applications refer to a slot by its name. For example, slot 9A is the
"Authentication" slot. There are times in the standard or in the documentation of an
application where something such as this is described, "...using the Authentication
key..." or "...the key in the Authentication slot...".

The table below lists each of the slots by number and name. Notice that slot 9A holds
an asymmetric key, 9B holds a symmetric key, and slots 9C, 9D, and 9E hold asymmetric
keys.

#### Table 1: List of PIV slots
| Slot number | Name | Firmware version
first offered | Description |
| :---: | :---: | :---: | :---: |
| 80 | PIN | 5.3 | Not a standard slot, used by the [Get metadata command](xref:Yubico.YubiKey.Piv.Commands.GetMetadataCommand) |
| 81 | PUK | 5.3 | Not a standard slot, used by the [Get metadata command](xref:Yubico.YubiKey.Piv.Commands.GetMetadataCommand) |
| 9B | Management | all | Triple-DES key, no cert |
| 9A | PIV Authentication | all | RSA or ECC key and cert,
authenticate the user, usually for system login |
| 9C | Digital Signature | all | RSA or ECC key and cert,
signing email, files, executables, etc. |
| 9D | Key Management | all | RSA or ECC key and cert,
encryption for confidentiality, e.g. decrypting email |
| 9E | Card Authentication | all | RSA or ECC key and cert,
authenticate the card, usually building access |
| F9 | Attestation | 4.3 | Not a standard slot,
RSA key and cert,
used to attest other PIV keys generated by the YubiKey |
| 82 | Retired 1 | 4.0 | RSA or ECC key and cert,
usually keys with expired certs,
used to decrypt older emails or other encrypted items |
| 83 | Retired 2 | 4.0 | RSA or ECC key and cert,
usually keys with expired certs,
used to decrypt older emails or other encrypted items |
| 84 - 94 | ... | ... | ... |
| 95 | Retired 20 | 4.0 | RSA or ECC key and cert,
usually keys with expired certs,
used to decrypt older emails or other encrypted items |

## Attestation Key

The attestation key (in slot `F9`) will be used to create an attestation statement (which
is an X.509 certificate) that attests a key in slot `9A`, `9C`, `9D`, or `9E` was
generated on the YubiKey.

Upon manufacture, a private key and cert pair is loaded into slot `F9`. This key is
generated by Yubico, the cert is signed by a Yubico CA and chains to a Yubico root. The
same key and cert are loaded onto many different YubiKeys. See the article on
[PIV attestation](attestation.md) for more information on this topic.

Note that if a private key was imported into one of those slots, it will not be possible
to create an attestation statement for that slot.

It is possible to have the YubiKey generate a key pair for one of the retired slots
(`82` - `95`). However, the attestation key will not generate an attestation statement for
a key in one of those slots, even if it was generated by the YubiKey.

## Generate and import asymmetric keys

Slots `9A`, `9C`, `9D`, `9E`, `82 - 95`, and `F9` hold asymmetric keys. Or put another
way, all slots other than `80`, `81`, and `9B` hold asymmetric keys.

The slots that hold asymmetric keys (other than F9) are manufactured "empty". There are no
keys in those slots. In order to fill them with keys, you must either generate a new key
pair (see
[GenerateKeyPairCommand](xref:Yubico.YubiKey.Piv.Commands.GenerateKeyPairCommand)),
or import a key (see
[ImportAsymmetricKeyCommand](xref:Yubico.YubiKey.Piv.Commands.ImportAsymmetricKeyCommand)).

It is possible to generate or import a new attestation key (slot `F9`). However, if you do
so, the old attestation key is lost and there is nothing you can do to recover it. The
YubiKey will no longer be able to create an attestation statement, unless you obtain, for
the new attestation key, a proper certificate that chains to a supported root.

## Signing

Slot `9C` is the key named "Digital Signature". You will likely use this key to sign
emails, git commits, or other items. However, it is also possible to sign using the keys
in slots `9A`, `9D`, `9E`, and `82` - `95` as well. The YubiKey will not compute a
signature if you specify any other slot. Slots `80`, `81`, and `9B` do not hold asymmetric
keys and while `F9` will sign an attestation statement, it does not perform
general-purpose signing.

To sign using the YubiKey, use the [Authenticate: sign](commands.md#authenticate-sign)
command.


text:
---
title: "Blockchain, Bitcoin, and Libra"
permalink: "/blockchain-bitcoin-and-libra"

author: alexsvetski

tags:
  - Alex Svetski
  - CY19 Q3

excerpt: Alex Svetski looks at Bitcoin in comparison to the blockchain meme and Libra. Posted July 30, 2019.

defaults:
  - scope:
      type: posts
---

{% include donation.md %}

# [Blockchain, Bitcoin, and Libra](https://blockchain.news/Post?id=Blockchain-Bitcoin--Libra-)
### By [Alex Svetski](https://twitter.com/AleksSvetski)
### Posted July 30, 2019

I've been told my "definition" of [Blockchain](https://blockchain.news/tag?tag=Blockchain) differs from the “commonly used” one.

This is probably true, so I decided to examine why.

First of all, what’s meant by “commonly used”. I have a problem with this idea because I'm yet to understand the parameters of "commonly used", and am still trying to understand what anybody is actually referring to when they begin to throw the word "blockchain" around [with so much certainty].

A number of years ago, I found myself personally entangled in the term "blockchain", attempting to explain it to people at the local "bitcoin & blockchain" meetups I was hosting. The argument centred around grouping data in blocks & cryptographically hashing it to the data in the previous block in an attempt to build some form of "immutable ledger" (another term that gets thrown around) that could act as the basis of a more secure method of data storage.

Whilst on the surface, this sounds interesting, in reality it’s neither special, unique, profound or even useful.

I still remember the "aha" lightbulb moment where I realised that this concept was not the basis of Bitcoin’s immutability, & that immutability of the historical data on the ledger had very little to do with hashing to the previous block & everything to do with using that data structure/methodology together with the economic game of probability that Bitcoin used to achieve autonomous consensus (ie; its inclusion of PoW in its solution to the Byzantine Generals problem, aka; Nakamoto Consensus).

What I've since informed anyone I come across (or at least those who choose to listen) is that defining "blockchain" outside of the context of Bitcoin is like trying to define a carburetor without the existence of a car.

**Blockchain for Lettuce**

The sad truth is that since Bitcoin's inception, there have been many, many attempts to take absolutely necessary parts out of the Bitcoin Recipe (see page 5 of https://bitcointimes.news) in order to deliver something called a "blockchain" - for just about every application imaginable.

When one tries to understand the application or definition of their 'blockchain', you quickly realise it's just a glorified database, utilising some simple time stamps, & maybe data segmentation. Something we've had for decades, & in most instances, the use of which is completely pointless.

In a bid to stay ahead of the curve, the "blockchain" narrative evolved into "DLT"; standing for Distributed Ledger Technology, where the "block" part of the blockchain was removed & the "distributed" element (also part of the broader Bitcoin recipe) was now glorified. The new narrative is just a variation of the old. "It's immutable because every validator has a record of the ledger".

The problem this time around is that if every validator is known, or permissioned, then have we really created something new? One could argue that in some corporate applications where more checks & balances are required, or there is a need for shared decision making, this could be useful, but by no means is this immutable, broadly applicable, or as one of the ridiculous narratives goes; "faster".

Having more parties participate in validation is by definition slower - and that's fine - especially if you want more checks, but digital immutability is a binary term. You either have it or you don't. The private, permissioned distributed ledgers are not immutable in any way, because there is no cost to change the historical record - only a requirement to collude via the quorum of the group. If anything, DLT is just a glorified multi-party authentication system - which like I said, is fine in some cases - but not when painted with "all the glorious things" it's going to give the world.

Open, decentralized networks [insert blockchain word here] such as Bitcoin are immutable, distributed & happen to group transactions together to achieve autonomous consensus amongst network participants who do not know each other. The result is a digital network that is inherently tied to real world cost via real-world energy. The more energy & cost associated with the network, the greater it's degree of immutability.

Therein lies the innovation.

It is unprecedented & its application to something that requires an extraordinary degree of censorship resistance, eg; a non-sovereign form of monetary unit, is where "the killer app" lies.
Not IBM's "Blockchain for Lettuce".

**Is this recipe useful anywhere else?**

Many have argued that using the entire Bitcoin recipe, & applying its output (ie; digital immutability & censorship resistance) to things outside of a non-sovereign monetary network + unit is a good idea. Enter Ethereum, etc.

Whilst I won't go into the details here, I am very skeptical of the need to apply immutability & censorship resistance (or make it an integral component of the stack) to many things in the world due to the inherent cost of doing so.

I'm extremely skeptical of the idea of "Dapps". Even if we assume Ethereum is censorship resistant (the numbers don't add up), & that the Dapps are also actually decentralized (they're not), how many applications really need to have immutability or resistance to censorship as a native part of their stack? Maybe an "ebay for Hitmen" - but by & large, if you have an idea for an app, in today's day and age - just go spin up an AWS instance, and make it happen!

Bitcoin's unique recipe is very useful for a specific set of things, much like building a tank is useful for a specific set of things. Putting the tank on a race track or removing all the armour to make it faster & then sending it into battle are both misguided.

**The Verdict?**

The world Blockchain was popularised in 2013 & 2014 by groups of banking consortiums, who are inherently threatened by the disintermediation that Bitcoin presents.

Seeing the banks, accounting & IT giants, like the Big 4, the IBM's & Accenture's of the world, tout DLT or Blockchain is just classic PR. They're announcements by large, (in most cases boring) corporates who are desperately trying to show that they’re still somehow “innovative”.
Similar to how they use (and exaggerate) the words “cloud”, or “Ai”, or “big data”.

There is very little innovation there. So I call Bull$#%!.

**Onto Libra**

Libra has red-pilled more people into Bitcoin recently than just about anything else.
The idea that a non-sovereign currency can be issued, by an entity which is not a government or a state has actually opened the minds of millions of people who could not perceive that earlier.

The fact that Facebook has circa 2bn people on their network makes it the largest community (country) in the world, & people’s ability to grasp the notion of that community having their own money is a much easier mental leap.

Once they make that leap, they then have an easier time understanding Bitcoin, as seen by the journey people like Joe Kernan from CNBC Squakbox are on.

Whilst Facebook has used the term "Blockchain" in their definition of what Libra is, it’s actually an inaccurate representation. I won't do the argument as much justice as Jameson Lopp did, so I'll link to his article here:

[https://onezero.medium.com/thoughts-on-libra-blockchain-49b8f6c26372 ](https://onezero.medium.com/thoughts-on-libra-blockchain-49b8f6c26372)

But suffice it say; it's not a blockchain.

In fact, Facebook & the Libra consortium have ZERO interest in building their own “blockchain”.
What they’re attempting with Libra is something so much larger, & it’s exactly why all the governments (big or small) are up in arms about it.

Let’s think about it for a moment.
- Libra is in direct competition with government based fiat currencies (and they all know it)
- Consider what a population in a jurisdiction with a weaker national currency would do if LIbra launches there. Why would a citizen hold their hard earned wealth in something that’s depreciating by 10 - 20% per annum?
- You quickly come to the realisation that there would be a “bank-run” on the national currency, with all the capital flowing onto Libra. The result would be catastrophic for the government in power, as it’s economic lever is the basis of its control of the population.

Governments are scared — and rightly so.
Facebook is arguably the largest “country” in the world, & their currency would be more useful to more people than the fragmented fiat ones they’re using today.

This is the real play. Libra is attempting to do their own Bitcoin - they couldn’t care less about Blockchain.

**Bitcoin is the “Killer App”.**

Bitcoin is fundamentally unique.

Its immaculate conception & organic introduction to the market, its initial broad based adoption for strong, libertarian reasons & the timely disappearance of the founder, leaving no “head of the snake” to cut off are just a few of the things that make it impossible to replicate.

Digital scarcity, Bitcoin’s core innovation is by definition, a one-time event. Every other cryptocurrency, optimising for complex, turing complete smart contracts on the base layer, or faster payments are quickly being shown as irrelevant or short term noise.

Bitcoin’s focus on being a censorship resistant public network, owned by the participants, that’s broadly decentralised & organically priced by the market is the killer app.

Libra by facebook reinforces all of the above, by making all other cryptocurrencies who optimised for the wrong thing obsolete, & giving us a strong contrast to what Bitcoin represents for the world, ie; a money of & for the people — that cannot be shut down, censored, confiscated or inflated.

This was a once-off opportunity.

The smart money knows this & Bitcoin will continue to suck up all the capital & liquidity whilst all the other noise (crypto) will continue to trend toward $0 against Bitcoin in both dollar terms & relevance.

**Bitcoin VS Libra: The real showdown.**

Libra inherently validates Bitcoin.

It opens people’s minds to the idea of a floating, global currency that is not managed or issued by a state of government.

The question is, would you use it?
Or would you prefer to use something like Bitcoin?

Whilst there are similarities between them, I would argue that Bitcoin is the antithesis to something like Libra, which to me sounds like we’re jumping from the frying pan into the fire.

Whilst I’m not a fan of government or central bank-decreed fiat money, I question the logic of moving from a global USD reserve to a Libra reserve.

Sounds a little like 1984 to me.
Not to sound conspiracy theorist, but the idea of one party, or even a consortium of the “big boys” having the final word on the most important resource in society, ie; money, ie; the unit of account that measures the input of all participants in society + the world at large → sounds dangerous to me.

This, fellow readers, is why Bitcoin matters.
It’s exactly why Bitcoin optimised for censorship resistance, privacy & self-sovereignty, not “fast payments” or “smart contracts” like all the other now-irrelevant crypto’s did.

All of the “potential” that’s promised (amongst the noise) in crypto will be built on top of the most robust network. What remains to be seen is whether that’s Bitcoin, or whether that’s Libra (or both).

A new Money is the real battle-ground.
Not Blockchain.

And in my mind, Bitcoin is the only real alternative to a potentially Orwellian future.

Verdict on Libra?
Big Brother.

**Conclusion**

Bitcoin is the most under-appreciated innovation in History, but as with all great zero to one innovations, it’s the contrarian aspect that makes it so much more profound.

In 1000yrs, the concept of money will very much still exist — because it’s the foundational element required for any society to function.

Much like the internet, Bitcoin is a public good — owned by the people — and by taking the **_only_** two finite resources that we know of (time & energy), Bitcoin is able to give the world a superior monetary network & unit that we can use to better collaborate & function as an open society.

Blockchain is not an alternative, nor does it even matter & nor will it even be spoken about in the coming 5yrs. The real alternative is potentially 1984-stye big brother currencies, whether they’re government issued (eg; China’s social credit system), or issued by non-state consortiums with disproportionate influence in the world (eg; what FB is doing with Libra).

It’s the opt out of those that gives Bitcoin the brightest future of all.

Verdict on Bitcoin?
**Brilliant.**

gha_language: CSS

gha_license_id: MIT


text:
[TOC]

Introduction
============
[ObjectBox](https://objectbox.io/) is a fast database for data objects.

Installation and examples
-------------------------
See [Installation docs](https://cpp.objectbox.io/installation) and check the [project's GitHub repository](https://github.com/objectbox/objectbox-c) for additional info and examples including the usage of flatcc.

Headers
-------------
* ObjectBox C99 support is provided by a single header file: **objectbox.h**
* ObjectBox C++11 support is available in an additional header: **objectbox.hpp**

Basic concepts
--------------
* Objects are the entities persisted in the database
* Objects are [FlatBuffers](https://google.github.io/flatbuffers/)
* Objects are "grouped" by their type; e.g. there is a Box (or "Cursor") for each type
* Objects are addressed using a 64 bit integer ID (`obx_id`)
* There is no query language; queries are build using a [query builder](\ref OBX_query_builder)

See [docs](https://cpp.objectbox.io) for more information on how to use ObjectBox in C and C++

API Naming conventions
----------------------
* C methods: obx_thing_action()
* C structs: OBX_thing {}
* C error codes: OBX_ERROR_REASON
* C++ namespace: obx::

Essential types
-----------------
Check the docs for the following types:

* [OBX_store](\ref OBX_store) and [obx::Store](\ref obx::Store): the "database"; "opens" data files in a given directory
* [OBX_box](\ref OBX_box) and [obx::Box](\ref obx::Box): object operations like put and get
* [OBX_query_builder](\ref OBX_query_builder) and [obx::QueryBuilder](\ref obx::QueryBuilder): used to construct queries using "query conditions"
* [OBX_query](\ref OBX_query) and [obx::Query](\ref obx::Query): the product of a query builder can be executed to find objects matching the previously defined conditions

Essential Links/Readings
------------------------
* High-level docs and examples: https://cpp.objectbox.io
* Entity (object) IDs: https://docs.objectbox.io/advanced/object-ids
* Meta model and UIDs: https://docs.objectbox.io/advanced/meta-model-ids-and-uids

repository: objectbox/objectbox-c

gha_language: C

repo_name: objectbox/objectbox-c


text:
The-M-Project Website
==========================

## Build

1. `npm install`
1. `bower install`
1. `grunt`

## Setup Development

1. `npm install`
1. `bower install`
1. `grunt` - gets the latest docs, generates the site
1. use ```grunt dev``` if you are editing templates or less files. (Note: doc pages will have to be regenerated)


text:
# MapThing Developer Documentation

MapThing is a desktop-based, extensible web map viewer. It is intended to be beautiful, easy to use and
allow rich, interactive visualisation of geospatial data which can be hard to achieve in a traditional GIS.
MapThing can (and should!) be extended through its' plugin system.

- [Documentation on developing plugins](plugins.md)
- [Documentation on developing the MapThing core](core.md)

repository: JBARiskManagement/MapPIG

repo_name: JBARiskManagement/MapPIG


text:
# country-count

Given an array of string, return a list of countries and number of times country appears in array

[![NPM](https://nodei.co/npm/country-count.png?downloads&stars)](https://nodei.co/npm/country-count/)

[![NPM](https://nodei.co/npm-dl/country-count.png)](https://nodei.co/npm/country-count/)

## Installation

```
npm install country-count
```

## Example

### Input

```javascript
var countryCount = require('./country-count')

console.log(countryCount(['USA', 'America', 'Sweden', 'Finland']))
```

### Output

```
{ 'United States': 2, Sweden: 1, Finland: 1 }
```

## Licence

Copyright (c) 2014 Lisa Övermyr & David Björklund

This software is released under the MIT license:

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

 IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

repository: well-knits/country-count

gha_language: JavaScript

repo_name: well-knits/country-count


text:
---
layout: post
title: "MacScripts"
date: 2013-01-01 16:12
comments: true
categories:
- scripting
- git
- OS X
---
I've had my Mac scripts dotted around my Macs over the years, so after spending twenty minutes looking for my script to add a login item to a user, I decided to put them all in one place on GitHub. Hopefully someone will find them useful, I'll be adding the rest as soon as I work out where I put them.

[MacScripts on GitHub](https://github.com/grahamgilbert/macscripts)

repository: poundbangbash/blog

extension: markdown

repo_name: poundbangbash/blog


text:
# d3-multichord

###This is a modification of `d3-chord` to enable another dimension of data to be shown between nodes.

Circular layout for visualizing relationships or network flows with multiple types.

## Installing

If you use NPM, `npm install d3-multichord`. Otherwise, download the [latest release](https://github.com/chornbaker/d3-multichord/releases/latest).

## API Reference

This module is an adaptation of [d3-chord](https://github.com/d3/d3-chord). Only the changes relevant to the multichord version are included here.

# d3.multichord() [<>](https://github.com/chornbaker/d3-multichord/blob/master/src/multichord.js "Source")

Constructs a new multichord layout with the default settings.

# chord(matrix) [<>](https://github.com/chornbaker/d3-multichord/blob/master/src/multichord.js#L19 "Source")

Computes the chord layout for the specified *three-dimensional* *matrix* of shape *n*×*n*×*m*, where the *matrix* represents the directed flow amongst a network (a complete digraph) of *n* nodes, and links between nodes are classified into *m* categories. The given *matrix* must be an array of length *n*, where each element *matrix*[*i*] is an array of *n* numbers, where each *matrix*[*i*][*j*] represents the flow from the *i*th node in the network to the *j*th node, and each element *matrix*[*i*][*j*][*k*] is a an array of length *m* that represents the value of the flow from each of *m* categories. Each number within *matrix*[*i*][*j*][*k*] must be nonnegative, though it can be zero if there is no flow from node *i* to node *j* for that category.

Sample data from the [US Census Migration Data by Nativity Example](https://bl.ocks.org/chornbaker/251c1f9aa12eed2cedcef8fc33b76df0):

```js
var matrix = [
    [[0,0,0],[49,0,6],[236,1,18],[135,11,6]],
    [[37,1,9],[0,0,0],[230,9,19],[94,3,0]],
    [[295,6,24],[282,8,25],[0,0,0],[323,18,29]],
    [[181,5,16],[107,13,13],[195,6,26],[0,0,0]]
];
```

The *chords* array also defines a secondary array of length *n*, *chords*.groups, where each group represents the combined inflow and outflow for node *i*, corresponding to the sum of the elements in *matrix*[*i*][0 … *n* - 1][0 … *m* - 1], and is an object with the following properties:

* `startAngle` - the start angle in radians
* `endAngle` - the end angle in radians
* `value` - `in`: the total incoming flow value for node *i*; `out`: the total outgoing flow value for node *i*
* `index` - the node index *i*


text:
# Kanban application

You can drag-and-drop kanban cards to organize your tasks.

## Application

```kanban
* TODO
  * task 1
  * **task 2**
  * task 3
* DOING
  * *task 4*
  * task 5
* DONE
  * task 6
  * Add kanban to [app list](#/doc/apps)
```

## Usage

* To add list or card, input a name and click 'add' button.
* To move list or card, drag-and-drop it.
* To remove list or card, drag it and drop to 'Drop here to remove'.

## Data format

Kanban application uses Markdown format. Markdown should be a nested unordered list and its depth should be two.

It also understands Markdown emphasises, e.g. `*empasis 1*`, `**empasis 2**`, and `***empasis 3***`. Emphasised items will be colored.
Emphasises should be applied to whole text.

Text may include Markdown links.


text:
# Methods / Pipelines

This tab links to the nextflow pipelines used to process wild isolate sequence data.

![](/static/img/overview.drawio.svg)

### FASTQ QC and Trimming
__[andersenlab/trim-fq-nf](https://github.com/andersenlab/trim-fq-nf) -- (Latest [f0b63e](https://github.com/AndersenLab/trim-fq-nf/tree/f0b63e))__

Adapters and low quality sequences were trimmed off of raw reads using [fastp (0.20.0)](https://github.com/OpenGene/fastp) and default parameters. Reads shorter than 20 bp after trimming were discarded.

### __Alignment__

__[andersenlab/alignment-nf](https://github.com/andersenlab/alignment-nf) -- ([892b37](https://github.com/AndersenLab/alignment-nf/tree/892b37))__

Trimmed reads were aligned to _C. elegans_ reference genome (project PRJNA13758 version WS276 from the [Wormbase](https://wormbase.org/)) using `bwa mem` [BWA (0.7.17)](http://bio-bwa.sourceforge.net/). Libraries of the same strain were merged together and indexed by [sambamba (0.7.0)](https://lomereiter.github.io/sambamba/). Duplicates were flagged with [Picard (2.21.3)](https://broadinstitute.github.io/picard/).

Strains with less than 14x coverage were not included in the alignment report and subsequent analyses.

### __Variant Calling__

__[andersenlab/wi-gatk](https://github.com/andersenlab/wi-gatk) -- ([c9dd1e](https://github.com/AndersenLab/wi-gatk/tree/c9dd1e))__

Variants for each strain were called using `gatk HaplotypeCaller`. After the initial variant calling, variants were combined and then recalled jointly using `gatk GenomicsDBImport` and `gatk GenotypeGVCFs` [GATK (<IP_ADDRESS>)](https://gatk.broadinstitute.org/hc/en-us/sections/360007279452-4-1-4-0?page=6#articles).

The variants were further processed and filtered with custom-written scripts for [heterozygous SNV polarization](https://github.com/AndersenLab/wi-gatk/blob/master/env/het_polarization.nim), GATK (<IP_ADDRESS>), and [bcftools (1.10)](http://samtools.github.io/bcftools/bcftools.html).

Warning

Heterozygous polarization and filtering thresholds were optimized for single nucleotide variants (SNVs).

Additionally, insertion or deletion (indel) variants less than 50 bp are more reliably called than indel variants greater than this size. In general, indel variants should be considered less reliable than SNVs.

#### Site-level filtering and annotation

1. __Heterozygous SNV polarization__: Because _C. elegans_ is a selfing species, heterozygous SNV sites are most likely errors. Biallelic heterozygous SNVs were converted to homozygous REF or ALT if we had sufficient evidence for conversion. Only biallelic SNVs that are not on mitochondria DNA were included in this step. Specifically, the SNV was converted if the normalized Phred-scaled likelihoods (PL) met the following criteria (a smaller PL means more confidence). Any heterozygous SNVs that did not meet these criteria were left unchanged.
    * If PL-ALT/PL-REF <= 0.5 and PL-ALT <= 200, convert to homozygous ALT
    * If PL-REF/PL-ALT <= 0.5 and PL-REF <= 200, convert to homozygous REF

2. __Soft filtering__: Low quality sites were flagged but not modified or removed.

    For the __site-level__ soft filter, variant sites that meet the following conditions were flagged as PASS. These stats were computed across all samples for each site.

    * Variant quality (QUAL) > 30 (this filter is very lenient, only three sites failed)
    * Variant quality normalized by read depth (QD) > 20
    * Strand bias of ALT calls: strand odds ratio (SOR) < 5
    * Strand bias of ALT calls: Fisherstrand (FS) < 100
    * Fraction of samples with missing genotype < 95%
    * Fraction of samples with heterozygous genotype after heterozygous site polarization < 10%

    For the __sample-level__ soft filter, genotypes that meet the following filters were flagged as PASS for each site in each sample:

    * Read depth (DP) > 5
    * Site is not heterozygous

3. __SnpEff Annotation__: The predicted impact of each variant site was annotated with [SnpEff (4.3.1t)](https://pcingola.github.io/SnpEff/SnpEff.html).

4. For the hard-filtered VCF, low quality sites were modified or removed using the following criteria.

    * For the __site-level__ hard filter, variant sites not flagged as PASS were removed.
	* For the __sample-level__ hard filter, genotypes not flagged as PASS were converted to missing (`./.`), with the exception that heterozygous sites on mitochondria where kept unchanged.

    After the steps above, sites that are invariant (`0/0` or `1/1` across all samples, not counting missing `./.`) were removed.

5. Variant impacts were then annotated using `bcftools csq`, which takes into consideration nearby variants and annotates variant impacts based on haplotypes.

#### Determination of filter thresholds

We re-examined our filter thresholds for this release. A variant simulation pipeline was used as part of this process:

* __Variant Simulations - __[andersenlab/variant-simulations-nf](https://github.com/andersenlab/variant-simulations-nf)

Please see the [filter optimization report](/static/reports/filter_optimization/20200803_optimization_report.html) for further details.

### __Isotype Assignment__
__[andersenlab/concordance-nf](https://github.com/andersenlab/concordance-nf) -- ([ae3d80](https://github.com/andersenlab/concordance-nf/tree/ae3d80))__

Isotype groups contain strains that are likely identical to each other and were sampled from the same isolation locations. For any phenotypic assay, only the isotype reference strain needs to be scored. Users interested in individual strain genotypes can use the strain-level data.

Strains were grouped into isotypes using the following steps:

1. Using all high quality variants (the hard-filtered VCF) and `bcftools gtcheck`, concordance for each pair of strains was calculated as a fraction of shared variants over the total variants in each pair.

2. Strain pairs with concordance > 0.9995 were grouped into the same isotype group. The threshold 0.9995 was determined by:

    * Examining the distribution of concordance scores.
    * Capturing similarity between strains to minimize the number of strains that get  assigned to multiple isotype groups.
    * Agreement with the isotype groups in previous releases.

3. The following issues, which were rare, were resolved on a case-by-case basis:

    * If one strain was assigned to multiple isotypes.
    * If one isotype from previous releases matches to multiple new isotype groups.
    * If one new isotype group contains strains from multiple isotypes from previous releases.

When issues arose, the pairwise concordance between all strains within an isotype were examined manually. Strains and isotypes may be re-assigned with the goal that strains within the same isotype group should have high concordance with each other, and strains from different isotype groups should have lower concordance.

### __Tree Generation__

Trees were generated by converting the hard-filtered VCF to Phylip format using [vcf2phylip (030b8d)](https://github.com/edgardomortiz/vcf2phylip/tree/030b8d). Then, the Phylip format was converted to Stockholm format using [Bioconvert (0.3.0)](https://bioconvert.readthedocs.io/en/master/index.html), which was then used to construct a tree with [QuickTree (2.5)](https://github.com/tseemann/quicktree) using default settings. The trees were plotted with [FigTree (1.4.4)](http://tree.bio.ed.ac.uk/software/figtree/) rooting on the most diverse strain XZ1516.

### __Imputation__

Imputation was done using [Beagle (5.1)](https://faculty.washington.edu/browning/beagle/beagle.html) with the following parameters: window=5 overlap=2 (except for chromosome X, which used window=3 overlap=1 to run with available memory) impute=true ne=100000 imp-segment=0.5 imp-step=0.01 cluster=0.0005.

gha_language: HTML


text:
---
layout: post
title: Get ready for the first virtual OHBM brainhack!
isStaticPost: true
permalink: /logistics/
image: schedule.jpg
---

OHBM brainhack Hubs

{% accordion hackathonHubs %}

{% collapse What are the hubs? %}
In addition to this event being the first ever virtual OHBM brainhack, it will also be happening three
times within a 24 hour window, across multiple time-zones. The goal here is to ensure that more people
from around the world are able to join in during reasonable working hours for them.
* Americas Hub
* Europe, Middle East, and Africa Hub
* Asia and Pacific Hub
{% endcollapse %}

{% collapse Hub structure and general organization %}
In order for the event to run as smoothly as possible across all hubs, each hub will have a list of
roles filled by volunteers.
Some of the roles are selected by the OHBM Open-Science Special Interest Group (OSSIG) while others
are open to volunteers.

Selected by the OS-SIG

{% collapse brainhack hub hosts %}
Each hub has two brainhack hosts who welcome the participants during the kickoff and project pitches,
moderate the unconferences, and liase with the other volunteers.
{% endcollapse %}

{% collapse Hub manager %}
The hub manager makes sure that everything is running smoothly in the background!
{% endcollapse %}

{% collapse TrainTrack instructors %}
Each TrainTrack session has an instructor, and there will be teaching assistants available for answering questions
while a TrainTrack session is being presented.
{% endcollapse %}

{% collapse Youtube master %}
The Youtube master ensures that all content is streamed to different platforms.
{% endcollapse %}

Open to volunteers

{% collapse Project advisor %}
In the weeks preceding the brainhack, project advisors will review submitted projects and make sure
that they are set up following brainhack project setup guidelines. Then during the brainhack they can
provide support and especially help project leaders who might feel a bit lost if it is their first time
managing a team.
{% endcollapse %}

{% collapse brainhack mentor %}
You’ve already attended a brainhack, are keen in welcoming new members in the brainhack community? As a
brainhack mentor, you will make sure that your mentees have followed the participant onboarding,
you will spend 1-hour a day with them to help with their on-boarding.
{% endcollapse %}

{% collapse Help desk %}
Signup for the brainhack helpdesk and answer attendees technical questions. As a member of the helpdesk
you will sign up for monitoring the helpdesk chat and answer any question.
{% endcollapse %}

{% collapse Social breakout host %}
Each hub will have a virtual room where people can go and chat, meet with people for their morning or
afternoon coffee, lunch, or evening drinks. The social breakout host's role is keep the conversation
going in a relaxed and inclusive atmosphere.
{% endcollapse %}
{% collapse Social media person %}

{% endcollapse %}

{% endcollapse %}

{% endaccordion %}

Platforms

{% accordion software %}

{% collapse Crowdcast %}
Each hub will be using Crowdcast for the main sessions, and this content will be streamed on Youtube
simultaneously such that those who are not formally registered can follow along as well.
{% endcollapse %}

{% collapse Mattermost %}
* Generic channels:
  * [HBM-hackathon](https://mattermost.brainhack.org/brainhack/channels/hbm-hackathon)
  * [General](https://mattermost.brainhack.org/brainhack/channels/general)
  * [Town Square](https://mattermost.brainhack.org/brainhack/channels/town-square)
* Project channel: Each project will have its own channel
* Help desk: We will have a set of topic and programming language specific channels where you can
  ping volunteers for help.
{% endcollapse %}

{% collapse Zoom/Jitsi %}
Each project will be using Zoom or Jitsi for their individual calls.
{% endcollapse %}

{% endaccordion %}

HackTrack

{% accordion HackTrack %}

As with previous OHBM hackathons, we've developed a project template that we would like project leads
to use for submitting their project proposals.

{% collapse Project submission %}
Each project is allocated to a project helper who will follow the project from submission until the
end of the brainhack.
To make sure that project helpers have enough time to review project submissions, we're setting two
deadlines:
- 2 weeks before the brainhack: submit a draft for project helpers to give feedback on
- 1 week before brainhack: hard deadline

[**Project checklist**](https://docs.google.com/spreadsheets/d/1SY-Hnhc-bPX0RA47icVukUVVJ-BrX9ykPFQUBY-U1go/edit?usp=sharing):
this is a spreadsheet to help the project helpers making sure that each project is on track and submitted everything.

[**Github issue template**](https://github.com/ohbm/hackathon2020/blob/master/.github/ISSUE_TEMPLATE/hackathon-project-template.md):
This will be the way to submit brainhack projects. It comes with a checklist to make sure that project
leaders and participants can start in the best conditions.

Once a project has been submitted the project helper can tag it using issue labels to make it easier to search amongst the different projects.
{% endcollapse %}

{% endaccordion %}

TrainTrack

The TrainTrack will cover topics ranging from good code practices, code and data version control,
high performance computing, data visualization, and machine learning. Each session will have its own
computing requirements, but at a baseline, we strongly recommend that participants make sure that
they have the following general computing requirements satisfied.
We will have a virtual help desk available to help participants with installation issues.

{% accordion TrainTrack %}

{% collapse General computing requirements %}
In addition to some software that we will list in the subsequent section, these are the general computing
requirements:
- Administrative access to your computer such that you can install software on your own.
- If you're using Windows, the required version is the 64 bit version of Windows 10. Please contact
  us if you don't have either MacOS, Linux, or a Windows 10 64-bit machine.
{% endcollapse %}

{% collapse Required software %}
Along with the platforms that we will be using for virtual TrainTrack presentations and communication,
we ask that you install the following software.
* A command-line shell: Bash
* A version control system: Git
* A remote-capable text editor: VSCode
* Python 3 via Miniconda
* A virtualization system: Docker
* A GitHub account
* Mattermost (can be used in the browser as well)
* Zoom
* A modern browser (for using Crowdcast and Jitsi)

{% endcollapse %}

{% endaccordion %}

OS-specific installation instructions

Coming soon!


text:
## [0.0.3] - 2020-12-29
- Scanner can recognize strings and nil tokens
### Added
- class `Datatype::Nil` for representing nil tokens in Lox
- class `Datatype::LXString` for representing Lox strings

### Changed
- Method `BuilinDatatype::validated_value` now accepts the input value as is.
- `Frontend::Scanner` updated code to recognize strings and nil
- File `scanner_spec.rb` Added tests for strings, nil scanning

## [0.0.2] - 2020-12-28
- Scanner can recognize keywords, boolean and numbers
### Added
- `Datatype` module and classes

### Changed
- `Frontend::Scanner` updated code to recognize booleans and numbers
- File `scanner_spec.rb` Added tests for keywords, booleans and number scanning
- File `README.md` fixed a couple of typos.

## [0.0.1] - 2020-12-27
### Added
- Initial Github commit

repository: GuiHeurich/loxxy

repo_name: GuiHeurich/loxxy


text:
# persistent_bottom_nav_bar example project

This is an example project which demonstrates the use of `persistent_bottom_nav_bar` package.

## Getting Started

Head over to the [official readme](https://github.com/BilalShahid13/PersistentBottomNavBar/blob/master/README.md) to get started.

gha_language: Dart


text:
# enchannel

Enchannel is a light spec for setting up communications between a frontend, like the [notebook](https://github.com/jupyter/notebook) or [jupyter-sidecar](https://github.com/nteract/jupyter-sidecar), and a backend kernel (the runtime, like Python, Julia, or R).

[![enchannel](https://cloud.githubusercontent.com/assets/836375/12282043/b19bb16e-b960-11e5-8661-ce2111ec0417.png)](https://cloud.githubusercontent.com/assets/836375/12282043/b19bb16e-b960-11e5-8661-ce2111ec0417.png)

The core functionality of the notebook is to send messages from a frontend to a backend, and from a backend to a frontend ([or many frontends](https://github.com/nteract/jupyter-sidecar)).

In the case of the Jupyter/IPython notebook, it communicates over websockets (which in turn reach out to 0MQ).

What if you want to serve the same HTML and Javascript for the notebook application itself while being able to work in a native 0MQ environment? What if websockets are fairly restricted in your *ahem corporate* environment and you need to send data via `POST` or stream updates using server-sent events?

We'd need a nice clean way to abstract the transport layer. Since Jupyter is messages all the way down, one way is to hook up a series of event emitters all with the same interface. That's [definitely do-able](https://github.com/nteract/jupyter-transport-wrapper). What is proposed here is that we would rely on `Observable`s, asynchronous data streams. Even better is to rely on RxJS's implementation, since we get a nice functional approach to messaging:

```javascript
iopub.filter(msg => msg.header.msg_type === 'execute_result')
     .map(msg => msg.content.data)
     .subscribe(x => { console.log(`DATA: ${util.inspect(x)}`)})
```

On top of that, since these are subjects, we can go ahead and submit messages to the underlying transport:

```javascript
var message = {
  header: {
    msg_id: `execute_${uuid.v4()}`,
    username: '',
    session: '00000000-0000-0000-0000-000000000000',
    msg_type: 'execute_request',
    version: '5.0',
  },
  content: {
    code: 'print("woo")',
    silent: false,
    store_history: true,
    user_expressions: {},
    allow_stdin: false,
  },
};

shell.send(message); // alias of onNext - the actual Rx parlance
```

In order to get this, we make 4 RxJS Subjects:

* Shell
* STDIN
* Control
* IOPub (Which can alternatively be an `Observable`)

Each backend has to implement this by providing an exported function that can take their choice of parameters, so long as they all return

```javascript
{
  iopub: iopubObservable,
  shell: shellSubject,
  control: controlSubject,
  stdin: stdinSubject
}
```

repository: captainsafia/enchannel

repo_name: captainsafia/enchannel


text:
[![Build Status](https://travis-ci.org/sczerwinski/android-delegates-shared-preferences.svg?branch=master)](https://travis-ci.org/sczerwinski/android-delegates-shared-preferences)
[![Download](https://api.bintray.com/packages/sczerwinski/android/delegates-shared-preferences/images/download.svg)](https://bintray.com/sczerwinski/android/delegates-shared-preferences/_latestVersion)

# Kotlin Delegates for Android SharedPreferences

A set of Kotlin delegates whose values are stored in Android `SharedPreferences`.

## Gradle Build Configuration

To use the library, add another line to `build.gradle` in your Android project:
```gradle
dependencies {
    compile 'it.czerwinski.android:delegates-shared-preferences:0.1'
}
```

## Usage

This library uses the concept of
[Delegated Properties](https://kotlinlang.org/docs/reference/delegated-properties.html) in Kotlin.

```kotlin
class MainActivity : Activity() {

    // Delegate a property in your Activity to a preference with the desired name:
    val username by stringSharedPreference("DEFAULT_USERNAME")

    // If you need to change the value stored in the preference, simply use "var" instead of "val":
    var fontSize by intSharedPreference("SETTINGS_FONT_SIZE")

    // If you don't want nullable types, provide the default value:
    val parentalControl by booleanSharedPreference("SETTINGS_PARENTAL_CONTROL", defaultValue = true)

    // [...]
}
```


text:
# Linux commands

`<cmd> --help ` command documentation (help)
`man <cmd(utility)>` system reference manual utility

`pwd` print working directory

`ls` list current directory content
`ls ~` list HOME
`l` alias for 'ls -CF'
`l <path>` list for path
`ls -l` ~ with long info
`ls -r` ~ reverse order
`ls -p` ~ append / indicator to directories
`ls -a` ~ do not ignore entries starting with .
`ls -A` ~ do not list implied . and ..

`cd <path>` change directory
`cd ..` jump to upper directory
`cd ~` jump HOME
`cd /` jump to the root

`clear` clear the terminal screen

`touch <FILE1> <FILE2> ...` change file timestamp, a FILE argument that does not exist is created empty
`mkdir <DIR1> <DIR2> ...` make directories
`cp <SRC1> <SRC2> ... <DIR>` copy files and directories
`mv <SRC> <DEST>` move (rename) files

`rm <FILE>` remove files or directories
`rm <PATH>/*` remove all files in directory
`rm <PATH>/*.txt` remove all files in directory with '.txt' extention
`rm <PATH>/test.*` remove all files in directory started with 'test.' prefix
`rm -d <PATH>` remove folder

`sudo <cmd>` execute command as the superuser or another user
`sudo su` execute command 'switch user' as a superuser
`sudo su <USER>` switch user to another user
`su mpalii` switch user to 'mpalii'

`!!` take content of a previous command

`chown [OWNER]:[GROUP] <FILE>` change file owner and group
`chmod [MODE] <FILE>` change file mode bits (000, 666, 664 etc.)

`apt [CMD]` high-level commandline interface for the package management system

`apt-get [CMD]` APT package handling utility
`apt-get install <application>` install application and dependencies
`apt-get remove <application>` remove application and dependencies
`apt-get update` retrieve new list of packages
`apt-get upgrade` perform an upgrade

`apt-cache [CMD]` query the APT cache (does not manipulate the state of the system)
`apt-cache policy <package>` show policy settings
`apt-cache search <name pattern>` search the package list

`add-apt-repository` adds a repository into the /etc/apt/sources.list or /etc/apt/sources.list.d or removes an existing one

`find` find for files in a directory hierarchy
`find . -type f -name "*.txt"` find all files in the current directory with '.txt' extension
`find . -type f -iname "main*"` find all files in the current directory with 'main' prefix, ignore case
`find . -type f -perm 0664` find all files in the current directory with 664 permissions
`find . -type f -size -1M` find all files in the current directory with size less than 1 megabyte
`find . -type d -size +1k` find all directories in the current directory with size greater than 1 kilobyte
`find . -type f -not -iname "*.txt"` find all files in the current directory with the extention NOT '.txt', ignore case

`cat [FILE]` concatenate files and print out on the standard output

`grep` print lines that match patterns
`grep "some text" file.txt` print lines from file.txt that contain "some text"
`grep "text" file1.txt file2.txt` print lines from file1.txt and file2.txt that contains "text"
`grep -n "text" test.txt"` print lines and line number from test.txt that contains "text"
`grep -n -i "tEXt" file.txt` print lines and line number from file.txt that contains "text", ignore case

`find . -type f -iname "*.cpp" -exec grep -i -n "main*" {} +` find files with 'cpp' extention and search for 'main*' in them

`sort [FILE]...` sort lines of text files
`sort file.txt > fileCopy.txt` sort lines of 'file.txt' and save result to 'fileCopy.txt'
`sort -n [FILE]` sort lines of file as numbers
`sort -r [FILE]` sort lines of file in reverse order

`top` display Linux processes

`free` display amount of free and used memory in the system
`free -h` ~ automatically scaled values

`ps` report a snapshot of the current processes
`ps -aux` ~ all

`tree` list contents of directories in a tree-like format

`tar` an archiving utility
`tar -cf archive.tar foo bar` create archive.tar wrapper from files foo and bar
`tar -tf archive.tar` list all files in archive.tar
`tar -xf archive.tar` extract all files from archive.tar

`xz archive.tar` compress archive.tar
`unxz archive.tar.xz` decompress archive.tar

`gzip archive.tar` compress archive.tar
`gunzip archive.tar.gz` decompress archive.tar
`gzip -d archive.tar.gz` decompress archive.tar

`bzip2 archive.tar` compress archive.tar
`bunzip2 archive.tar.bz2` decompress archive.tar
`bzip2 -d archive.tar.bz2` decompress archive.tar

`zip -r archive.zip folder` compress folder to archive.zip
`unzip archive.zip` decompress archive.zip

`wget [URL]` network downloader

`dpkg` package manager fot Debian
`dpkg --install [downloaded package].deb` install package
`dpkg --remove [application name]` remove package/app

`ping [URL]` send ICMP(Internet Control Message Protocol) ECHO_REQUEST to network hosts
`ping -c 2 [URL]` ~ 2 times

`host [URL]` DNS lookup utility

`netstat` print network connections, routing tables, interface statistics, masquerade connections, and multicast membership
`ifconfig` configure a network interface
`ip` show/manipulate routing, network devices, interfaces and tunnels
`route` show/manipulate the IP routing table
`ssh` OpenSSH remote login client
`ufw allow 22` open port 22

`w` show who is logged on and what they are doing

license: Unlicense

repository: mpalii/knowledge-base

gha_license_id: Unlicense

repo_name: mpalii/knowledge-base


text:
---
layout: posterinfo
title: Vivio in Typo poster (2008)
artist: Philippe Apeloig
date: 2000s
type: exhibition
color:
  - red
  - white
  - black
country: France
image:

dimensions: 175 x 118 cm (5 ft. 8 7/8 in. x 46 7/16 in.)
source:  https://collection.cooperhewitt.org/objects/35460843
---

 This is a Poster. It was designed by Philippe Apeloig and published by Philippe Apeloig and printed by Serica. It is dated 2008 and we acquired it in 2013. Its medium is screenprint on coated paper. It is a part of the Drawings, Prints, and Graphic Design department.

 The French designer Philippe Apeloig made this poster to promote a 2008 exhibition of his own poster designs held at the Espace Topographie de l’art in Paris. To create the typography, Apeloig produced pages of computer-generated punctuation—such as periods, hyphens, and backslashes—and manipulated them into individual text forms by varying the spacing. Using a palette of only red, white, and black, Apeloig layered these text forms in different scales so that the title of the exhibition, Vivo in Typo, emerges as if it were woven together.

 This object was depositor: Philippe Apeloig. It is credited "Gift of Philippe Apeloig."

repository: emchun/poster-design

repo_name: emchun/poster-design


text:
---
layout: post
title: Send Emails with a Custom Domain Using SendGrid and Rails
date: 2017-08-09 18:30:00 +5:30 GMT
share: y
sweetcode: true
---

SendGrid is a cloud-based email delivery service that assists businesses with email delivery. And that probably is the most boring definition ever, considering the powerhouse of tools that SendGrid provides.

With SendGrid you don't just get to send emails, you get to monitor link clicks, un-subscription rates, geographical tracking and a lot more. In this tutorial I'll walk you through building your own email sending service on Rails with SendGrid.

The first thought that probably pops up into your mind is why SendGrid? Believe me, you'll figure out for yourself by the end of this tutorial.

Let's first create a Rails project via command line. I'll be using Rails 5 here.

```
rails new sendemail
```

_(How [creative of a name](http://i.imgur.com/y3dhJFQ.jpg?1) is sendemail though?)_

We would like for users to sign up and then maintain a personal profile of all the emails they send. So, we'll be adding the [Devise](https://github.com/plataformatec/devise) gem here. I won't be covering how to add Gems and customize Devise in this tutorial, but here's a few [sources](https://github.com/plataformatec/devise/blob/master/README.md) [that](http://guides.railsgirls.com/devise) [cover](https://launchschool.com/blog/how-to-use-devise-in-rails-for-authentication) it.

We'll then create a scaffold called `email`. The model attributes will be the `receiver` field, `subject` field and `body` field. Everything necessary to create an email! Oh, we'll also be adding a `user_id` mapping column, which will be used to query `emails belonging to an user`.

```
rails g scaffold email user_id:integer:index receiver:string subject:string body:text
```

Here's where the fun starts.

To send emails from your Rails application, you need to,

1. Add email provider credentials to your devlopment.rb (production.rb if in production environment). I'll be using Gmail here.

_ProTip: If you've enabled two factor authentication on your account, you need to create an [app password](https://support.google.com/accounts/answer/185833) to bypass it._

It'll look something like this,

```
config.action_mailer.smtp_settings = {
   :address => 'smtp.gmail.com',
   :port => 587,
   :user_name =><EMAIL_ADDRESS>   :password => 'frsghrjdyquftlsh',
   :authentication => :plain,
   :enable_starttls_auto => true
}
```

2. Add this to your `app/mailers/application_mailer.rb`. This will send the email.

```
def send_it(email)
  @email = email

  mail(
    from: email.user.email,
    to: email.receiver,
    subject: email.subject
  )
end
```

_Notice that I have specified the from address as well. Also, I have NOT provided a default_from in ApplicationMailer._

You'll also have to create a `app/views/application_mailer/send_it.html.erb` and add this to it,

```
<%= @email.body %>
```

This prints the body of the email into what is being sent out.

3. I've also added the method to trigger email sending to our model. Here's how it looks,

```
class Email < ApplicationRecord
  belongs_to :user
  validates_format_of :receiver, with: /\A([^@\s]+)@((?:[-a-z0-9]+\.)+[a-z]{2,})\z/i

  after_create :deliver
  after_update :deliver

  def deliver
    ApplicationMailer.send_it(self).deliver_now
  end
end
```

I've created simple `after_create` and `after_update` callbacks that send the email.

Let's test and see if this works.

I'll create an user by signing up with `swaathi @ skcript` email address. And then send an email to `swaathikakarla @ gmail`.

Let's see what happens.

Hey! It got sent from `swaathi16 @ gmail`. But, the `from` section was set to `swaathi @ skcript`. Why did this happen?

It's because of the credentials in `development.rb`. We added Gmail credentials belonging to `swaathi16 @ gmail`. So, no matter what you over-ride it'll be sent from the email that's tied to the credentials.

This works perfect when you don't want someone masking your identity.  But, with SendGrid you can send emails as other addresses - but it's not as scary as it sounds - it'll be appended with an 'via Domain' text. And, any replies to the email will automatically be send to the address itself.

Let's see how that works!

1. Sing up on SendGrid
Headover to the signup page of [SendGrid](https://sendgrid.com/signup), and to test things out select the FreeTrial. This allows you to send 40,0000 emails per day for 30 days.

2. Head over to the Whilelabels section
After that visit the Settings section in the sidebar and then click on the [Whitelables link](https://app.sendgrid.com/settings/whitelabel) from the dropdown.

_P.S, I wrote this article for [Sweetcode](https://sweetcode.io/), a place for techies to share what they know._


text:
# KerlHash -> New()
New returns a new Kerl.
> **Important note:** This API is currently in Beta and is subject to change. Use of these APIs in production applications is not supported.

## Output

| Return type     | Description |
|:---------------|:--------|
| IotaHash |  |

gha_language: Go


text:
# ManyBabies website

Visible at http://manybabies.stanford.edu.

To run locally:
1. Install jekyll and bundle
2. Clone repository
3. ``cd`` into directory
4. Execute ``bundle exec jekyll server``

To modify:
- ``_config.yaml`` has many of the basic configuration options (e.g., social media and links to other pages)
- ``index.html`` is the main page, but currently (mostly) generated from...
- ``_posts/`` has blog posts, the most recent of which are featured in index.html
- ``about.md`` and other pages can be created in the root directory

gha_license_id: NOASSERTION


text:
# Cush

## Facts:

Cush was the oldest son of Noah’s son Ham. He was also the ancestor of Nimrod. Two of his brothers were named Egypt and Canaan.

* In Old Testament times, “Cush” was the name of a large region of land south of Israel. It is probable that the land was named after Ham’s son Cush.
* The ancient region of Cush covered an area of land that may have, at different times, included parts of the modern-day countries of Sudan, Egypt, Ethiopia, and possibly, Saudi Arabia.
* Another man named Cush is mentioned in the Psalms. He was a Benjamite.

(Translation suggestions: [How to Translate Names](../../translate/translate-names))

(See also: [Arabia](../names/arabia.md), [Canaan](../names/canaan.md), [Egypt](../names/egypt.md), [Ethiopia](../names/ethiopia.md))

## Bible References:

* [1 Chronicles 1:8-10](rc://en/tn/help/1ch/01/08)
* [Ezekiel 29:10](rc://en/tn/help/ezk/29/10)
* [Genesis 2:13](rc://en/tn/help/gen/02/13)
* [Genesis 10:6-7](rc://en/tn/help/gen/10/06)
* [Jeremiah 13:23](rc://en/tn/help/jer/13/23)

## Word Data:

* Strong’s: [[H3568]], [[H3569]], [[H3570]], 3571

repository: mandolyte/uw-obsidian

repo_name: mandolyte/uw-obsidian


text:
# llswift proof-of-concept

[![swift-ast 0.3.5](https://img.shields.io/badge/swift‐ast-0.3.5-C70025.svg)](https://github.com/yanagiba/swift-ast)
[![Travis CI Status](https://api.travis-ci.org/ryuichis/llswift-poc.svg?branch=master)](https://travis-ci.org/ryuichis/llswift-poc)

A proof-of-concept that parses Swift code,
generates `.ll` file,
and then either runs it directly with `lli`, or generates executable binary with `clang`.

It will be part of [Yanagiba Project](http://yanagiba.org). If you are interested in working with me on this, send me an email to<EMAIL_ADDRESS>Look forward to it!

repository: ryuichis/llswift-poc

repo_name: ryuichis/llswift-poc


text:
# IE 9 Bugs

### Your bug not here? Check out [common IE bugs](all.md)

#### CSS Selectors Limit

This is one of our favourites. IE9 (and IE 8) has various limits in CSS parsing:
* one CSS file can have up to 4096 selectors
* there can be up to 30 (or 31 depending on who you ask) css files included in a page. No more!


text:
### Handle large Parquet files unloaded from Redshift

When I cleaned up some old data from Redshift , specifically [Snowplow](https://snowplowanalytics.com/) data I ended up wanting to access the information locally on my Mac, long after I exported it. This turned out to be a bit tricky.

When I used the (awesome) sqlite3 library, it failed because of the timestamp
data format. Other tools couln't handle the size because they tried to keep all in memory. Instead I switched to use the Go library at
`https://github.com/xitongsys/parquet-go` and wrote some routines.

### INPUT Data

To export .parquet data from Redshift and store it on s3 you might do something like this:

```
unload ('select * from atomic.events
where etl_tstamp < ''2020-07-01 00:00:00.000''')
   to 's3://my-snowplow-dump-bucket/00dump/evt2020_'
   PARQUET
   iam_role 'arn:aws:iam::123456789012:role/RedshiftUnLoadRole';`
```
Ending up with a set of files like

`evt2020_0000_part_00.parquet,
evt2020_0001_part_00.parquet`

Each with the size up to the max size 6.2 GB .

### Hands on

If you copied these files down to your local machine you can generate a
model file for their structure by cloning this repo and doing:

```go run model_generator.go -file=./evt2020_0000_part_00.parquet > model.go```

then try to read the content by doing:

``` go run try_the_model.go model.go -file=./evt2020_0000_part_00.parquet ```

In this sample code I do a little conversion to make the timestamp fields a bit easier to read.

### Now the fun starts

From this you should be able to build out your code to export into other databases or file formats. Go to town.


text:
# Futures
The library provides facilities to obtain values that are returned and to catch exceptions that are thrown by asynchronous tasks.

## Requirements
- iOS 9.0+
- Swift 4.2

## Usage

### Sample 1
```swift
DispatchQueue.global(qos: .background).async {
	return try String(contentsOf: URL(string: "https://google.com")!)
}.then { result in
	print(result)
}.catch {error in
	print(error)
}.finally {
	//executes after 'then' and 'catch'
}
```

### Sample 2
```swift
func doSomeWork() -> Future<Int> {
	let promise = Promise<Int>()
	DispatchQueue.global(qos: .background).async {
		try! promise.fulfill(10)
	}
	return promise.future
}

doSomeWork().then { result in
	print(result)
}
```

### Sample 3
```swift
DispatchQueue.global(qos: .utility).async {
	return 1
}.then { result in
	return result + 1
}.then { result in
	//Future<Future<T>> will be unwrapped to Future<T>
	return DispatchQueue.global(qos: .utility).async {
		return result + 1
	}
}.then { result in
	print(result)
}
```

repository: mrdepth/Futures

repo_name: mrdepth/Futures


text:
Sometimes, you just have to search a USD file for some text. Most of
the time, if you're looking for something specific, you can open
up usdview and dig around until you find what you need. But if, for
example, you need to find every relationship containing the phrase "foo"
or "bar", you're in for some trouble.

USD files can refer anywhere on-disk. In studio pipelines, it's very
common for Asset paths to be URIs referring to completely different
folder locations or even refer to files in a database.

To search USD files, you must also traverse its dependencies (wherever
those dependencies may be) and search the dependency's contents.
usd-search a command-line utility / Python module that searches through
USD files, recursively.

## Features
- Search through USD Layers, even crate files
- Optionally searches through Asset paths
- Supports regex matching

## How To Use
```bash
PYTHONPATH=$USD_INSTALL_ROOT/lib/python:$PWD/python:$PYTHONPATH ./bin/usd-search foo /some/usd/file.usda
```

## Requirements
- USD must be importable in Python
- (Optional) Your environment should be set up however it needs to be in
order to resolve paths correctly

repository: ColinKennedy/USD-Cookbook

gha_language: Mathematica

repo_name: ColinKennedy/USD-Cookbook


text:
# Website for [Garden Serv](http://www.gardenserv.com.br/)

This website was created to help my cousin on his new business.

The template was based on [Stylish Portfolio](http://startbootstrap.com/template-overviews/stylish-portfolio/).

## Copyright and License

Copyright 2015 Garden Serv. You can use everything, except the logo.

Copyright 2013-2015 Iron Summit Media Strategies, LLC. Code released under the [Apache 2.0](https://github.com/IronSummitMedia/startbootstrap-stylish-portfolio/blob/gh-pages/LICENSE) license.

repository: mfilipe/gardenserv

repo_name: mfilipe/gardenserv