text:
Forms

## Table of Contents

* [About the Project](#about-the-project)
  * [Built With](#built-with)
* [Required Installations](#required-installations)
* [Instalation of This App](#instalation)
* [License](#license)
* [Contact](#contact)
* [Acknowledgements](#acknowledgements)

## About The Project

This project will teach you the multiple ways to build a form in rails that can send parameters to a controller so that it can build an instance of a model with those parameters.

Full task description:  Odin Project.

### Built With

* Git
* Ruby
* Ruby on Rails
* HTML
* CSS

## Required Installations

If you want a copy of this project running on your machine you have to install:

* Ruby 2.6.4
* Gem 3.0.3
* Rails 6.0.1
* Bundler 2.0.2

 Installation Instructions

## Instalation of This App

Once you have installed the requiered packages shown on the [Required Installations](#required-installations), proceed with the following steps

Clone the Repository,

```Shell
your@pc:~$ git clone https://github.com/FelipeEnne/Re-former/tree/development
```

Move to the downloaded folder

```Shell
your@pc:~$ cd Re-former
```

install gems

```Shell
your@pc:~$ bundle install --without production
```

migrate the database

```Shell
your@pc:~$ rails db:migrate
```

Run the app in a local server:

```Shell
your@pc:~$ rails server
```

Then, go to [http://localhost:3000/users/new](http://localhost:3000/users/new)

## License

Distributed under the MIT License. See `LICENSE` for more information.

## Contact
* Felipe Enne -<EMAIL_ADDRESS>| [Github Account https://github.com/FelipeEnne](https://github.com/FelipeEnne)
* MaryAnn Chukwuka -<EMAIL_ADDRESS>| [Github Account https://github.com/adaorachi](https://github.com/adaorachi)

## Acknowledgements

*  Microverse  and  Odin Project .

repository: FelipeEnne/Re-former

branch_name: refs/heads/development

gha_language: Ruby

repo_name: FelipeEnne/Re-former


text:
# setup

- [setup](#setup)
  - [Basics](#basics)
  - [MacOS configuration](#macos-configuration)
  - [Ubuntu configuration](#ubuntu-configuration)
    - [Installation](#installation)
    - [System configuration](#system-configuration)
    - [Minimal `ssh` setup](#minimal-ssh-setup)
    - [Install software & configurations](#install-software--configurations)
      - [Package list](#package-list)
      - [`kitty`](#kitty)
      - [`zsh`](#zsh)
      - [`git`](#git)
      - [`tmux`](#tmux)
      - [Visual Studio Code](#visual-studio-code)
    - [FAQ](#faq)
      - [Bluetooth devices do not work correctly on dual boot (Windows + Linux)](#bluetooth-devices-do-not-work-correctly-on-dual-boot-windows--linux)

## Basics

1. Clone this repo to `~/setup`. This guide will assume this location in most cases.
2. Initialize submodules `git submodule update --init --recursive`.
3. Follow instructions depending on the platform.

## MacOS configuration

Install [Homebrew](https://brew.sh/) and run `brew bundle`.

Setup kitty just like on Ubuntu.

Setup zsh just like on Ubuntu.

Install [nvm](https://github.com/nvm-sh/nvm) and Node using `nvm`: `nvm install node`.

Setup itsycal, format: `E dd/MM/YYYY (MMM)`

## Ubuntu configuration

### Installation

**Before** you install Ubuntu on a new machine:

- Ensure you have an Ethernet cable. WiFi may not work initially without updates/proper drivers.

Install Ubuntu with recommended settings, encrypt home if possible. If you'll skip encryption you can do it later, just follow [this guide on www.howtogeek.com](https://www.howtogeek.com/116032/how-to-encrypt-your-home-folder-after-installing-ubuntu/).

Once installed, update packages:

```Bash
sudo apt update         # update package index
sudo apt upgrade        # download & install packages
sudo do-release-upgrade # upgrade to newest release
```

### System configuration

1. Remap Caps Lock to Ctrl using "Tweaks" tool (from `gnome-tweak-tool` package)
2. Install better fonts like [nerd fonts](https://github.com/ryanoasis/nerd-fonts/)
    1. Download them and place in `~/.fonts`
    2. Run `fc-cache -fv` to rebuild the font cache
3. Create SSH key for `git`, `ssh` etc.

    ```sh
      ssh-keygen -t rsa -b 4096 -C<EMAIL_ADDRESS># generate keys
      eval "$(ssh-agent -s)"  # start ssh-agent in the background
      ssh-add ~/.ssh/id_rsa  # add key
    ```

4. Unbind SUPER + P since it may cause problems when you are used to MacOS. [Follow the steps from Stack](https://unix.stackexchange.com/q/208682):
    1. Install `dconf-editor`.
    2. Go to `/org/gnome/mutter/keybindings/switch-monitor`
    3. Disable "Use default value".
    4. Delete: `'<Super>p',` from the input below.

5. On dekstops, Settings > Power > Blank screen option: set to Never. Alternatively run in the terminal: `

6. Set org.gnome.desktop.interface.clock-show-weekday to true

### Minimal `ssh` setup

1. [`zsh`](#zsh)
2. [`git`](#git)

### Install software & configurations

#### Install `apt` dependencies.

```Bash
ubuntu_apt_list=(
  clementine # audio player
  cmake
  dconf-monitor # configuration tool, see 'System configuration'
  fzf # fuzzy search
  gimp
  git
  gitk # graphical history viewer for git
  gnome-tweak-package # tweaks for Gnome, see 'System configuration'
  gpick # color picker
  htop # fancy top
  imagemagick # image manipulation toolset
  kitty # terminal emulator
  moreutils # chronic, vipe, parallel and more (https://joeyh.name/code/moreutils/)
  net-tools # ifconfig (deprecated, use ip tool, for IP: ip -c a)
  python3-pip
  python3-venv
  silversearcher-ag # faster alternative for grep
)

sudo apt install ${ubuntu_apt_list[*]}
```

#### Install `snap` dependencies.

```Bash
snap_install_list=(
  discord
  spotify
)

sudo snap install ${snap_install_list[*]}

# classic snaps:
#   code # Visual Studio Code
```

#### Install Rust

Follow the instructions on [https://rustup.rs/](https://rustup.rs/)

```Bash
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
```

#### Install exfat drivers

[How to mount an exfat drive on Ubuntu via fosslinux.com](https://www.fosslinux.com/17725/how-to-mount-an-exfat-drive-on-ubuntu.htm):

```Bash
sudo add-apt-repository universe
sudo apt update
sudo apt install exfat-fuse exfat-utils
```

#### Package list

| Package            | Description                             | URL                                         |
| ------------------ | --------------------------------------- | ------------------------------------------- |
| alacritty          | terminal emulator                       | https://github.com/jwilm/alacritty          |
| curl               |                                         |                                             |
| git                |                                         |                                             |
| gnome-tweak-tool   | system tweaks: keyboard re-mapping etc. |                                             |
| neovim             |                                         |                                             |
| nodejs             |                                         | https://github.com/nodesource/distributions |
| tmux               |                                         |                                             |
| virtual box        |                                         |                                             |
| virtualenv         |                                         |                                             |
| visual studio code |                                         |                                             |
| xclip              | clipboard support                       |                                             |
| imagemagick        | image toolset, used by kitty            |                                             |
| zsh                | shell                                   |                                             |

### `nvim`

Create symlink:

```Bash
ln -s /home/kamil/code/setup/config/nvim ~/.config/nvim
```

Install `vim-plug` for plugins:

```Bash
# Install vim-plug
curl -fLo ~/.local/share/nvim/site/autoload/plug.vim --create-dirs \
    https://raw.githubusercontent.com/junegunn/vim-plug/master/plug.vim
```

Run `nvim`, run `:PlugInstall`. Close `nvim`. Ready for use.

#### `kitty`

Install `kitty`:

```Bash
sudo apt install kitty

# link the configurations
ln -sv /home/kamil/code/setup/config/kitty ~/.config/kitty
ln -sv /home/kamil/code/setup/config/kitty/dracula-kitty ~/.config/kitty
```

#### `zsh`

```Bash
sudo apt install zsh
chsh -s /usr/bin/zsh $(whoami)
git clone https://github.com/tarjoilija/zgen.git "${HOME}/.zgen"
ln -s /home/$(whoami)/setup/config/zsh/.zshrc ~/.zshrc
```

#### `git`

Install latest `git` using [official instructions](https://git-scm.com/download/).

On Ubuntu you probably want to add apt repository:

```Bash
sudo add-apt-repository ppa:git-core/ppa
sudo apt update
sudo apt install git
```

Set up your identity:

```Bash
git config --global user.name "first and last name"
git config --global user.email "email address"
```

Open `~/.gitconfig` and append the following:

```ini
[include]
  path = /home/<USER>/setup/config/git/git
```

Setup password caching in `git`. Follow GitHub official guide [here](https://help.github.com/en/github/using-git/caching-your-github-password-in-git), likely you want:

```Bash
# 1h = 60 * 60s = 3600s
# 8h = 8 * 60 * 60s = 28800s
git config --global credential.helper 'cache --timeout=28800'
```

Setup pager, for `zsh` the following is fine:

```Bash
git config --global core.pager "less -F -X"
```

#### `tmux`

Install `tmux` and symlink the configurations:

```Bash
ln -s /home/$(whoami)/setup/config/tmux/tmux.conf ~/.tmux.conf
```

#### Visual Studio Code

```Bash
cd ./setup/config

# -- Visual Studio Code
# Make symlinks
# on mac: ~/Library/Application Support/Code/User/settings.json etc
ln -s $(pwd)/vscode/settings.json ~/.config/Code/User/settings.json
ln -s $(pwd)/vscode/keybindings.json ~/.config/Code/User/keybindings.json
ln -s $(pwd)/vscode/snippets ~/.config/Code/User/snippets
# Install extensions
cat vscode/extensions.txt | xargs -n1 code --install-extension
```

### FAQ

#### Ubuntu changes Windows clock and vice versa

Ubuntu stores clock time in UTC, Windows doesn't. You have to fix one of them, to force Ubuntu to use local time use:

```Bash
timedatectl set-local-rtc 1 --adjust-system-clock
```

#### Bluetooth devices do not work correctly on dual boot (Windows + Linux)

Bluetooth pairing generates unique key during pairing and re-pairing on another system overrides this key, after reboot keys do not match. See the guide for fix: https://unix.stackexchange.com/q/255509

license: MIT

repository: sienkiewiczkm/setup

branch_name: refs/heads/main

extension: md

license_type: permissive

repo_name: sienkiewiczkm/setup


text:
{:tags ("clean code" "comments"), :title "Code Comments are Lies", :date "2015-02-02"}
-----
### Post Update

This post generated some interesting comments on the [code project site](http://www.codeproject.com/script/Articles/ArticleVersion.aspx?waid=158130&aid=872073).
Here are some things to note before you continue.

* This post is aimed at higher level languages like C# and Java. Please comment your FORTRAN etc.
* The best code that you can write is code that is maintainable as this is the true cost of software. If this requires you to add a comment please do that. I also do that.
* A comment as an excuse for writing bad code, is unacceptable.

Please read the comments on code project. There are some valuable usecases there.

 *"My comments are informative and well written."* *"How else would people understand this awesome piece of efficient code?"*

Although there are use cases for comments, I am going to tell you that comments are like little poops that you leave all over your code for you fellow devs to step in.

**Let's look at some truths about comments**

* They can state the obvious, thus being redundant.

// The Persons First Name
public string FirstName { get; set;}

* Having comments scattered throughout your code, explaining functionality in detail, makes it hard to read code fluently. This means that extra effort and concentration is needed to read between the comment lines. The intent does not jump out at you.

>**Comments Become Lies**

* Comments rarely get updated when the code changes. This means that overtime the functionality documented by the comments, drifts away from what is actually happening. This causes confusion and again makes the comments worthless.

* ToDo Comments. They usually don't get done.

* Comments get copied with code, and as the code changes again the comments don't. First of all you should think twice when you need to copy code, and if you have a great reason you have to make real sure to update (or even just remove) the comments made.

* Finally, like Uncle Bob says in his book [Clean Code](http://www.amazon.com/Clean-Code-Handbook-Software-Craftsmanship/dp/0132350882), if you need to write a comment you have failed to express the intent of the code. You want your code to read well, to be self-documenting so that you do not need to document it.

I have written all these types of comments.

>**I need to write a comment about my code, thus I have failed**

This is another bold statement. But it is exactly what I tell myself when I need to write a comment. It forces me to rethink my code and usually rewrite it to be more readable and maintainable. And if I still need to write the comment, it reminds me that I still have room for improvement. I can decide to leave out the comment and not feel bad, but I write it in hopes that it might aid me or whoever comes after me to understand my poorly expressed code.

**So What Can I Do To Fix This Problem?**

First of all, **Name Things Well**. Well named classes and functions and properties help eliminate confusion and clarifies code intent. A function should do what it says, a class be what it is named.

Functions should do one thing, it should do it well and it should do it only. Following the **Single Responsibility Principle** will eliminate most reasons to comment code.

But I hear you shouting from afar: *"Pieter, not all comments can be bad"*. This is true, although I still maintain comments are bad.

**Some Good Comments**

* Comments that you write to document badly written legacy code. It is not always possible to refactor code, so in these cases help yourself and others to document things that might have bad side effects.

* Comments documenting framework limitations.

public FooConstructor()
{
	//EntityFramework requires a default constructor
}

* ToDo comments can be a great way to remind yourself of something that you need to do. Only use them while working on feature branches. They should be implemented or removed (placed on your backlog as a ticket) once you merge back to your main branch.

* In the rare case that you need to write optimised code that might be confusing and not easily readable. This is a rare case for most of us.

* API documenting code. In some languages you can add comments to classes, methods and properties and these will get rendered by your IDE or can be exported to actual documents. This should only live on public facing code and has no place inside your application where some random user will not use it.

Remember you want to write readable, self-documenting, maintainable code for yourself and others. So stop writing that comment and be awesome instead.

If you can think of other examples please add them in the comments and if you liked this post share it with the hashtag `#badcomment`. For those of you that are still angry, [here are some funny comments.](http://stackoverflow.com/questions/184618/what-is-the-best-comment-in-source-code-you-have-ever-encountered)

CodeProject

branch_name: refs/heads/master

gha_language: CSS

gha_license_id: MIT


text:
---
layout: post
title: Packaging Attempt
---

Today I went over the packaging process with my mentors.

I used my Ubuntu 18.04 VirtualBox to go through the process here: [https://www.freecadweb.org/wiki/GitBuildpackage](https://www.freecadweb.org/wiki/GitBuildpackage)

My mentors told me about dot files, which I understand to be a type of hidden file that contains configuration settings. One of the questions I had was where to put the dotfiles. I found the answer to that:

Dot files should be stored in the home directory.

Once the dot files were in my home directory I used
`gbp buildpackage -us -uc`
to begin the packaging process.

Unfortunately I ran into some errors during this step. After investigating the cause by using
`git-pbuilder login --save-after-login` and `cat /etc/apt/sources.list`
to check my distribution configuration we found that this:

`deb http://mirrors.kernel.org/ubuntu/ bionic main restricted universe multiverse
      #deb-src http://mirrors.kernel.org/ubuntu/ bionic main restricted universe multiverse`

was my sources.list. I needed to be using the Debian unstable distribution. Further attempts to change this file did not succeed.

Eventually, we decided that I should use a Debian Test distribution and set it as the unstable version. I downloaded and set up the virtual machine for my next attempt.


text:
Euro2016
=================

UEFA EURO 2016 results for hackers.

[http://api.football-data.org/index](http://api.football-data.org/index)

Data Source: [http://api.football-data.org/v1/soccerseasons/424](http://api.football-data.org/v1/soccerseasons/424)

<img src="screenshot.png" />

### Installing

After cloning the repository:

```
composer install
```

**Important: You should set date.timezone on your php.ini**

Usage
------------

`php app.php fixtures`

Default argument is today. Also **current**, **finished** and **all** are supported arguments.
With argument **all** you can specified team you want to see schedule/results. For example:

`php app.php fixtures all -t Turkey`

You can always add **euro2016** alias on your .bashrc or .zshrc (That's how I use)

`alias euro2016='php projects/euro2016/app.php fixtures'`

and use like this:

`euro2016 current`

License
-------------

[MIT License](http://emir.mit-license.org/)

gha_language: PHP


text:
### [CVE-2020-13468](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-13468)
![](https://img.shields.io/static/v1?label=Product&message=n%2Fa&color=blue)
![](https://img.shields.io/static/v1?label=Version&message=n%2Fa&color=blue)
![](https://img.shields.io/static/v1?label=Vulnerability&message=n%2Fa&color=brighgreen)

### Description

Gigadevice GD32F130 devices allow physical attackers to escalate their debug interface permissions via fault injection into inter-IC bonding wires (which have insufficient physical protection).

### POC

#### Reference
- https://www.usenix.org/system/files/woot20-paper-obermaier.pdf

#### Github
No PoCs found on GitHub currently.

repository: trickest/cve

gha_language: HTML

repo_name: trickest/cve


text:
## Installation instructions for core lightning and lspd
### Requirements
- CLN (compiled with developer mode on)
- lspd
- lspd plugin for cln
- postgresql

### Installation
#### CLN
Follow compilation steps for CLN [here](https://github.com/ElementsProject/lightning/blob/master/doc/getting-started/getting-started/installation.md) to enable developer mode.

#### lspd
Needs to be built from source:
```
git clone https://github.com/breez/lspd
cd lspd
go build .                                # compile lspd
go build -o lspd_plugin ./cln_plugin/cmd  # compile lspd cln plugin
```

#### Postgresql
Lspd supports postgresql backend. To create database and new role to access it on your postgres server use:
##### Postgresql server
```
CREATE ROLE <username>;
ALTER ROLE <username> WITH NOSUPERUSER INHERIT NOCREATEROLE NOCREATEDB LOGIN NOREPLICATION NOBYPASSRLS PASSWORD '<password>';
CREATE DATABASE <dbname> WITH TEMPLATE = template0 ENCODING = 'UTF8' LC_COLLATE = 'en_US.UTF-8' LC_CTYPE = 'en_US.UTF-8';
ALTER DATABASE <dbname> OWNER TO <username>;
```
##### RDS on AWS
```
CREATE ROLE <username>;
ALTER ROLE <username> WITH INHERIT NOCREATEROLE NOCREATEDB LOGIN NOBYPASSRLS PASSWORD '<password>';
CREATE DATABASE <dbname> WITH TEMPLATE = template0 ENCODING = 'UTF8' LC_COLLATE = 'en_US.UTF-8' LC_CTYPE = 'en_US.UTF-8';
ALTER DATABASE <dbname> OWNER TO <username>;
```

### Configuration
1. Create a random token (for instance using the command `openssl rand -base64 48`, or `./lspd genkey`)
1. Define the environment variables as described in [sample.env](./sample.env). If `CERTMAGIC_DOMAIN` is defined, certificate for this domain is automatically obtained and renewed from Let's Encrypt. In this case, the port needs to be 443. If `CERTMAGIC_DOMAIN` is not defined, lspd needs to run behind a reverse proxy like treafik or nginx.

ENV variables:
- `LISTEN_ADDRESS` defines the host:port for the lspd grpc server
- `CERTMAGIC_DOMAIN` domain on which lspd will be accessible
- `DATABASE_URL` postgresql db url
- `AWS_REGION` AWS region for SES emailing
- `AWS_ACCESS_KEY_ID` API key for SES emailing
- `AWS_SECRET_ACCESS_KEY`API secret for SES emailing
- `MEMPOOL_API_BASE_URL` uses fee estimation for opening new channels (default: https://mempool.space/api/v1/)
- `MEMPOOL_PRIORITY` priority with which open new channels using mempool api
  (options: minimum, economy, hour, halfhour, fastest) (default: economy)
- `NODES` which nodes are used by lspd (see below for example, multiple nodes supported and more examples can be found in [sample.env](../sample.env))

Example of NODES variable:
```
NODES='[ { "name": "${LSPName}", "nodePubkey": "$PUBKEY", "lspdPrivateKey": "$LSPD_PRIVATE_KEY", "token": "$TOKEN", "host": "$EXTERNAL_IP:8888", "publicChannelAmount": "1000183", "channelAmount": "100000", "channelPrivate": false, "targetConf": "6", "minConfs": "6", "minHtlcMsat": "600", "baseFeeMsat": "1000", "feeRate": "0.000001", "timeLockDelta": "144", "channelFeePermyriad": "40", "channelMinimumFeeMsat": "2000000", "additionalChannelCapacity": "100000", "maxInactiveDuration": "3888000",  "cln": { "pluginAddress": "<IP_ADDRESS>:12312", "socketPath": "/home/lightning/.lightning/bitcoin/lightning-rpc" } } ]'
```

### Running lspd on CLN
In order to run lspd on top of CLN, you need to run the lspd process and run cln with the provided cln plugin. You also need lightningd compiled with developer mode on (`./configure --enable-developer`)

1. Run cln with the following options set:
    - `--plugin=/path/to/lspd_plugin`: to use lspd as plugin
    - `--max-concurrent-htlcs=30`: In order to use zero reserve channels on the client side, (local max_accepted_htlcs + remote max_accepted_htlcs + 2) * dust limit must be lower than the channel capacity. Reduce max-concurrent-htlcs or increase channel capacity accordingly.
    - `--dev-allowdustreserve=true`: In order to allow zero reserve on the client side (requires developer mode turned on)
    - `--allow-deprecated-apis=true`: lspd currently uses a deprecated api, so needs this flag set.
    - `--lsp-listen=<IP_ADDRESS>:<port>`: Set on which port the lspd_plugin will listen for lspd communication, must be the same port that is used in pluginAddress parameter in NODES env variable.
1. Run lspd

### Final step
1. Share with Breez the TOKEN and the LISTEN_ADDRESS you've defined (send to contact@breez.technology)

repository: breez/lspd

gha_language: Go

repo_name: breez/lspd


text:
---
layout: post
title: AnimatedCluster pan related bug... fixed !!!
date: 2013-02-08 18:59
tags:
- animatedcluster
- gis
- openlayers
---
If you regularly follow this blog and are web mapping developer that works with OpenLayers, (too much coincidences???) probably you know about the the Animated marker cluster strategy for OpenLayers I created some time ago.

Unfortunally, the last version (v0.2) has a ugly bug. The code works fine when you change the zoom level but clusters are not updated when you pan the map.

I'm happy to say right now I have uploaded a new version (v0.3) which fixes this bug on my GitHub repository. Basically, now the code controls if the action is a zoom change or a pan movement and updates and animates the clusters accordingly. That is, if you pan the map the clusters on the current level are recomputed.

Take into account this can cause the features where clustered in different clusters, so you can see how bubbles changes its position and number of features within it.

In addition, the demo page has been updated with the new version. Check it !!!

Thanks to all the great people that has sent me their experiences when using the AnimatedCluster !!!

branch_name: refs/heads/gatsby

gha_language: JavaScript


text:
---
title: Building on Multiple CPU Architectures
layout: en
permalink: /user/multi-cpu-architectures/
---

> This is an alpha stage of this feature and we are [eager to hear back from you](https://travis-ci.community/c/environments/multi-cpu-arch), both for `Arm`-based and `IBM`-based feedback. The definition keys used in the `.travis.yml` file may be further adapted on short notice.
{: .alpha}

> `IBM Power`, `IBM Z` and `Arm`-based building is only available for Open Source repositories (at both travis-ci.org and travis-ci.com). While available to all Open Source repositories, the concurrency available for multiple CPU arch-based jobs is limited during the alpha period.
> An attempt to run `IBM Power`, `IBM Z` and `Arm`-based builds for a private repository will result in a build run on standard, `AMD`-based infrastructure. For any commercial queries with regards to multi-arch builds before they are available, please [contact us](mailto:support@travis-ci.com).

If your code is used on multiple CPU architectures it probably should be tested on multiple CPU architectures. Travis CI can test on amd64, ppc64le (IBM Power CPUs), s390x (IBM Z CPUs) and arm64 (run on ARMv8 compliant CPUs) if the operating system is Linux.

## Default CPU Architecture

The default CPU architecture used in Travis CI builds is `amd64`. It is used when no `arch` key is present.

## Identifying CPU Architecture of Build Jobs

You can identify for which CPU architecture a build job is run via

- GUI
  - in the build job list, there's a specific label and architecture name based on `arch` tag value
  - in the build job view, the same specific label is displayed near the operating system identifier
- A default environmental variable printed out during your build job: `$TRAVIS_CPU_ARCH` (for a complete list of available default environmental variables please see our [Environment Variables - Default Environment Variables](https://docs.travis-ci.com/user/environment-variables/#default-environment-variables) documentation.

## Testing on Multiple CPU Architectures

To enable testing on multiple CPU architectures add the `arch` key to your `.travis.yml`:

```yaml
arch:
  - amd64
  - ppc64le
  - s390x
  - arm64
os: linux  # different CPU architectures are only supported on Linux
```
{: data-file=".travis.yml"}

If you are already using a [build matrix](/user/customizing-the-build/#build-matrix) to test multiple versions, the `arch` key also multiplies the matrix.

- The ppc64le (IBM Power) and s390x (IBM Z) build jobs are run in an LXD compliant Linux OS image.
- The arm64 CPU architecture build job is run in an LXD compliant Linux OS image.
- The default LXD image supported by Travis CI is Ubuntu Xenial 16.04 and by using `dist` you can select different supported LXD images. Also see our [CI Environment Overview - Virtualisation Environment vs Operating System](https://docs.travis-ci.com/user/reference/overview/#virtualisation-environment-vs-operating-system) documentation. The LXD host, on which LXD-based builds are run, is on Ubuntu 18.04.
- The amd64 CPU architecture build job currently runs as a regular VM and will be transitioned to an LXD compliant Linux OS image usage over time.

## Example Multi Architecture Build Matrix

Here’s an example of a `.travis.yml` file using the `arch` key to compile against `amd64`, `arm64`, `ppc64le` (IBM Power), `s390x` (IBM Z) under Linux and using C as the programming language.

```yaml
language: c

arch:
  - amd64
  - arm64
  - ppc64le
  - s390x

compiler:
  - gcc
  - clang

install: skip

script:
  - cd src
  - make all
```
{: data-file=".travis.yml"}

The `.travis.yml` file above creates a 2x4 [build matrix](/user/customizing-the-build/#build-matrix): compilers x each architecture.

There are many options available and using the `matrix.include` key is essential to include any specific entries. For example, this matrix would route builds to the arm64 and amd64 architecture environments:

```yaml
jobs:
  include:
   - os: linux
     arch: amd64
   - os: linux
     arch: arm64
```
{: data-file=".travis.yml"}

Similarly, this matrix would route builds to the `ppc64le` (IBM Power) and `s390x` (IBM Z) architecture environments:

```yaml
jobs:
  include:
   - os: linux
     arch: ppc64le
   - os: linux
     arch: s390x
```
{: data-file=".travis.yml"}

Please note, that explicitly included builds inherit the first value in an array:

```yaml
arch:
  - amd64
  - arm64
  - ppc64le
  - s390x
jobs:
  include:
   - os: linux
     env: LIB_PATH="/usr/bin/shared/x86_64/v1"
   - os: linux
     env: LIB_PATH="/usr/bin/shared/x86_64/v2"
```
{: data-file=".travis.yml"}

For example, the above `.travis.yml`, would result in running both jobs with the environmental variable LIB_PATH assigned different values being run only on `amd64` architecture.

## Using Docker in Multiple CPU Architecture-Based Builds within LXD Containers

It is possible to use Docker in multiple CPU architecture-based builds within an LXD container. You may need a specific CPU architecture compliant docker image as a base or ensure relevant libraries required by your build are added to your Dockerfile.

An example of building a docker image from a Dockerfile adjusted to arm64:

```yaml
arch: arm64
language: c
compiler: gcc
services:
  - docker
script: docker build -t my/test -f Dockerfile.arm64 .
```
{: data-file=".travis.yml"}

An example of running docker image:

```yaml
arch: arm64
services:
  - docker
script: docker run my/test #assuming docker image my/test is arm64v8 ready
```
{: data-file=".travis.yml"}

You can try it out also for `ppc64le` (IBM Power) and `s390x` (IBM Z) based docker builds, assuming all dependencies and/or a CPU architecture compliant base docker image are used.

You can also have a look at [Using Docker in Builds](user/docker/).

## Security and LXD Container

### Access to Privileged fs/Features (Apparmor)

> Due to security reasons, builds run in LXD containers will be denied access to privileged filesystems and paths - a privileged container with write access to e.g. /sys/kernel/debugfs might muddle an LXD host.

As a result, for instance a command in `.travis.yaml` like:
```yaml
sudo docker run --privileged --rm -t -v /sys/kernel/debug:/sys/kernel/debug:rw
```
{: data-file=".travis.yml"}

would result in an error.

Also have a look at the [Github issue relevant to the topic](https://github.com/lxc/lxd/issues/2661) and the [LXD apparmor setup](https://github.com/lxc/lxd/blob/master/lxd/apparmor/apparmor.go) for more details.

### System Calls Interception

If you run into a message like:

> System doesn't support syscall interception

It most probably means a system call interception is outside of the list of the ones considered to be safe (LXD can allow system call interception [if it's considered to be safe](https://github.com/lxc/lxd/blob/master/doc/syscall-interception.md)).

## Hugepages Support from within LXD Container

A build job can’t enable hugepages within an LXD container - this is something that may change in the future, yet it depends on potential Linux kernel changes, which is something that needs to be reviewed and developed.


text:
# Relación entre análisis de riesgo y toma de decisiones: lectura

* Importante: identificar precisamente fuentes de riesgo e incertidumbre que
  afectan decisiones
* AD estudia riesgo e incertidumbre en decisiones, diagnosticando, tratando de
  mitigarlo, transferirlo o evadirlo
* DA uses data modeling, probability, statistics and Data Science to analyse
  risks and uncertainty.

## Notes on risk and uncertainty

* Three main postures on risk and uncertainty:
    * there is no distinction
    * risk as a category of uncertainty
    * risk and uncertainty are different
* In this course:
    * **Uncertainty**: the outcome of a decision is undefined. Usually used in
      non-quantitative scenarios.
    * **Risk**: the outcome of a decision or event can be modelled by a
      probability distribution.
* Given a scenario of **uncertainty** the **risk** refers to the measure of the
  possible outcomes potentially produced by the situation and the way it affects
  the individual or organisation.

## Types of decisions

* **Deterministic**: results are known
* **Assuming risk**: the probability distribution of the outcome is known
* **Under uncertainty**: results and probability distribution are not known


text:
# Live App: https://damp-island-80230.herokuapp.com/

# Note Taker

This app allows users to type notes to themselves and save those notes for later using JSON storage through local servers. It uses a get function to display saved notes, post functions to upload new notes, and a delete function to remove stored notes once the user no longer wants them to be saved

## Table of Contents

* Installationg and Usage
* Dependencies and Code Snippet
* Languages used
* Author
* License
* Acknowledgements

### Installation and Usage

This app needs to be initialized from the terminal and thereafter is accesible from a browser through localhosts. Anybody who has access to the files can type "node server.js" in the terminal to initiate the server and then can access the app through the browser.

### Dependencies and Code Snippet

This app uses the following dependencies:
npm fs (for reading and writing files)
npm express (for setting up middleware to respond to http requests)
npm path (for routing the files and objects)

Here is an example of the code used to delete a saved note from the archives:

```
app.delete("api/notes/:id", function(req, res) {
    var note = req.params.id;

    fs.readFile(path.join(__dirname,"db/db.json"), function(err, data) {
        if(err) {
            console.log(err);
        }
        const result = JSON.parse(data);
        result.splice(note,1);

        fs.writeFile(path.join(__dirname,"db/db.json"), JSON.stringify(result), function(err, data) {
            if(err) throw err;
            res.json(result);
        })

    })
})
```

## Languages Used

* [HTML](https://developer.mozilla.org/en-US/docs/Web/HTML)
* [CSS](https://developer.mozilla.org/en-US/docs/Web/CSS)
* [Javascript](https://developer.mozilla.org/en-US/docs/Web/JavaScript)

## Authors

* Sam Poppe

- [Link to Portfolio Site](https://popsizzle.github.io/Portfolio/)
- [Link to Github](https://github.com/PopSizzle)
- [Link to LinkedIn](https://www.linkedin.com/in/sam-poppe-623281193/)

## License

This project is licensed under the MIT License

## Acknowledgments

* Shout out to Ana Medrano for help figuring out bugs in this app, and to Jerome Chenette, Kerwin Hy, and Mahisha Gunasekaran for teaching me the skills necessary to build this app.

repository: PopSizzle/NoteTaker

repo_name: PopSizzle/NoteTaker


text:
# Description
Machine folder context of the popular NGINX web server.

# Command Tips
Build your image locally
* zcmd build

Terminal into your image to tinker
* zcmd run-ct

Run the webserver so you can access from your browser
* docker run -p 8080:80 -d local/nginx1
  - Browser address ...
    * localhost:8080
  - Kill the running server ...
    * zcmd kac

# Interesting Links
* https://hub.docker.com/_/nginx
* https://github.com/nginxinc/docker-nginx

repository: bah-insignia/zcmd-demo-rudimentary

repo_name: bah-insignia/zcmd-demo-rudimentary


text:
# terraform-iac-usecases
Terraform Use Case Gallery

*Note: These are simple Terraform use cases for learning purpose and for work-environment configuration. DO NOT blindly use it for any production uses*

Also check **[Vagrant IaC Examples ](https://github.com/ginigangadharan/vagrant-iac-usecases)**.

## Installing Terraform

[Download](https://www.terraform.io/downloads.html) and [Install](https://learn.hashicorp.com/tutorials/terraform/install-cli) Terraform.

repository: carnells/terraform-iac-usecases

repo_name: carnells/terraform-iac-usecases


text:
# fs-scrollanimate

A simple JavaScript library to animate your HTML elements when they are scrolled into view. This library doesn't have any dependencies and can be used with any webpage.

You can view a demo [here](https://www.cssscript.com/demo/fade-slide-animation-scroll-fs/)

### CDN Links:

[https://cdn.jsdelivr.net/gh/FaisalST32/fs-scrollanimate<EMAIL_ADDRESS>
[https://cdn.jsdelivr.net/gh/FaisalST32/fs-scrollanimate<EMAIL_ADDRESS>

### NPM

npm i fs-scrollanimate

[https://www.npmjs.com/package/fs-scrollanimate](https://www.npmjs.com/package/fs-scrollanimate)

### Usage:

* To use this library, you can simply add a reference to the .js file at the end of the body of your html and the .css file in the head of your html page.

* To animate an element when it's scrolled into view, you need to add the 'fs-scroll' class to that element.

* Next you need to add the animation class to the same element. At the moment, following classes are supported:

  * **fs-fadein**: Gives the element a fade effect when it's scrolled into view.

  * **fs-slideup**: The element will slide up with a fade-in effect when scrolled into view.

  * **fs-slideright**: The element will slide right with a fade-in effect when scrolled into view.

  * **fs-slideleft**: The element will slide left with a fade-in effect when scrolled into view.

* Additionally you can add a timing class to the animation. At the moment, following class is supported:

  * **fs-slow**: Slows down the animation.

* As an example, if you want to add **slide-up** animation to an element with a slowed down effect, you will do the following:

    **
 This is a paragraph
**

* That's it. Your elements will be animated when they are scrolled into view.


text:
## Towards Multi-Label Image Retrieval for Remote Sensing

This repository contains the implementation of the methods described in:
#### Towards Multi-Label Image Retrieval for Remote Sensing
R. Imbriaco, C. Sebastian, E. Bondarev, P.H.N de With [IEEE](https://ieeexplore.ieee.org/document/9491804)

## Setup

The code provided in this repository assumes that the datasets have been already
acquired from their corresponding sources. Furthermore, additional steps are
necessary for utilizing BigEarthNet. BigEarthNet should be first pre-processed
using the scripts in  `src/common/preprocessing/prepare_splits`. These will
generate the corresponding LMDB objects used during training.

Afterward, the script `/src/common/preprocessing/retrieval.py` should be executed.
This generates the necessary ground truth files for testing BigEarthNet.

Packages employed in developing this project are listed on `requirements.txt`.

## Train

Training of a network is done by preparing a YAML configuration file like the
ones provided in `configs`. This contains all the necessary information for
training/testing a model including backbones, losses, learning rate,
augmentations, etc. Some of the functionalities included in this codebase
 are not covered in the original paper. Some of these are:

* Addition of attention modules.
* Convolutional backbones other than ResNet.
* Different part-based models.
* Embedding dimensionality compression.
* Usage of original BigEarthNet labels.

Be aware that functionalities/modules not covered in the original paper
have not been tested and as such no guarantee of them working as intended is made.

## Test

Testing is done similarly to training in that the same YAML configuration is used.
However, the 'mode' attribute should be changed to 'test'. This will run the
extraction and retrieval processes. Be aware that due to BigEarthNet's size
retrieval can take a long time and use large amounts of RAM memory.

Checkpoints used in our research are available below:

[Checkpoints](https://mega.nz/file/gJQHhQLJ#zux6Db4NCK__A5lGtydRitsUTq84WmlVD-RveZzmCDk)

repository: RImbriaco/OML

repo_name: RImbriaco/OML


text:
# osnoise
Utility to characterize parallel application scaling issues caused by effects including OS noise.

Quick start :

  (1) edit "makefile" and set the MPI C compiler and options

  (2) make

  (3) mpirun --bind-to core -np 1024  ./osnoise -c 3.0 -t 300 -x 100 -n 31

  (4) inspect output, including the reported parallel efficiency

Many factors can impact scaling of MPI parallel applications, including the
effective latency and/or bandwidth of the messaging layers, possible interference
from daemon activity, congestion in the network, and actual variations in computation
or communication speeds due to any combination of hardware and software effects.
We have found that a simple synthetic benchmark can be very useful when it comes to
identifying issues that impact scaling for a given system.  There is a long history
of similar work, including benchmarks such as P-SNAP (Performance and Architecture
Laboratory System Noise Activity Program, from Los Alamos National Laboratory),
fixed work-quantum (FWQ) and fixed time-quantum (FTW) benchmarks (used for example
by Lawrence Livermore National Laboratory as Sequoia benchmarks), and Netguage from
ETH, for operating system noise measurement.  The current project was developed
independently.  It uses a calibrated sequence of steps, where each step consists of
a compute phase followed by MPI communication.  The default communication pattern
is nearest-neighbor boundary exchange on a 2D Cartesian process grid.  One can
optionally choose MPI_Alltoall using column communicators on the 2D process grid,
or globally synchronizing MPI calls, MPI_Allreduce or MPI_Barrier,
after every computation step.  Message sizes can be specified at run time via command
line arguments.  The use of MPI at each step ensures that this benchmark is very
similar to many real-world simulations.  The MPI connections make the benchmark
very sensitive to daemon activity or to any source of inhomogeneity.  If any MPI
rank is slowed down at any stage of a given step, other MPI ranks will be impacted.
The resulting delays will be local for a nearest-neighbor communication pattern,
or global for a blocking collective communication pattern.

To build the executable, edit the makefile and set CC (normally mpicc) and optionally
set compiler options.  Then "make" should build the executable.  It is recommended to
use GNU compilers with an architecture flag that will generate hardware square-root
instructions if they are available  (for example, -march=broadwell, -march=znver2,
or -mcpu=power9, etc.).

Typical use of this benchmark would be to scan over a range of parameters, and map
out the parallel efficiency and the distribution of timing variations.  The scan
can be at a fixed scale or number of nodes, where the computation interval covers
a range such as {1.0 msec, 3.0 msec, 10.0 msec, 30.0 msec, 60.0 msec, 100.0 msec},
or the scan can be over the number of nodes for a fixed computation interval. This
second type of scan is a traditional "weak scaling" measurement.  Roughly speaking,
a nominal computation interval of ~1.0 msec corresponds to a rather fine-grained
parallel application : there will be either local or global communication at ~1.0
msec intervals.  Not all parallel applications exchange messages that frequently,
and it is often useful to map out the scaling behavior as a function of the
computation interval.

It is often desirable to place one MPI rank on each core, and use options like these :

mpirun --bind-to core -np 1024  ./osnoise -c 3.0 -t 300 -x 100 -n 31

Options :

flag argument

 -c  float : specifies the compute interval in msec (3.0 msec above)

 -t  int   : specifies the target measurement time in seconds (300 sec above)

 -x  int   : specifies the message size for exchange or alltoall (100 bytes above)

 -n  int   : specifies the number of histogram bins (31 bins above)

 -m  char  : specifies the communication method (-m [exchange, alltoall, allreduce, barrier])

 -k  char  : specifies the compute kernel (-k [sqrt, lut])

 -d        : requests a dump of all step times

 -b        : requests an added barrier every 100 steps

 -h        : prints a short help message

With a 3.0 msec time interval for computation, and 100 byte message for exchange,
the elapsed time should be totally dominated by computation, and the measurement
should complete in very close to the specified target time.  Daemon activity and
other sources of OS noise will result in a longer than expected overall elapsed time.
Generally speaking, sensitivity to daemon activity is highest when all cores are
used for computation, and when the time interval between communication events is in
the ~10 msec range or lower.

The default compute kernel computes the square-root of a pseudo-random number.  One
can optionally request a compute kernel that uses a look-up table to compute an
approximation to the exp() function (command-line option -k lut).  These two compute
kernels may behave differently.  The look-up table kernel is more sensitive to use
of caches, and may exhibit variable performance due to address-space layout
randomization.  In a large-scale parallel job, there may be cache-line aliasing
effects caused by randomized address assignments, and that can result in variations
in compute performance even though the amount of work is identical across ranks.
On most systems, the Linux kernel uses address-space randomization by default, but
one can launch the job with a helper that disables this feature, see below.  The
default compute kernel is less sensitive to cache and address assignments.

The code prints a lot of information, including a histogram of all step times, where
one step represents completion of one pair of {compute, communicate} stages on any
MPI rank.  Ideally, this histogram should resemble a delta-function, with a peak
at very close to the time interval specified for the "compute" function.  The code
prints the ratio of the measured elapsed time to the expected time for an ideal system;
and that ratio provides a measure of the effective parallel efficiency.  The key
output items are the overall parallel efficiency, and the histogram of step times.

A summary is printed at the end with the host and cpu assignments for each MPI rank,
along with some cumulative data including the average compute time in msec, the
percent relative deviation, the accumulated time in MPI, and the number of involuntary
context switches.  This information has often been useful for identifying problems
with a specific host or cpu.  For example, if a certain cpu is not performing at the
typical level, all of the other MPI ranks will wait on the slow one; thus the slow
rank will have the minimum cumulative time in MPI.  If the computation rate is truly
uniform over all MPI ranks, there should be a small spread in cumulative MPI times.

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

As mentioned above, some compute kernels, including the lookup table kernel, are
sensitive to caches, and may show variable performance when Linux uses address space
layout randomization (ASLR).  One can launch the job with a helper process that uses
the Linux personality() routine to disable ASLR and then exec the real program, which
will inherit the personality setting.  Note that it is not possible to directly set
the ALSR behavior in the parent process, because that has already been set by Linux.
An example of a launcher that disables ASLR is shown below:

#include <stdio.h>

#include <stdlib.h>

#include <unistd.h>

#include <sys/personality.h>

int main(int argc, char * argv[])

{

  int rc;

  rc = personality(ADDR_NO_RANDOMIZE);

  if (rc == -1) fprintf(stderr, "personality failed ... not disabling ASLR\n");

  rc = execvp(argv[1], &argv[1]);

  if (rc == -1) {

    fprintf(stderr, "execvp failed for %s ... exiting\n", argv[1]);

    exit(0);

  }

  return 0;

}

Use of such a launcher would be : mpirun ... -np 128 ./launcher  ./your.exe  [program args] .

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

It can be useful to look at the sequence of measured compute intervals for a given rank
or a set of ranks.  You can dump a binary file that contains all samples (-d option),
and then extract the compute times or the step times for a specified MPI rank.  The time
spent in communication routines = (step - compute).  Utilities to extract data from the
binary file are provided : extract_comp.c  and  extract_step.c .

repository: ghas-results/osnoise

repo_name: ghas-results/osnoise


text:
---
title: Using Jitsi Meet to reach out to others
date: 2020-04-15T21:37:50.843Z
featuredBlog: false
priority: null
author: Frederic Passeron
authorimage: https://gravatar.com/avatar/3397fa0097e38d890aac69b996879a4e?s=96
thumbnailimage: null
tags:
  - opensource
---
![picture11](https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture11-1586988196990.png)

Along with half of the world’s population, I find myself working from home due to sheltering requirements imposed on us by the COVID-19 pandemic. In these troubled times, communication with my friends and family have become critical. I found that I needed ways to stay in touch with them easily. Of course, mobile apps offer a great diversity of options to stay in touch with one another. But conducting a multi-user video conference on a mobile phone is not very easy nor is it convenient.

There are many people today who are looking for ways to virtually connect through video conferencing. From a more professional standpoint, teachers need to be able to instruct and stay in touch with their students. Small companies could use it for meetings. Doctor/patient interactions could also benefit from such a tool. But it would need to be very inexpensive (i.e. free) and easy to use.

As a Hewlett Packard Enterprise (HPE) employee, the company offers us many ways to video conference one another, i.e. through Teams, Skype, and Zoom. Even though these technologies try to make it easy for everyone to use and affordable, I thought I might check on the open source side of the world to see what was available there as well.

Zoom is free for 40 minutes. That’s nice, but is it enough? Looking a little deeper, I found [Jitsi Meet.](https://jitsi.org/jitsi-meet/) Jitsi is free and ready to go whenever you want. Just open the [link](https://meet.jit.si/) and start your meeting. That’s all it takes.

For those who are a bit more technical, you may want consider setting up your own Jitsi Meet server at home. This could be a service you could deliver to your family and friends that would offer you a bit more control and privacy, as you wouldn’t have to rely on an external provider. To help those of you who may be interested in setting up your own Jitsi Meet instance, I thought I’d share my experience. In this post, I’ll run you through the installation process and show you how to use it.

## What is Jitsi?

Jitsi is a set of open-source projects that allows you to easily build and deploy secure videoconferencing solutions. At the heart of Jitsi are two projects, Jitsi Videobridge and Jitsi Meet. These projects allow you to have conferences on the internet, while other projects in the community enable features such as audio, dial-in, recording, and simulcasting.

## Why Jitsi?

Why was I interested in finding an open source-based solution? I wanted a living project with a community behind it in case I ran into issues or required support. I also needed a solution that I could set up myself and would not require too many resources.

Official minimum hardware requirements for the Jitsi Meet server are very low. One gigabyte of memory and one CPU core plus 25 gigabytes of storage should be enough when used on a Linux platform. It can be set up directly on your own hardware or on a virtual machine hosted by your favorite Cloud provider.

I own a ProLiant Microserver Gen8 at home. It runs a hypervisor and hosts several virtual machines (OPNsense Firewall and Nextcloud server). I use this server to perform tests on new operating systems and software stacks. Since I had some room left on the system, I decided to create a new VM for this new project.

My home network looks like this:

![picture2](https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture2-1586988219313.png)

I followed several guides on how to perform the installation:

Jitsi-Meet official install guide:
https://github.com/jitsi/jitsi-meet/blob/master/doc/quick-install.md

https://vitux.com/how-to-install-jitsi-meet-video-conference-platform-on-ubuntu/

HA Proxy – Let’s Encrypt:
https://forum.opnsense.org/index.php?topic=12126.0

Advanced Configuration: Behind NAT configuration
https://github.com/jitsi/jitsi-meet/blob/master/doc/quick-install.md

Secure Domain: First four steps only.
https://github.com/jitsi/jicofo#secure-domain

## Installing Jitsi Meet

My VM requirements:
CPU 1
RAM 2 GB
HDD 25 GB
Nic 1

## Initial Setup

Jitsi Meet requirements:
A server running Ubuntu 18.04 LTS
A non-root user with sudo privileges
Before starting, update your operating system with the latest version with the following command:

```bash

sudo apt-get update -y
sudo apt-get upgrade -y

```

Once your system is up-to-date, restart your system to apply the changes.
Next, you will need to set up a relevant hostname (node1 in this example) and FQDN to your system. You can do this by running the following commands:

`sudo hostnamectl set-hostname node1`

Next, open /etc/hosts file and add FQDN.

`sudo nano /etc/hosts`

Add the following line and adapt to your host and domain names:

`<IP_ADDRESS> node1.example.com node1`

Add the public IP and the relevant domain name, too.

`X.X.X.X node1.example.com node1`

Save and close the file. Then, verify the hostname with the following command:

`hostname -f`

## Swap Adjustment

With a limited set of resources, I needed to adapt and adjust the swap file.

```bash

sudo dd if=/dev/zero of=/swapfile count=2048 bs=1M
sudo chmod600 /swapfile
sudo makeswap /swapfile
sudo swapon /swapfile
echo ‘/swapfile none swap sw 0 0’ | sudo tee –a /etc/fstab

```

## Install Java

Next, you will need to install Java to your system. You can install OpenJDK JRE 8 by running the following command:

`sudo apt-get install -y openjdk-8-jre-headless -y`

Once Java is installed, verify the Java version with the following command:

`java -version`

Output:

```markdown

openjdk version "10.0.2" 2018-07-17
OpenJDK Runtime Environment (build 10.0.2+13-Ubuntu-1ubuntu<IP_ADDRESS>)
OpenJDK 64-Bit Server VM (build 10.0.2+13-Ubuntu-1ubuntu<IP_ADDRESS>, mixed mode)

```

## Install Nginx

Jitsi Meet uses Nginx as a reverse proxy. So you will need to install it to your system. You can install it with the following command:

`sudo apt-get install nginx -y`

Once Nginx is installed, you can check the Nginx service with the following command:

`sudo systemctl status nginx`

Output:

```markdown

? nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2018-11-14 10:06:01 UTC; 1h 4min ago
     Docs: man:nginx(8)
 Main PID: 16734 (nginx)
    Tasks: 2 (limit: 1114)
   CGroup: /system.slice/nginx.service
           ??16734 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
           ??18709 nginx: worker process

Nov 14 10:06:01 node1 systemd[1]: Starting A high performance web server and a reverse proxy server...

```

## Install Jitsi Meet

By default, Jitsi Meet is not available in the Ubuntu 18.04 default program repository. So, you will need to add the Jitsi Meet download server to the Ubuntu program repository.

You can do this by running the following command:

```bash
wget -qO - https://download.jitsi.org/jitsi-key.gpg.key | sudo apt-key add -
sudo sh -c "echo 'deb https://download.jitsi.org stable/' > /etc/apt/sources.list.d/jitsi.list"

```

Next, update the repository and install Jitsi Meet with the following command:

```bash

sudo apt-get update -y
sudo apt-get install jitsi-meet -y

```

During the installation process, you will need to provide your hostname as shown below:

![picture4](https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture4-1586988248790.png)

Provide your hostname and click on the __OK__ button. You will be asked to select the SSL certificate as shown below:

![picture6](https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture6-1586988261662.png)

## Local Firewall Update
Jitsi Meet requires ports 80 and 433 (TCP) and a range from 10000 to 20000 (UDP) to be opened locally and on your network to work properly. Consider also opening SSH for remote access You update the firewall rules by running the following commands:

Ubuntu Firewall rules:

Allow SSH for remote access:

`sudo ufw allow ssh`

Allow Ports 80, 443, and range from 10000 to 20000 for Jitsi Meet.

```bash

sudo ufw allow http
sudo ufw allow https
sudo ufw allow in 10000:20000/udp

```

Enable firewall rules.

`sudo ufw enable`

## Local Area Network Firewall Rules (OPNsense)
As I had already set up several servers behind my firewall, I had to use HA Proxy to distribute the different HTTP / HTTPS requests to the different servers (Nextcloud, Jitsi Meet).

I followed the thread below to set everything up. https://forum.opnsense.org/index.php?topic=12126.0

The thread actually covered the HA Proxy and the Let’s Encrypt topics. This proved to be very helpful in the end. Certificates are now handled by the HA Proxy layer in front of the different servers.

## Secure Meeting Room Access
By default, anyone who has access to your Jitsi instance will be able to start a conference. If your server is open to the world, anyone can have a chat with anyone else. (Unfortunately, I don’t have unlimited resources. Therefore, I needed to set some limits.)

If you want to limit the ability to start a conference to registered users only, set up a "secure domain". Follow the instructions at https://github.com/jitsi/jicofo#secure-domain.

Follow the first 4 steps and you’ll be ready to go! This will allow you to set up a user with credentials with the necessary rights to create a room.

## Access Jitsi Meet
If you have followed my instructions, Jitsi Meet is now up and listening on port 443. Open your web browser and type the URL `https://node1.example.com` or `https://your-server-ip.` You will be redirected to the following page:

![picture7](https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture7-1586988283022.png)

Here, provide the room name you desire and click on the __GO__ button. You should see the following on your screen:

![picture8](https://hpe-developer-portal.s3.amazonaws.com/uploads/media/2020/3/picture8-1586988303937.png)

Click on the __Allow__ button to start the live video conference.

## Some last comments

After a few hours of work, I had the server ready to go and I could open my first room. If you’ve followed my instructions, you should now be all set to easily meet with your friends, family, and colleagues!

Keeping a social link is very important during these times. That’s why simple solutions like Jitsi Meet can really be helpful. Jitsi Meet offers an easy way for people to connect.

There are so many possible applications for this technology. Educators can use this solution with the relevant hardware configuration so that a complete class could easily attend a course. Small companies meetings could be hosted via a dedicated server, too.

For those who might find technology a little more daunting or less accessible, Jitsi Meet still can be easily used with a simple browser. Think how this could help the elderly keep in touch and interact with their families.

Stay tuned to the [HPE DEV blog site](/blog) for more helpful tutorials.

repository: hpe-dev-incubator/hpe-dev-portal

repo_name: hpe-dev-incubator/hpe-dev-portal


text:
# 9 NPA rebs, 46 others arrested

Published at: **2019-11-02T10:50:46+00:00**

Author: **Zaida Delos Reyes**

Original: [Journal Online](https://journal.com.ph/news/metro/9-npa-rebs-46-others-arrested)

This was confirmed yesterday by  Captain Cenon C. Pancito, spokesman of Philippine Army’s 3rd Infantry Division ( ID).
Pancito identified those considered  high value individuals of the NPA as  John Milton Lozande, Noli Lazeran Rosales, Proceso Quiatchon, Albert Delacerna, Mary Ann Krueger alias Mary Anne Dela Concepcion,  Romulo Buto-on, Amatlin Chin-Chin  Bito-on, Danilo Tabura and Roberto Lachica.
Pancito said the troops conducted simultaneous implementation of search warrants for violation of  RA 10591 or  Comprehensive Forearms and Ammunition  Regulation Law issued by Quezon City Regional Trial Court (RTC)  in Barangays  33, Bata and Taculing all in Bacolod City  at around 5:45 p.m. on October 31,2019.
The operation resulted in the recovery of 32 assorted firearms, 3 fragmentation grenade, 2 ammunition of 40MM Grenade Launcher and assorted live ammunition.
Pancito said the operation also resulted in the accidental arrest of 40 identified members of Komiteng Rehiyon-Negros, Cebu, Bohol and Siquijor (KR-NCBS) and 15 newly recruited individuals, 14 of them minors coming from different areas of Negros Occidental who are undergoing combat-related training and indoctrination .
The minors were turned over to the custody of the City Social Welfare Development Office (CSDWO).

license: Unlicense

repository: dsadlollioygjmzvcxv/tdc-exported

repo_name: dsadlollioygjmzvcxv/tdc-exported


text:
# Alternative Classes with Different Interfaces

### Signs and Symptoms

Two classes perform identical functions but have different method names.

![img](https://sourcemaking.com/images/refactoring-illustrations/alternative-classes-with-different-interfaces-1.png)

### Reasons for the Problem

The programmer who created one of the classes probably didn't know that a functionally equivalent class already existed.

### Treatment

Try to put the interface of classes in terms of a common denominator:

- [Rename Method](https://sourcemaking.com/refactoring/rename-method)s to make them identical in all alternative classes.
-  [Move Method](https://sourcemaking.com/refactoring/move-method), [Add Parameter](https://sourcemaking.com/refactoring/add-parameter) and [Parameterize Method](https://sourcemaking.com/refactoring/parameterize-method) to make the signature and implementation of methods the same.
-  If only part of the functionality of the classes is duplicated, try using [Extract Superclass](https://sourcemaking.com/refactoring/extract-superclass). In this case, the existing classes will become subclasses.
-  After you have determined which treatment method to use and implemented it, you may be able to delete one of the classes.

### Payoff

- You get rid of unnecessary duplicated code, making the resulting code less bulky.
- Code becomes more readable and understandable (you no longer have to guess the reason for creation of a second class performing the exact same functions as the first one).

![img](https://sourcemaking.com/images/refactoring-illustrations/alternative-classes-with-different-interfaces-2.png)

### When to Ignore

- Sometimes merging classes is impossible or so difficult as to be pointless. One example is when the alternative classes are in different libraries that each have their own version of the class.

repository: liupd/JavaParty

path: /docs/编程/高效编程/重构/异曲同工的类.md

path: /docs/编程/高效编程/重构/异曲同工的类.md

repo_name: liupd/JavaParty


text:
# FEASTsolver

A Julia implementation of the FEAST eigensolver algorithm [1]. Being develloped for learning and testing purposes.

Current functionality:
* Cauchy contour integral implemented using Gauss-Legendre quadrature along a circle.
* Solution of linear system using direct solver based on LU decomposition.

Limitation:
* The subspace dimension must estimated.

## References
[1] "A Density Matrix-based Algorithm for Solving Eigenvalue Problems", Eric Polizzi, Phys. Rev. B 79, 115112 (2009), e-print: arXiv:0901.2665[](https://arxiv.org/abs/0901.2665)


text:
Checks that arrays are sliced with endless ranges instead of
`ary[start..-1]` on Ruby 2.6+.

# Examples

```ruby
# bad
items[1..-1]

# good
items[1..]
```

[Source](http://www.rubydoc.info/gems/rubocop/RuboCop/Cop/Style/SlicingWithRange)

repository: codacy/codacy-rubocop

gha_license_id: NOASSERTION

repo_name: codacy/codacy-rubocop


text:
Dashbord Web
============

> now the main develope branch is on feature/flask

## What is Dashbord ?

Dashbord is a web front base to build a dashbord application

## Design

### Web

* [Flask](http://flask.pocoo.org/) : Flask is a microframework for Python based on Werkzeug, Jinja 2 and good intentions.

* [Angular.js](http://angularjs.org/) : HTML enhanced for web apps!.

* [Sass](http://sass-lang.com/): the most mature, stable, and powerful professional grade CSS extension language in the world.

* [Jade](http://jade-lang.com/): A Simple Node Template Engine.

* [Bootstrap](http://getbootstrap.com/):  Sleek, intuitive, and powerful mobile first front-end framework for faster and easier web development.

* [angular-ui/bootstrap](http://angular-ui.github.io/bootstrap//): Native AngularJS (Angular) directives for Twitter's Bootstrap

### Database

* [Redis](http://redis.io/): an open source, BSD licensed, advanced key-value store.

### Build Tools

* [Yeoman](http://yeoman.io/): MODERN WORKFLOWS FOR MODERN WEBAPPS

* [Grunt](http://gruntjs.com/): The JavaScript Task Runner

* [Bower](http://bower.io/): A package manager for the web

### Code Quality Tools

* [Jshint](http://www.jshint.com/): a JavaScript Code Quality Tool

* [Pylint](http://www.pylint.org/): start your Python code

## How To Develop ?

### Web Front-end Development?

Dashbord is build by [yeoman](http://yeoman.io/index.html) a **MODERN WORKFLOWS FOR MODERN WEBAPPS** And develop by [Angular.js](http://angularjs.org/).

* install the npm requirements.

```
npm install
```

* install the bower requirements

```
bower install
```

* the base operation of yeoman

```
yo angular:controller myController
yo angular:directive myDirective
yo angular:filter myFilter
yo angular:service myService
```

* run test

```
grunt test
```

* run application

```
grunt server
```

this operation will start a simple http server on port 9000, [http://<IP_ADDRESS>:9000/](http://<IP_ADDRESS>:9000/), and this server will auto reload the bowers if the any file has change.

* build

```
grunt build
```

this operation will build the application to the **dict** floder.

### Server Development ?

> This project now is base on Python Flask Web Framework.

The Global target of server is proxy the http request to vms serer. on same time server will cahce the query data from vms.

The Cache update will base on two way. First, when user send a http request by 'POST', 'PUT', 'DELETE' server will send a signal to update the cache.
The sencod way is subscription the rabbitmq message and update the cache.

## ScreenSnapshot

![home](app/images/2014-01-12-1.41.09.png)

![create_vm](app/images/2014-01-12-1.41.41.png)

![](app/images/2014-01-12-1.41.55.png)

![](app/images/2014-01-12-1.42.12.png)

![](app/images/2014-01-12-1.42.33.png)

repository: yunlzheng/dashbord

repo_name: yunlzheng/dashbord


text:
# DashR NCBI Explorer App

## About this App:

## How to Run the App

Clone the repository:

```
git clone https://github.com/plotly/dash-sample-apps
cd dash-sample-apps/apps/dashr-ncbi-explorer
```

Run `app.R`

The app will load into your default browser window. If it does not, navigate to <IP_ADDRESS>:8050.

## Screenshots

![ncbiexplorer.png](assets/ncbiexplorer.png)

## More

Learn more about Plotly and Dash [here](https://plot.ly/dash).


text:
# Salix columbiana (Dorn) Argus SPECIES

#### Status
ACCEPTED

#### According to
The Catalogue of Life, 3rd January 2011

#### Published in
null

#### Original name
Salix exigua var. columbiana Dorn

### Remarks
null

repository: mdoering/backbone

repo_name: mdoering/backbone


text:
[![Django CSV generator on pypi](https://img.shields.io/badge/pypi-1.0.2-green.svg)](https://pypi.python.org/pypi/csv_generator)
![MIT license](https://img.shields.io/badge/licence-MIT-blue.svg)
![Build status](https://travis-ci.org/fatboystring/csv_generator.svg?branch=master)

# CSV Generator

A configurable CSV Generator for Django

## Support

Supports the following versions of Django:

 - Django 2.2
 - Django 3.0

## Installation

`pip install csv_generator`

Add 'csv_generator' to the `INSTALLED_APPS` setting in your django project

`INSTALLED_APPS += ('csv_generator',)`

`python manage.py migrate`

## Configuration

### CSV Writer

By default csv_generator uses the [unicode CSV writer class](https://github.com/fatboystring/csv_generator/blob/master/csv_generator/utils.py) bundled with the app.
However, you can use your own CSV Writer class by specifying the import path for your custom writer class (as a string) using the `CSV_GENERATOR_WRITER_CLASS` setting:

```
CSV_GENERATOR_WRITER_CLASS = 'my_app.utils.MyCustomCsvGeneratorClass'
```

Any custom CSV writer class must implement the same methods (with the same signatures) as the UnicodeWriter class bundled with this app.

### Non model field attributes

It is also possible to process extra model attributes (non-field attributes) such as methods and properties defined on the model.
Extra attributes which can be processed by the csv_generator are defined in the settings file using the following setting:

```
CSV_GENERATOR_AVAILABLE_ATTRIBUTES = {
    'all': {
        'commonattribute': 'An attribute that is common to all models in the system'
    },
    'myapp.mymodel': {
        'somecustomattribute': 'My custom model attribute'
    }
}
```

Each key in the `CSV_GENERATOR_AVAILABLE_ATTRIBUTES` dict should be in the format `<app_label>.<model_name>` (lower case).
The value for each of these keys should be another dict where the key is the attribute name on the model and the value is the human readable description of the attribute used in the admin forms.
The special case here is the `all` key in the `CSV_GENERATOR_AVAILABLE_ATTRIBUTES`.  Values defined under the `all` key will be made available to all models in the system.

Each attribute defined can be a property or a method.  Methods will be called when a csv file is generated.  It is important to note that any method made available to the csv generator will be called with no arguments. As such the signature of the methods exposed to the csv generator should be callable without any arguments.

### CSV Value Overrides

In some situations it may be desirable to override certain csv column values before they are written to the csv file.
For example, by default, the CSV generator will render `None` values as `'None'` in the CSV file. It may be preferable to render these values as empty strings.
To achieve this, you can specify a `CSV_GENERATOR_VALUE_OVERRIDES` setting, which is a dictionary where the key is the value to be mapped, and the corresponding value is the overridden value to be rendered into the CSV column.

#### Example

```
CSV_GENERATOR_VALUE_OVERRIDES = {
    None: ''
}
```

In the example above, all `None` values will be rendered as empty strings in the generated CSV file.  Note that the default behaviour is to render the raw values string represention by passing it through pythons `str()` function (e.g. `str(None)`).

## Usage

### Django Admin integration

To integrate CSV generator admin functionality for your models just call `admin.register` passing `csv_generator.admin.CsvExportAdmin` as the ModelAdmin class
This will add an extra action to the listing view allowing you to export selected records as a csv file using one of your configured CSV generators.

#### Example

```
from csv_generator.admin import CsvExportAdmin
from django.contrib import admin
from my_app.models import MyModel

admin.site.register(MyModel, CsvExportAdmin)
```

### Setting up a csv generator

Once logged into the Django admin you can create a csv generator instance.  Each CSV generator has the following fields:

 - title: Human readable name used to identify the CSV writer in form fields etc
 - include_headings: Whether or not to write column headings to the generated csv file
 - content_type: The content type this csv_generator can process

Once the CSV generator has been created you will be able to specify the columns on the specified content_type model to write out to generated CSV files.
Each column has the following fields:

 - column_heading: Will be used instead of the model fields verbose_name if it has been specified (verbose name is the default)
 - model_field: The name of the field from the specified content type to write into the column
 - order: The order of the csv column relative to other columns belonging to the specified csv generator instance
 - generator: A foreign key to the generator instance that uses this column (not visible in the django admin)

### Using the generators in your app

You can use the csv_generator in your own app.

#### Retrieving CsvGenerator models

##### For a particular model class or instance
```
from csv_generator.models import CsvGenerator
from my_app.models import MyModel

generators = CsvGenerator.objects.for_model(MyModel)
generators = CsvGenerator.objects.for_model(MyModel())
```

##### For a particular content_type
```
from csv_generator.models import CsvGenerator
from django.contrib.contenttypes.models import ContentType
from my_app.models import MyModel

ctype = ContentType.objects.get_for_model(MyModel)
generators = CsvGenerator.objects.for_content_type(ctype)
```

##### For a particular content_type id
```
from csv_generator.models import CsvGenerator
from django.contrib.contenttypes.models import ContentType
from my_app.models import MyModel

ctype = ContentType.objects.get_for_model(MyModel)
generators = CsvGenerator.objects.for_content_type_id(ctype.pk)
```

#### Generating CSV files

##### Write to file

```
from csv_generator.models import CsvGenerator
from my_app.models import MyModel

generator = CsvGenerator.objects.get(pk=1)  # Assume this is a generator instance for MyModel content type
file_handle = open('my_csv.csv', 'wB')
queryset = MyModel.objects.all()
csv_file = generator.generate(file_handle, queryset)
```

##### Write to Http Response

```
from csv_generator.models import CsvGenerator
from django.http import HttpResponse
from my_app.models import MyModel

generator = CsvGenerator.objects.get(pk=1)  # Assume this is a generator instance for MyModel content type
response = HttpResponse(content_type='text/csv')
response['Content-Disposition'] = 'attachment; filename="my_csv.csv"'
queryset = MyModel.objects.all()
response = generator.generate(response, queryset)
```

## Release History

- 0.1: Initial Release
- 0.2: Bug fixes relating to missing templates in pypi package
- 0.3: Django 1.7 compatibility
- 0.4: Attribute descriptor/Empty cell implementation
- 0.5: ForeignKey descriptor implementation allowing foreign key and OneToOne key relationships to be traversed
- 0.6: Support for django 1.10, python 3.4 and python 3.5
- 0.7: Support for django 1.11
- 0.8: Support for django 2.0 and python 3.6
- 0.8.1: Support for django 2.1, 2.2 and python 3.7, 3.8
- 0.8.2: Fix regression where form would not instantiate correctly in django admin
- 0.9.0: Ensure content type options in admin forms are ordered by model name.  Allow attribute values to be overridden at a global level.
- 0.9.1: Fix bug where an extra `CsvGeneratorColumn` record was automatically added when editing an existing `CsvGenerator` model instance.
- 1.0.0: Removed support for older, unsupported versions of django.  Removed compatibility fixes and add support for django 3.0
- 1.0.1: Fixed bug with 'Take me back' button in django admin export form that caused csv file to be downloaded again.
- 1.0.2: Ensure changelist filters are persisted when the 'Take me back' button on the csv export view is clicked

gha_language: Python


text:
# K8s 1.25.x with SR-IOV in a K3d cluster

Provides a pre-deployed containerized k8s cluster with version 1.25.x that runs
using [K3d](https://github.com/k3d-io/k3d)
The cluster is completely ephemeral and is recreated on every cluster restart. The KubeVirt containers are built on the
local machine and are then pushed to a registry which is exposed at
`<IP_ADDRESS>:5000`.

This version requires to have SR-IOV enabled nics (SR-IOV Physical Function) on the current host, and will move
physical interfaces into the `K3d`'s cluster agent node(s) (agent node is a worker node on k3d terminology)
so that they can be used through multus and SR-IOV
components.

This provider also deploys [multus](https://github.com/k8snetworkplumbingwg/multus-cni)
, [sriov-cni](https://github.com/k8snetworkplumbingwg/sriov-cni)
and [sriov-device-plugin](https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin).

## Bringing the cluster up

```bash
export KUBEVIRT_PROVIDER=k3d-1.25-sriov
export KUBECONFIG=$(realpath _ci-configs/k3d-1.25-sriov/.kubeconfig)
make cluster-up
```
```
$ kubectl get nodes
NAME                 STATUS   ROLES                  AGE   VERSION
k3d-sriov-server-0   Ready    control-plane,master   67m   v1.25.6+k3s1
k3d-sriov-agent-0    Ready    worker                 67m   v1.25.6+k3s1
k3d-sriov-agent-1    Ready    worker                 67m   v1.25.6+k3s1

$ kubectl get pods -n kube-system -l app=multus
NAME                   READY   STATUS    RESTARTS   AGE
kube-multus-ds-z9hvs   1/1     Running   0          66m
kube-multus-ds-7shgv   1/1     Running   0          66m
kube-multus-ds-l49xj   1/1     Running   0          66m

$ kubectl get pods -n sriov -l app=sriov-cni
NAME                            READY   STATUS    RESTARTS   AGE
kube-sriov-cni-ds-amd64-4pndd   1/1     Running   0          66m
kube-sriov-cni-ds-amd64-68nhh   1/1     Running   0          65m

$ kubectl get pods -n sriov -l app=sriovdp
NAME                                   READY   STATUS    RESTARTS   AGE
kube-sriov-device-plugin-amd64-qk66v   1/1     Running   0          66m
kube-sriov-device-plugin-amd64-d5r5b   1/1     Running   0          65m
```

### Conneting to a node
```bash
export KUBEVIRT_PROVIDER=k3d-1.25-sriov
./cluster-up/ssh.sh <node_name> /bin/sh
```

## Bringing the cluster down

```bash
export KUBEVIRT_PROVIDER=k3d-1.25-sriov
make cluster-down
```

This destroys the whole cluster, and gracefully moves the SR-IOV nics to the root network namespace.

Note: killing the containers / cluster without gracefully moving the nics to the root ns before it,
might result in unreachable nics for few minutes.
`find /sys/class/net/*/device/sriov_numvfs` can be used to see when the nics are reachable again.

## Using podman
Podman v4 is required.

Run:
```bash
systemctl enable --now podman.socket
ln -s /run/podman/podman.sock /var/run/docker.sock
```
The rest is as usual.
For more info see https://k3d.io/v5.4.1/usage/advanced/podman.

## Updating the provider

### Bumping K3D
Update `K3D_TAG` (see `cluster-up/cluster/k3d/common.sh` for more info)

### Bumping CNI
Update `CNI_VERSION` (see `cluster-up/cluster/k3d/common.sh` for more info)

### Bumping Multus
Download the newer manifest `https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/deployments/multus-daemonset-crio.yml`
replace this file `cluster-up/cluster/$KUBEVIRT_PROVIDER/sriov-components/manifests/multus/multus.yaml`
and update the kustomization file `cluster-up/cluster/$KUBEVIRT_PROVIDER/sriov-components/manifests/multus/kustomization.yaml`
according needs.

### Bumping calico
1. Fetch new calico yaml (https://docs.tigera.io/calico/3.25/getting-started/kubernetes/k3s/quickstart)
   Enable `allow_ip_forwarding` (See https://k3d.io/v5.4.7/usage/advanced/calico)
   Or use the one that is suggested here https://k3d.io/v5.4.7/usage/advanced/calico whenever it is updated.
2. Prefix the images in the yaml with `quay.io/` unless they have it already.
3. Update `cluster-up/cluster/k3d/manifests/calico.yaml` (see `CALICO` at `cluster-up/cluster/k3d/common.sh` for more info)

Note: Make sure to follow the latest verions on the links above.

repository: kubevirt/containerized-data-importer

repo_name: kubevirt/containerized-data-importer


text:
---
layout: post
title:  "Migrating from Mailgun to SES"
date:   "2020-04-17"
excerpt_separator: <!--more-->
---

While sheltered in place, I finally had the time to do something I've been meaning to do for years: migrate Instapaper's email service from Mailgun to Amazon's Simple Email Service (SES).

In 2018, Mailgun changed their email pricing from volume-based to tier-based and, as a result, the cost of sending the [Instapaper Weekly](https://blog.instapaper.com/post/74088929566) nearly doubled. Shortly after the price change, we disabled the Weekly email while taking the steps toward making Instapaper GDPR compliant, but it's remained off because it was prohibitively expensive to continue operating the email.

After a few weeks of work, we are almost entirely migrated onto SES. As a result, we finally re-enabled the Weekly email after an almost 2 year hiatus! I [open sourced several tools](https://github.com/Instapaper/ses-tools) we used for the integration, and I'll share the steps we took for the migration below...

<!--more-->

## Migrating an Internal Use Case

All errors from the Instapaper servers get emailed directly to me. In the past, we've had fancier ways of handling errors, but I've found that getting them emailed directly to me is a cheap, easy, and efficient way to get visibility.

After verifying our internal email address for errors in Simple Email Service, I applied for a limit increase to allow me to send 20,000 emails per day which more than covers the volume. I then modified some functions as follows:

```
def deliver_exception(traceback, extra='', server=''):
    subject = "[Instapaper %s] Exception: %s" % (server, traceback[-1])
    deliver(sender=config.INTERNAL_EMAIL_ADDRESS,
            recipient=config.EXCEPTION_EMAIL_ADDRESS,
            subject=subject,
            text="\n".join([extra] + traceback),
            use_mailgun=False)

def _deliver_email(sender, recipient, subject, text=None,
                   html=None, attachments=None, tag=None,
                   campaign=None, use_mailgun=True):
    if use_mailgun:
        _deliver_mailgun_email(
            sender, recipient, subject, text=text,
            html=html, attachments=attachments, tag=tag,
            campaign=campaign)
    else:
        ses.send_email(sender, recipient, subject,
                       text=text, html=html)
```

In the above, we added a `use_mailgun` keyword argument to determine whether to send to Mailgun or SES, then set the argument to `False` for exception emails.

## Handling Bounces and Complaints

In order to send to any email address, not just a verified email address, you need to both verify a domain and handle bounce events. SES allows you to send email events (e.g. Bounce, Complaint, Delivery, etc.) to a Simple Notification Service topic, which can then send messages to Simple Queue Service to be consumed by your application:

![](/img/posts/ses-to-instapaper.png)

We never previously tracked bounced emails directly in Instapaper since Mailgun stores bounced emails and prevents duplicate sends to permanently bounced addresses. Storing the bounce information is a bit of extra overhead for Instapaper to track, but ultimately more efficient and the bounce information can be used elsewhere. First step is to store the bounced information in a user's email settings:

```
ALTER TABLE email_settings ADD COLUMN bounced TINYINT(1) DEFAULT 0, ALGORITHM=INPLACE, LOCK=NONE;
```

Next we add the `bounced` column to our object model. Lastly, I created the SQS queue workers to read the SES events, and store the bounced information with a user's email settings. We open sourced the [SQS worker](https://github.com/Instapaper/ses-tools/blob/master/workers/sqs_worker.py) and [task](https://github.com/Instapaper/ses-tools/blob/master/workers/sqs_tasks.py) we implemented to process bounce and complaints events on GitHub if you're interested in the full code.

## Reducing Bounce Rate

In the reputation dashboard, SES tracks your bounce rate and complaint rate, and has the following notice about bounce rates:

> We expect our senders' bounce rates to remain below 5%. Senders with a bounce rate exceeding 10% risk a sending Pause.

After reviewing our sending metrics in Mailgun, I determined our bounce rate was roughly 15%, and is largely driven by our welcome email. Investigating further, I determined that 15% of our new user signups are SEO spammers that are trying to use Instapaper to game Google for traffic.

We did at one point allow Instapaper public profiles to be indexed by Google, however, after determining that spammers were abusing it we disabled all public profiles for indexing. Nevertheless, spammers seem to think that its still a viable spam vector, and our bounce rate is pretty high as a result.

The strategy for reducing bounce rate was to detect the spam signups and automatically mark those accounts as bounced. We used a number of strategies in order to do this with high confidence. Having said that, fighting spammers is funamentally an adversarial game, so I'm hesitant to outline all of the approaches here, but I will share one.

For high value email domains (e.g. gmail.com), we integrated Mailgun's email validation service to help us determine if those accounts were legitimate. While this service is also expensive ($1.20 for 100 validations) and our long-term goal is to migrate off of Mailgun completely, this seemed to be a worthwhile effort in order to ensure a low bounce rate.

After testing a variety of approaches, we were able to get the bounce rate down to 2% within Mailgun, and can now migrate our first external email.

## Migrating the Welcome Email

I chose the welcome email for the first public use case because it would allow us to process bounces immediately on sign up via our integration from SES=>SNS=>SQS=>Instapaper. The migration itself was pretty straightforward. First, I added a `use_mailgun` flag to our `Message` object, defaulted to True:

```
class Message(object):
    def __init__(self, subject):
        self.subject = subject
        self.sender = None
        self.recipient = None
        self.html = None
        self.text = None
        self.attachments = None
        self.tag = None
        self.campaign = None
        self.use_mailgun = True
```

Then when creating the welcome email, we just set the `use_mailgun` flag to `False`:

```
@email
def welcome_email(user):
    message = Message("Welcome to Instapaper!")
    message.sender = config.SUPPORT_EMAIL_SENDER
    message.recipient = user.email
    message.text = tornado_mailer.render('welcome.txt')
    message.html = tornado_mailer.render('welcome.html')
    message.use_mailgun = False
    return message
```

Ultimately, these fields get passed to the `_deliver_email` function defined above, and sends via Simple Email Service.

## Backfilling Bounces

With the welcome email migrated to SES, we are receiving bounce events and storing the bounce flag for all new sign ups. However, all email bounces from the past 5 years were still in Mailgun. In order to ensure that we do not resend to those bounced emails on SES, we needed to export that historical data from Mailgun and run a backfill to mark those emails as bounced. Here’s the simple backfill script we used:

```
with open(sys.argv[1]) as f:
    reader = csv.reader(f)
    for row in reader:
        email = row[4].strip()
        user = User.query.filter_by(email=email).first()
        if not user:
            continue
        mail_settings = MailSettings.get_or_create(user)
        mail_settings.set_bounced()
```

Mailgun stored 1.3M bounced email addresses from the prior 5 years, and the above script completed in just under 24 hours.

## Return of the Weekly

To ensure that there weren't additional bounced addresses in the Weekly, I decided to re-enable it to run on Mailgun in order to collect the new bounces without affecting our SES bounce rate. After re-enabling the Weekly on April 5th, I was able to pull the latest bounce export, and mark another 250K bounced emails using the backfill script above.

I also learned that the Weekly email list had grown to over 3 million addresses, which would have made it even more cost prohibitive to operate over the past few years without this migration. I estimate the cost of sending this test run to be $2,500 based on our current Mailgun plan.

Of the 3M emails we sent out, about 1.2M were ratelimited by a handful of domains like outlook.com, live.com, and hotmail.com due to unknown IP reputation issues. In order to better understand how the lack of IP reputation affected our delivery rate, I decided to test run the Weekly again on Mailgun for April 12th (despite being pained by the cost). The rate limiting dropped by 40% to about 700K of these soft bounces.

While we do have dedicated IP addresses in SES and there is an "IP warm-up" feature, I expect these soft bounces to spike again as we migrate the Weekly to SES, but it's good to know that they should decrease over time as we build up reputation on the new IP addresses.

## Migrating Kindle Sends

Instapaper has long offered the ability to package up your most recent articles as an ebook and have them delivered to your Kindle. With a Kindle account, you receive an email address where you can send attachments to be added to your Kindle. Instapaper runs scheduled jobs to compile parsed articles and images into ebooks, then email them as attachments to a user's Kindle email address.

In order to migrate this functionality, I needed to build in the ability to pass attachments to SES, and deliver the email attachments using their `SendRawEmail` API:

```
def format_email(sender, recipient, subject,
                 text=None, html=None, attachments=None):
    msg = MIMEMultipart('mixed')
    msg['From'] = sender
    msg['To'] = recipient
    msg['Subject'] = subject

    msg_body = MIMEMultipart('alternative')
    if text:
        text_msg = MIMEText(text, 'plain', 'utf-8')
        msg_body.attach(text_msg)
    if html:
        html_msg = MIMEText(html, 'html', 'utf-8')
        msg_body.attach(html_msg)
    msg.attach(msg_body)

    attachments = attachments or []
    for filename, data in attachments:
        attachment_msg = MIMEApplication(data)
        attachment_msg.add_header('Content-Disposition', 'attachment', filename=filename)
        msg.attach(attachment_msg)

    return msg.as_string()

def send_email(sender, recipient, subject,
               text=None, html=None, tag=None,
               attachments=None):
    client = boto3.client('ses')
    formatted_email = format_email(
        sender, recipient, subject, text=text, html=html, attachments=attachments)
    tags = [{'Name': 'campaign', 'Value': tag}] if tag else []

    client.send_raw_email(
        Source=sender,
        Destinations=[recipient],
        RawMessage={'Data': formatted_email},
        Tags=tags)
```

The [SES wrapper](https://github.com/Instapaper/ses-tools/blob/master/ses.py) above is available on GitHub.

SES has an email limit of 10MB versus Mailgun's 25MB. I added some logging to verify that a small portion of Kindle sends are over 10MB. In order to ensure that Kindle is being sent properly, I decided to perform the migration to SES as follows:

```
def send_mobi(sender, recipient, mobi_filename, mobi_data):
    """Send a mobi file to the user"""
    message = tornado_mailer.Message('Kindle delivery')
    message.sender = sender
    message.recipient = recipient
    message.attachments = [(mobi_filename, mobi_data)]
    message.tracking = False
    # SES total email limit is 10MB
    message.use_mailgun = len(mobi_data) / pow(1024.0, 2) > 9.5
    return message
```

Note that SES limit is 10MB for the total email, including all the headers, text, and attachment. I'm using 9.5MB as the limit for sending to SES to account for this extra space.

## Completing the Migration

To complete the migration for most use cases, we just need to set the `use_mailgun` flag to `False` by default. While the goal is to move off of Mailgun completely, there are a few remaining pieces of functionality that Mailgun is still powering:

1. Email validation: It is likely we will keep email validation, but seek a service with a volume-based pricing model.
2. Kindle sends over 10MB: We need to investigate a good solution here, I'm considering just sending these over SMTP directly from Instapaper servers rather than an email provider.
3. Inbound email saves: Instapaper provides every user with a unique @instapaper.com address where they can send links to save for later. Migrating this functionality to SES would work similarly as the bounce events we set up before:
![](/img/posts/ses-to-instapaper.png)
After setting up the receipt rule to receive emails with SES, we need to migrate our MX records to point at an Amazon SMTP server.

## Email Logs

As part of customer support, we sometimes look into email logs to check if emails were sent to the customer (e.g. forgot password emails, Kindle sends). Mailgun offers this logging out of the box in their customer dashboard.

With SES you can get similar functionality with a bit of additional work. SES allows you to create "Configuration Sets" that publish email events (e.g. Send, Delivery, Bounce, Complaint) to either CloudWatch, Simple Notification Service, or Kinesis Firehose. From Kinesis Firehose delivery stream you can create a destination to AWS-hosted Elasticsearch, which is a powerful document search service:

![](/img/posts/ses-to-elasticsearch.png)

Once the data is in Elasticsearch, you can use Kibana as a frontend to query it. I open sourced the [SES Elasticsearch wrapper](https://github.com/Instapaper/ses-tools/blob/master/search/ses_search.py) to make querying easier, and a [test script that counts events based on different inputs](https://github.com/Instapaper/ses-tools/blob/master/search/ses_event_count.py).

In order to get the data to flow into Kinesis, you just need to send the email with the appropriate configuration set:

```
client.send_raw_email(
    Source=sender,
    Destinations=[recipient],
    RawMessage={'Data': formatted_email},
    Tags=tags,
    ConfigurationSetName=configuration)
```

## Pricing

After it's all said and done, we estimate that our total cost for...

* 12M emails per month
* 3 dedicated IP addresses
* Hosted Elasticsearch with 350GB storage

... will be about $1,500 per month, which is roughly 40% cheaper than Mailgun.

## Learnings

While SES is a powerful service, it's also far more complicated to configure than Mailgun. Mailgun offers a lot of convenient functionality like automatically handling bounces/suppressions, easier email attachment support, and out-of-the-box logging. Additionally, it was fairly cheap to use Mailgun with a moderate email load. Without the Weekly email, it cost us $363 per month to send ~500k emails with 2 dedicated IP addresses.

With that said, their move to tier-based pricing makes it more costly to operate at larger volumes. It seems that the goal of Mailgun's pricing changes is to maximize their value per customer by increasing cost at larger scale. On the other hand, AWS trends to reduce cost over time in order to gain market share. I'm confident that over time if SES prices do change they will drop, not increase.

The savings here definitely feel small, about $1,000 monthly. However, for a modestly profitable service like Instapaper, cost control is the key to long-term success. Seemingly small savings add up and make it possible for us to [fulfill our commitment](https://blog.instapaper.com/post/176732408411) to ensure Instapaper is a product that withstands the test of time.

extension: markdown


text:
---
date: 2014-08-02
excerpts: 2
---

# Boltzmann machines and hierarchical models

The [restricted Boltzmann machine](http://en.wikipedia.org/wiki/Restricted_Boltzmann_machine) (RBM) is a key statistical model used in [deep learning](http://en.wikipedia.org/wiki/Deep_learning). They are special form of Boltzmann machines where the underlying graph is a bipartite graph. Personally, I am more interested in [Boltzmann machines](http://en.wikipedia.org/wiki/Boltzmann_machine) because they represent a class of discrete [energy models](http://www.cs.nyu.edu/~yann/research/ebm/) where the energy is quadratic. The dynamics of the model bears a lot of resemblance to those of [Hopfield networks](http://en.wikipedia.org/wiki/Hopfield_network) and [Ising models](http://en.wikipedia.org/wiki/Ising_model). As an aside, normal distributions are continuous energy models where the energy is quadratic and positive definite.

If the energy of the model is a polynomial of higher degree (e.g. cubic, quartic), then the model is _hierarchical_. They are a kind of [graphical model](http://en.wikipedia.org/wiki/Graphical_model) where the underlying graph is a [simplicial complex](http://en.wikipedia.org/wiki/Simplicial_complex) (a special type of [hypergraph](http://en.wikipedia.org/wiki/Hypergraph)). Here are some slides and papers on hierachical models:

1. [Hierarchical models and monomial ideals](http://www.fields.utoronto.ca/programs/scientific/11-12/graphicmodels/Wynn_Fields2.pdf)

_Daniel Bruynooghe and Henry Wynn_

2. [Toric ideals in algebraic statistics](http://www4.ncsu.edu/~smsulli2/Pubs/thesis.pdf)

_Seth Sullivant_

3. [Decomposable models](http://www.math.vt.edu/people/fhinkel/AlgebraicStatistics/DecomposableModels2.pdf)

_Franziska Hinkelmann_

4. [Betti numbers of Stanley-Reisner rings determine hierarchical Markov degrees](http://arxiv.org/abs/0910.1610)

_Sonja Petrović and Erik Stokes_

5. [Hierarchical models for contingency tables from the viewpoint of abstract simplicial complex](http://park.itc.u-tokyo.ac.jp/atstat/takemura-talks/0812casta.pdf)

_Akimichi Takemura_