text:
## Benchmarks for MIP rail scheduling
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.5171817.svg)](https://doi.org/10.5281/zenodo.5171817) [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/ROMSOC/benchmarks-mip-rail-scheduling/HEAD?labpath=Benchmark.ipynb)

This benchmark provides ten test cases for integrated locomotive scheduling and driver rostering problem in rail freight. They are developed in colaboration with DB Cargo Polska S.A.

## Description
The aim of this benchmark is to provide the basis for comparison of various solution algorithms and approaches against a realistic dataset, reflecting the trains to be performed in February 2020, the available locomotives and drivers, as well as information about the compatibilit.
All the benchmark cases are included in the ``instances`` directory.
A step-by-step guide on how to use these benchmarks can be found in ``documentation`` directory.

## Benchmark execution
The benchmark case can be run using the script files ``main.py`` at each instance subfolder.

## Run Jupyter notebooks
The entire benchmark repository can be executed in a provided Docker container where a full installation of Gurobi is available. Once you have clone or downloaded this repository, to build the container just type
```bash
docker build -t benchmarks-mip-rail-scheduling .
```
and for running it locally:
```bash
docker run -it --rm -p 8888:8888 benchmarks-mip-rail-scheduling jupyter-lab --ip=<IP_ADDRESS> --port=8888 --allow-root
```
Alternatively, please use the following link to run a user-friendly Jupyter Notebook (``Benchmark.ipynb``) with different instances. For instance, Benchmark ``1W_4`` is available at:
[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/ROMSOC/benchmarks-mip-rail-scheduling/HEAD?labpath=Benchmark.ipynb). Please, notice that mybinder cloud computations are limited to 2GB of RAM memory.

## Disclaimer
In downloading this SOFTWARE you are deemed to have read and agreed to the following terms:
This SOFTWARE has been designed with an exclusive focus on civil applications. It is not to be used
for any illegal, deceptive, misleading or unethical purpose or in any military applications. This includes ANY APPLICATION WHERE THE USE OF THE SOFTWARE MAY RESULT IN DEATH,
PERSONAL INJURY OR SEVERE PHYSICAL OR ENVIRONMENTAL DAMAGE. Any redistribution of the software must retain this disclaimer.  IF YOU DO NOT AGREE TO
THESE TERMS, DO NOT INSTALL OR USE THE SOFTWARE

## Acknowledgments

The ROMSOC project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie Grant Agreement No. 765374.
This repository reflects the views of the author(s) and does not necessarily reflect the views or policy of the European Commission. The REA cannot be held responsible for any use that may be made of the information this repository contains.

repository: ROMSOC/benchmarks-mip-rail-scheduling

repo_name: ROMSOC/benchmarks-mip-rail-scheduling


text:
# GNfiles allows to download/restore/update files used in GlobalNames projects

Working with multiple large files is cumbersome. To make it easier to download,
upload, and update files using a standard approach (IPFS in this case) allows
to keep data decentralized and reused with more ease.

This program is an exploration how to utilize features availabe at IPFS for
transferring files.

## Usage

Currently the app depends on running IPFS node on a local computer.
Install IPFS and run:

```bash
ipfs daemon &
```

### Upload directory to IPFS

A directory can by uploaded to IPFS if IPFS node is running locally.
Optionally, you can set a permalink using IPNS keys.

```bash
# To sync a directory to IPFS without permalink
gnfiles sync some_dir

# To sync a directory to IPNS with permalink (takes about a minute)
gnfiles sync somedir ipns-name

# You can find available key names with
ipfs key list
```

### Get directory from IPFS

The program rebuilds directory structure getting a meta-data file with
information about paths and files. In general it works as the following:

Files can be downloaded from anywhere, assuming that they are available.
If there are only available on one node, that node has to be running.

```bash
gnfiles get [source] [destination]
```

Where the source is a place to get metafile, and destination is a local
directory, probably empty, where to download all the files. If files with
the same name exist already, they will be overwritten.

```bash
# get metadata from local file system
gnfiles get /some/dir/_META.json target_dir

# get metadata from a url
gnfiles get https://ipfs.io/ipfs/Qmhash target_dir

# get metadata from an IPFS path (requires local IPFS node)
gnfiles get /ipfs/Qmhash target_dir
gnfiles get /ipns/k5hash target_dir

# get metadata from a CID (requires local IPFS node)
gnfiles get Qmhash target_dir
```

license: MIT

repository: gnames/gnfiles

branch_name: refs/heads/master

extension: md

license_type: permissive

repo_name: gnames/gnfiles


text:
# pynia

- - -
#### This is a fork of David Ng's (discontinued?) pynia project on Google Code: http://code.google.com/p/pynia/
- - -

# Getting started
## Dependencies
* http://code.google.com/p/pyglet/
* http://libusb.wiki.sourceforge.net/
* http://sourceforge.net/projects/pyusb/
* http://sourceforge.net/projects/numpy/
* http://webpy.org/

On Mac it's simply enough to run `pip install pyglet pyusb numpy web.py`. On GNU/Linux I had to compile and install the libusb code. Also, see the [Troubleshooting - Access Denied section](#access-denied-warning-on-gnulinux).

## Usage
There are two user interfaces for pyNIA: pyglet and HTML5.
### pyglet
![pyglet](/screenshots/pynia-pyglet.png)

The pyglet interface has a histogram of the 6 BrainFingers at top, a spectograph that I don't understand in the bottom left, and a general compiled waveform of the raw output from the device in the bottom right.

#### running it
I never got pyglet working on Mac, but it works fine on GNU/Linux. Simply run `python pynia.py`.
### HTML5
![html5](/screenshots/pynia-http.png)

[Click here for the annotated version](/screenshots/pynia-http-annotated.png). The HTML5 interface has the same histogram as the pyglet version at top, but the other two images are different. The bottom left has a distorted hexagon I have nicknamed the Brain Shape, where the distortion to each vertex is determined by a specific BrainFinger. The hexagon is bluer when all frequencies are low, redder when beta frequencies are high, and greener when alpha frequencies are high. The bottom right is a historical graph of all six BrainFinger values, instead of a single compiled waveform.

#### running it
1. You will need to install CoffeeScript, have `coffee` on your path, and run `batch/coffee-compile.sh`. Alternatively, you can use [Coffee2JS](http://js2coffee.org/#coffee2js) to convert `/src/coffee/index.coffee` and save the output to `static/js/index.js`.
2. Once you have compiled the CoffeeScript code, run `python http.py` and visit `http://localhost:8080`.

# Troubleshooting
#### Versions
I ran into quite a few problems getting **pynia** working. David Ng's project
had 2 fairly different versions of the code. I'm going to be developing
primarily off the version **0.0.1** codebase because it required less
modifications to make it functional, and it also has much better documentation.
#### "Access Denied" warning on GNU/Linux
By default, libusb doesn't provide read/write access to USB devices. In order to
get **pynia** working on Linux without root privileges, you need to add an
exception for the NIA to your udev rules in `/etc/udev/rules.d` (a sample file
has been included in the `udev` folder of this repo), and then you need to run
the command `udevadm control --reload-rules` to reload the rules. After that,
unplug the NIA and plug it back in.

# License #
#### [MIT License](http://opensource.org/licenses/mit-license.php)

# Credits
The original documentation of the project follows:

>#### This is a simple GUI to play with the NIA on Mac and Linux until OCZ release a set of drivers for these platforms.
>
>It requires the following dependencies:
>* http://code.google.com/p/pyglet/
>* http://libusb.wiki.sourceforge.net/
>* http://sourceforge.net/projects/pyusb/
>* http://sourceforge.net/projects/numpy/

repository: herrlustig/pynia

repo_name: herrlustig/pynia


text:
---
layout: post
title: Ethereum & Solidity
last_updated: August 09, 2022
---

Resources about Solidity.

* [Crypto Zombies][1]{:target="_blank"} : Learn to Code Blockchain DApps By Building Simple Games.

* [Solidity by Example][2]{:target="_blank"} :An introduction to Solidity with simple examples.

* [Solidity Language Documentation][3]{:target="_blank"} : Solidity Language official site

* [Solidity, Blockchain, and Smart Contract Course in YT][4]{:target="_blank"} : Learn to Code Blockchain DApps By Building Simple Games, by freeCodeCamp.org.

Resources about Ethereum

* In progress..

[1]: https://cryptozombies.io/
[2]: https://solidity-by-example.org/
[3]: https://docs.soliditylang.org/en/v0.8.15/solidity-by-example.html
[4]: https://www.youtube.com/watch?v=M576WGiDBdQ


text:
Today I will be working on this Kata.
The code provided is supposed replace all the dots . in the specified String str with dashes -

But it's not working properly.

https://www.codewars.com/kata/fixme-replace-all-dots

used this to learn more about RegExp
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions

repository: Calamario/code-wars

gha_language: JavaScript

gha_license_id: MIT

repo_name: Calamario/code-wars


text:
# Replacing a static asset with Lottie
One of the primary motivations behind Lottie is to make shipping animations just as easy as shipping static assets.

### Switch from static assets to animations with 3 lines of code!

## Static Asset Workflow
Put your pngs or vector drawables (xml) in `res/drawable`
Include it in your layout
```xml
<ImageView
        android:id="@+id/image_view"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:scaleType="centerCrop"
        android:src="@drawable/your_drawable" />
```

## Lottie workflow

Put your lottie json in `res/raw`
Include it in your layout:
```xml
<com.airbnb.lottie.LottieAnimationView
        android:id="@+id/animation_view"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:scaleType="centerCrop"
        app:lottie_rawRes="@raw/your_animation"
        app:lottie_loop="true"
        app:lottie_autoPlay="true" />
```

`LottieAnimationView` extends `ImageView` so you can start by simply replacing your ImageViews with LottieAnimationViews even if you are still using drawables!


text:
# Alfred2 'pwme' workflow

> Alfred2 workflow wrapper for `pwgen`

Generate secure random passwords using pwgen and copy to the clipboard.

## Prerequisites / installation

`pwme` depends on `pwgen` and expects `pwgen` binary to be available at `/usr/local/bin/pwgen`

`pwgen` is available as a Homebrew package, to install simply run:

```bash
brew install pwgen
```

## Usage

`pwme {numberOfChars}`

{numberOfCharacters} is optional, defaults to 8 if omitted.

### Under the hood

Calls pwgen with the following arguments:

```bash
pwgen -cnysB1
```

## License

Apache-2


text:
---
title: teedoc
keywords: teedoc, markdown, jupyter notebook, html, document generation, alternative gitbook, website generation, static website
desc: teedoc, convert markdown or jupyter notbook into html static webpage
---

Official website: [teedoc.github.io](https://teedoc.github.io/)
Source file of this document: [github.com/teedoc/teedoc.github.io](https://github.com/teedoc/teedoc.github.io)
Source code: [https://github.com/teedoc/teedoc](https://github.com/teedoc/teedoc) Welcome star

Convert documents in `Markdown` or `Jupyter Notebook` format to `HTML` web pages

`teedoc` can be used in the following scenarios:
* Build a document website, and preferably support multiple documents and custom pages
* Build a `WiKi` website
* Build personal or corporate knowledge base
* Build personal or corporate website

## Features

- [x] Simple to use, cross-platform, only dependent on `Python3`
- [x] The deployment is simple, the generated website is a fully static page, which can be deployed directly by copying to the server or uploading to a third party organization
- [x] Easy to write, using Markdown syntax
- [ ] Jupyter notebook support
- [x] Multi-document support
- [x] Plug-in support
- [x] Multi-theme support (implemented by plugin)
- [x] Control the style accurate to the page through css (implemented by customizing the id and class of each page)
- [x] Multi-level directory support
- [x] Multi-language support (manual translation) (Internationalization/i18n)
- [ ] Multilingual support (automatic translation)
- [x] Multi-version support (implementation method is the same as multi-language)
- [x] Search support
- [x] SEO friendly
- [x] Real-time preview of changes
- [x] Multi-threaded construction, faster construction speed
- [ ] Blog support

## Similar tools

In fact, there are already many tools of this type. Just choose one according to your needs.

* docusaurus: The UI layout of teedoc is almost similar to it, but it uses vue to write, teedoc is native js, if you are using vue, you can consider this
* gitbook: a tool that used to be very useful, but it is no longer maintained by the official website
* docsify: Only one page is needed. Markdown is rendered in the browser instead of pre-rendered into HTML. The advantage is that it is lightweight, but SEO is not friendly. You can use its SSR function, written in nodejs
* readthedocs: A tool used by many open source projects. It also has a website service like gitbook. You can start writing documents after registering and logging in. You can also download the software to generate the website yourself, which is friendly to RST format support

If you have choice difficulties, you are recommended to use teedoc if you meet some of the following conditions:
* Does the function meet your needs?
* Does the interface meet your aesthetics (you can customize css, or change the theme plug-in)
* Familiar with Python? You can customize the plugin at any time

## Some usage suggestions

* Add `Generate with teedoc` in footer to help more people discover teedoc and promote project growth
* Use the template project to start a new document project, you can run it first, and then modify it according to your own needs, so that you can get started faster

repository: BigQubot/WhatToDoNext

branch_name: refs/heads/main

repo_name: BigQubot/WhatToDoNext


text:
---
name: feathers
template: default.hbt
---

## Why feathers

feathers is a starting point for your next [Angularjs](https://angularjs.org/)
project. Here we'd like to share a lot of best practices on coding with AngularJS
with you.

feathers use the following open sources projects:
- [AngularJS](https://github.com/angular/angular.js)
  AngularJS lets you write client-side web applications as if you had a smarter
  browser.
- [Angular-ui Router](https://github.com/angular-ui/ui-router)
  The de-facto solution to flexible routing with nested views
- [karma](https://github.com/karma-runner/karma)
  A simple tool that allows you to execute JavaScript code in multiple real
  browsers.
- [protractor](https://github.com/angular/protractor)
  Protractor is an end-to-end test framework for AngularJS applications
- [mocha](https://github.com/visionmedia/mocha)
  Mocha is a simple, flexible, fun JavaScript test framework for node.js and the
  browser. For more information view the documentation.
- [chai](https://github.com/chaijs/chai)
  Chai is a BDD / TDD assertion library for node and the browser that can be
  delightfully paired with any javascript testing framework.
- [sinon](https://github.com/cjohansen/Sinon.JS)
  Standalone and test framework agnostic JavaScript test spies, stubs and mocks.

Moreover, with these cool things.
- [jade](https://github.com/visionmedia/jade)
- [node-sass](https://github.com/sass/node-sass)
- [coffeescript](https://github.com/jashkenas/coffeescript)

---

## Getting Start

### Quick Start

```
$ git clone https://github.com/Wiredcraft/feathers.git
$ cd feathers
$ npm install && bower install
$ gulp dev
```

### Development

First of all, you should start an *dev server*.

```
$ gulp dev
```

### Tests

#### Unit Tests

libs:
- [karma](https://github.com/karma-runner/karma)
- [mocha](https://github.com/visionmedia/mocha)

```
$ karma start
```

#### e2e Tests

libs:
- [protractor](https://github.com/angular/protractor)
- [mocha](https://github.com/visionmedia/mocha)

```
$ webdriver-manager start
$ gulp dev
$ gulp test:e2e
```

### Build Documentation
- [metalsmith](https://github.com/segmentio/metalsmith)

```
$ cd docs
$ npm install
$ node metalsmith.js
$ node web-server.js
```

Now go to [http://localhost:3000](http://localhost:3000).

### Deploy Document

```
$ cd ..
$ pwd
~/workspace/feathers
$ git add -f docs/build && git commit -m "commit for gh-pages"
$ git sutree push --prefix docs/build origin gh-pages
```

---

## Contributing

1. Fork it
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Add some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
5. Create new Pull Request

---

## About

- [Versions](https://github.com/Wiredcraft/feathers/releases)
- [Change log](https://github.com/Wiredcraft/feathers/commits/master)
- [Authors](https://github.com/Wiredcraft/feathers/graphs/contributors)

Built by .

repository: markandrewj/feathers-bootstrap

repo_name: markandrewj/feathers-bootstrap


text:
# Firebase Cookbook
This is the code repository for [Firebase Cookbook](https://www.packtpub.com/web-development/firebase-cookbook?utm_source=github&utm_medium=repository&utm_campaign=9781788296335), published by [Packt](https://www.packtpub.com/?utm_source=github). It contains all the supporting project files necessary to work through the book from start to finish.
## About the Book
Do you feel tired just thinking or even hearing about backend technologies, authentication or the tedious task of deployment? Firebase is here to change the way you develop and make your app a first-class citizen of the cloud.

This books takes a solution based approach by providing you recipes that would help you understand the features of Firebase and implement them in your existing web or mobile applications.

We start-off by creating our first Firebase application and integrating its services into different platforms and environments for mobile as well as web applications. Then we deep dive into Real-time Database and Firebase Storage that allows your users to access data across various devices with realtive ease. With each chapter you will gradually create the building blocks of your application from securing your data with Firebase Rules to authenticating your users with O-Auth.

Moving along we would explore modern application development techniques such as creating serverless applications with Firebase Cloud Functions or turning your traditional applications into progressive apps with Service workers.

Finally you will learn how to create cross-platform mobile apps, integrate Firebase in native platforms, and learn how to monetize your mobile applications using Admob for Android and iOS.

## Instructions and Navigation
All of the code is organized into folders. Each folder starts with a number followed by the application name. For example, Chapter02.

The code will look like the following:
```
//Getting the file reference.
var rootRef = firebase.storage().ref();
var imageRef = rootRef.child('images/<image-name>.<image-ext>');
```

The content requirements are fairly easy for the first 10 chapters; no matter what your operating system or your code editor is, you're always covered.
However, in Chapter 11, Integrating Firebase with Android/iOS Natively, we'll start developing for mobile, so while you will be perfectly fine with any operating system (macOs, Linux, or Windows) when developing for Android, we will definitely need a macOS-based computer in order to follow along with iOS-based recipes.

## Related Products
* [Build a Reddit-Like Messaging Board with React and Firebase [Video]](https://www.packtpub.com/web-development/build-reddit-messaging-board-react-and-firebase-video?utm_source=github&utm_medium=repository&utm_campaign=9781788479431)

* [Learning Firebase [Video]](https://www.packtpub.com/web-development/learning-firebase-video?utm_source=github&utm_medium=repository&utm_campaign=9781784392321)

repository: satheeshkumars/Firebase-Cookbook

repo_name: satheeshkumars/Firebase-Cookbook


text:
# ListEdit
A Tool to edit SparkPost stored lists

This is a simple tool that will:
 - connect to your SparkPost account
 - list all your stored recipients lists
 - allow you to select one for edits
 - perform edits
 - replace the list with a new one complet ewith edits
 - also gives you the ability to upload or download entire lists as needed

## Setup
Clone this repo, then modify env.php with your API key

Then point your browser to https://<YourWebServerLocation>/listedit

repository: tommairs/listedit

repo_name: tommairs/listedit


text:
# generator-react-divilon [![Build Status][travis-image]][travis-url]
Generate opinionated react applications.

It generates basic react application using
webpack and babel to process `JSX`.

Here in Divilon we often doing front-end for java application.
So generator supports source layouts feasible for `gradle` and `maven`
project source layout.

## Features

### Layouts
Generator supports laying out sources in a `gradle` and `maven` friendly manner.
For both maven and gradle sources are generated into `src/main/webapp` directory.
Build results are in `target/www` and `build/www` for maven and gradle respectively.
Please take a look at brilliant [JHipster](https://jhipster.github.io/) for insights
on how to handle this type of layout with java build tools (more support for
maven and especially gradle is planned)

### Proxying
When running on top of existing server it is handy to proxy this server while
developing front-end. When you have Spring Boot REST service as backend you can
proxy it and get all benefits of live reloading and HMR with your webpack dev server
running on different port. Default dev server port is 8080, however if proxied url
is on default port it changes to `8081`.

## Generated application
`react-divilon` generates [React](https://facebook.github.io/react/docs/getting-started.html)
based front-end application backed by [Redux](https://github.com/reactjs/redux)

Application is build using [webpack](http://webpack.github.io/).
Dev server is added to run either as standalone server of proxy to existing REST backend.
Dev server enables rebuilds on file change as well as browser live reloading.

### Application layout
```
+ actions/           # redux actions and action creators
+ components/        # application components
+- hello/            # directory for particular component
+ reducers/          # contains application reducers
+- index.js          # groups reducers by properties and exports it as object
+ application.js     # application entry point
+ configureStore.js  # set up redux store with middlewares and combined reducers
```

## Installation

First, install [Yeoman](http://yeoman.io) and generator-react-divilon using [npm](https://www.npmjs.com/) (we assume you have pre-installed [node.js](https://nodejs.org/)).

```bash
npm install -g yo
```

This generator is not published to npm yet.
In order to use it clone it from github and link

```bash
git clone git@github.com:Divilon/generator-react-divilon.git
cd generator-react-divilon
npm link
```

Then generate your new project:

```bash
yo react-divilon
```

[travis-image]: https://travis-ci.org/Divilon/generator-react-divilon.svg?branch=master
[travis-url]: https://travis-ci.org/Divilon/generator-react-divilon

repository: Divilon/generator-react-divilon

repo_name: Divilon/generator-react-divilon


text:
# Nilable options

You may have noticed that **Clip** adds in the help message if the option has a default value or is required.
It looks like this:

```console
  --repeat INTEGER  [default: 1]
```

or

```console
  --repeat INTEGER  [required]
```

Sometimes you may not want those messages to be shown.
Maybe you want the option to be optional and have a default value, but you don't want to expose this value to the user.

We can do this by making the attribute's type nilable and set its default value `nil`:

```Crystal hl_lines="9 29"
require "clip"

module Myapplication
  VERSION = "0.1.0"

  struct Command
    include Clip::Mapper

    getter repeat : Int32? = nil
    getter name : String
  end

  def self.run
    begin
      command = Command.parse
    rescue ex : Clip::Error
      puts ex
      return
    end

    if command.is_a?(Clip::Mapper::Help)
      puts command.help
    else
      hello(command.name, command.repeat)
    end
  end

  def self.hello(name, repeat)
    if repeat.nil?
      puts "Hello #{name}"
    else
      repeat.times { puts "Hello #{name}" }
    end
  end
end

Myapplication.run
```

!!! Tip
    As our attribute does have a default value we don't have to explicitly add the `Clip::Option` annotation.

!!! Note
    `command.repeat` is now either an `Int32` or a `Nil` value, so we can't directly call `#times` on it or the compiler will complain that this method is not defined for `Nil`.

The help now looks like this:

```console hl_lines="11"
$ shards build
Dependencies are satisfied
Building: myapplication
$ ./bin/myapplication --help
Usage: ./bin/myapplication [OPTIONS] NAME

Arguments:
  NAME  [required]

Options:
  --repeat INTEGER
  --help            Show this message and exit.
```

And the behavior is still the same:
```console
$ ./bin/myapplication Alice
Hello Alice
$ ./bin/myapplication --repeat 2 Alice
Hello Alice
Hello Alice
```

Whether you choose to use a default `nil` value or not is up to you.

repository: erdnaxeli/clip

path: /docs/user guide/options/nilable.md

repo_name: erdnaxeli/clip


text:
### Create and enable connection in Auth0

[Set up the Apple social connection](https://auth0.com/docs/dashboard/guides/connections/set-up-connections-social) in Auth0. Make sure you have the generated IDs and keys:

* Services ID (Client ID)
* Client Secret Signing Key
* Apple Team ID
* Key ID

Besides the standard social connection settings, the Apple social connection configuration includes the following:

| Setting | Description |
|---------|-------------|
| Client Secret Signing Key | The content of the file that you downloaded when registering the Key. The file has a '.p8' extension. |
| Team ID | ID of the organization Apple approved to receive their development kit. Developer accounts are associated with an approved team. You can find your Team ID in the Apple Developer Portal under **Membership Details**. |
| Key ID | ID of the key that Apple issued to allow your app to perform Sign in With Apple. You can find your Key ID in the Apple Developer Portal under **Certificates, Identifiers & Profiles**. |


text:
[![PyPI version](https://badge.fury.io/py/schematable.svg)](https://badge.fury.io/py/schematable)
[![Build Status](https://travis-ci.com/json2d/schematable.svg?branch=master)](https://travis-ci.com/json2d/schematable)
[![Coverage Status](https://coveralls.io/repos/github/json2d/schematable/badge.svg?branch=master)](https://coveralls.io/github/json2d/schematable?branch=master)

# schematable

a Python utility library for working with SQL tables

## Install

```
pip install schematable
```

## Basic usage

Go from zero to SQL query in seconds with near-zero boilerplate:

```py
import schematable as st

schdl = st.SchemaTable('schedule') # at minimum it needs a table name

print(schdl.db_url) # sqlite:///:memory:
print(schdl.engine) # <class 'sqlalchemy.engine.base.Engine'>

create_table_stmt = '''
  create table schedule
  (
    start_time timestamp,
    end_time timestamp,
    event_name text,
    id int not null
      constraint schedule_pk
        primary key
  );
'''

schdl.engine.execute(create_table_stmt)

insert_records_stmt = '''
  INSERT INTO schedule (start_time, end_time, event_name, id) VALUES ('1583531261000', '1583534865000', 'walk the doggy 🐶', 1);
'''

insert_records_stmt_2 = '''
  INSERT INTO schedule (start_time, end_time, event_name, id) VALUES ('1583708400000', '1583632800000', 'take a nap 😴', 2);
'''

schdl.engine.execute(insert_records_stmt)
schdl.engine.execute(insert_records_stmt_2)

rows = schdl.engine.execute('select event_name from schedule').fetchall()

print(rows) # ['walk the doggy 🐶', 'take a nap 😴']

```

## More advanced usage

With `sqlalchemy` under the hood you can connect to tables from all types of SQL databases:

```py
import schematable as st

schdl = st.SchemaTable(
  db_url='postgres://user:password@localhost:5432/foo',
  schema='bar',
  table='schedule'
)

```

`schematable` also integrates with `pandas` workflows like a charm, reducing the noise and friction involved with managing the arguments for their SQL related functions:

```py
import pandas as pd
import datetime

df = pd.read_sql_table(schdl.table, schdl.engine, schld.schema) # neat - everything's in one place

df['event_name']
df['start_time'] > datetime.datetime.now()
df['extracted_time'] = datetime.datetime.now()
```

## Cross database workflow

Working with multiple databases at the same time should be dead simple.

Eg. Lets say we wanted to create a new local test database on-the-fly with some data queried from our production database.

Here's the `schematable` + `pandas` way:

```py
import schematable as st
import pandas as pd

# extract dataset from production db

schdl = st.SchemaTable(
  db_url='postgres://user:password@localhost:5432/foo',
  schema='bar',
  table='schedule'
)

# select everything from 2019

select_dataset_query = '''
  select * from {schema_table}
  where start_date between date('{from_date}') and date('{to_date}')
'''.format(
  schema_table=schdl.st,
  from_date='2019-01-01',
  to_date='2019-12-31'
)

df = pd.read_sql(select_dataset_query, schdl.table, schdl.engine)

# load dataset into (new) test db

test_schdl = st.SchemaTable(
  db_url='sqlite:///db/test-2019.sqlite',
  table='schedule'
)

df.to_sql(test_schdl.table, test_schdl.engine, if_exists='replace', index=false)

```

## Extended URLs

Take a standard db URL and append a schema and table to the end - you get what we call a  **schematable URL**, which can be parsed into a `SchemaTable` instance:

```py
from schematable import SchemaTable

schdl = SchemaTable.parse('postgres://user:password@localhost:5432/foo#bar.schedule')

print(schdl.db_url) # postgres://user:password@localhost:5432/foo
print(schdl.schema) # bar
print(schdl.table) # schedule

```

They're primarily handy for usecases involving serialization and deserialization of schematables:

```env
# staging.env
SCHEDULE_SCHEMATABLE_URL=sqlite:///staging.db#bar.schedule
```

```py
schdl = SchemaTable.parse(os.environ['SCHEDULE_SCHEMATABLE_URL'])

print(schdl.db_url) # sqlite:///staging.db
print(schdl.schema) # bar
print(schdl.table) # schedule
```

More specifically, schematable URLs are composed of a `db_url`, `table`, and `schema` component where (just like the `SchemaTable` constructor) only the `table` component is required:

```py
SchemaTable.parse('schedule')
SchemaTable.parse('bar.schedule')
SchemaTable.parse('sqlite:///staging.db#schedule')
SchemaTable.parse('postgres://user:password@localhost:5432/foo#bar.schedule')
```

## Boilerplate reduction

Here are before-and-after snippets showing some code you won't have to write anymore using `schematable` to do your SQL things:

Eg. Here's checking if a table exists:

#### before
```py
import sqlalchemy as sa

schdl_db_url = 'sqlite://'
schdl_table = 'schedule'
schdl_engine = sa.create_engine(schdl_db_url)
if schdl_engine.has_table(schdl_table):
  # do the thing
```

#### after
```py
import schematable as st

schl = st.parse('schedule')
if schdl.engine.has_table(schdl.table, schdl.schema):
  # do the thing
```

#### after next
```py
import schematable as st

schl = st.parse('schedule')
if schdl.exists():
  # do the thing
```


text:
# MiaServer

This provides an environment for running Mia games, where the players are bots programmed by participants in an exercise session.

The game is run by a server (provided here). The players communicate with that server using a simple text-based protocol over UDP.

## Running

If required, edit config files in `/config` to adjust server port, timeout, and logging level.

Start the server with:

> mix mia.server

## The MIA Protocol

### Communication between server and clients

* Communication by lines of data via UDP (UTF-8 encoded strings, newline terminated)
* The server opens an UDP port (default 4080)
* Clients send a message to the server to register as players, and will get notified from now on the start of each new round
* The clients have a rather short time window for any replies (default 250 ms)
* During the game rounds, the server will provide a unique UUID token together with each message send to the active player/players. The player must include this token in his reply message.
* A client can also register itself as a spectator. A spectator cannot participate actively, but gets all messages that are broadcast to all clients.

### Registration
* client->server: `REGISTER;<name>`
* client->server: `REGISTER_SPECTATOR`

If `<name>` is valid and not used already:
* The server will now communicate with the client via the source IP/port of the registration message
* server->client: `REGISTERED`

If the name was already registered from the same IP:
* The server changes to the new port for further communication
* server->client: `ALREADY REGISTERED`

Otherwise:
* server->client: `REJECTED`

Criteria for a valid player name:
* no whitespace, semicolons, colons, or commas
* not longer than 20 characters

### Invitation to a new game round
* server->clients: `ROUND STARTING;<token>`
* client->server: `JOIN;<token>`

If at least one player joins:
* The joined players are shuffled randomly
* server->clients: `ROUND STARTED;<round_number>;<player names>`
`<player names>` as a comma-separated list of participating players, in the order how they will play in this round

Otherwise:
* server->clients: `ROUND CANCELED;NO PLAYERS`
In that case, a new round is started.

Rounds with only one participating player are cancelled immediately after they started:
* server->clients: `ROUND CANCELED;ONLY ONE PLAYER`

### Playing a round
In order of the shuffled player sequence:
* server->client: `YOUR TURN;<token>`
* client->server: `<command>;<token>`
`<command>` one of: **ROLL**, **SEE**

#### ROLL:
* server->clients: `PLAYER ROLLS;<name>`
* server->client: `ROLLED;<dice>;<token>`
* client->server: `ANNOUNCE;<announcement>;<token>`
* server->clients: `ANNOUNCED;<name>;<announcement>`

`<dice>` and `<announcement>` format are two comma-separated die values: `<dice1>,<dice2>`

In case of announced MIA (`2,1`) the roll is immediately revealed:
* If a MIA was really rolled, all other players lose
* If any other combination was rolled, the announcing player loses

* server->clients: `PLAYER LOST;<names>;<reason>`
`<names>` a comma-separated list of the losing player names

#### SEE:
The server determines whether the formerly announcing player has bluffed or correctly announced, and determines who has lost
* server->clients: `PLAYER WANTS TO SEE;<name>`
* server->clients: `ACTUAL DICE;<dice>`
* server->clients: `PLAYER LOST;<name>;<reason>`

#### Timeout or any other incorrect behaviour
* server->clients: `PLAYER LOST;<name>;<reason>`

#### Aftermath of a round:
* server->clients: `SCORE;<scores*>`
`<scores*>` as a comma-separated list of entries `<name>:<score>`

#### Messages indicating reasons why player/players lost:

* `SEE BEFORE FIRST ROLL`: Player wanted to see, but he was the first player in a round (no announcement was yet made)
* `ANNOUNCED LOSING DICE`: Announcement was lower or equal to the last announced dice
* `DID NOT ANNOUNCE`: Player did not announce before timeout limit
* `DID NOT TAKE TURN`: Player did not send a command before timeout limit
* `INVALID TURN`: Player message was invalid in any way (malformed, wrong dice, wrong command etc.)
* `SEE FAILED`: Player wanted to see, but the announcement was correct
* `CAUGHT BLUFFING`: Player announced more than he actually rolled, and next player wanted to see
* `LIED ABOUT MIA`: Player announced MIA without having rolled a MIA
* `MIA`: Player announced and rolled a MIA; all other players lose this round


text:
# Numpy
NumPy is a python library used for working with arrays.  It also has functions for working in domain of linear algebra, fourier transform, and matrices.

# Why Use NumPy ?
In Python we have lists that serve the purpose of arrays, but they are slow to process.

NumPy aims to provide an array object that is up to 50x faster that traditional Python lists.

The array object in NumPy is called ndarray, it provides a lot of supporting functions that make working with ndarray very easy.

Arrays are very frequently used in data sciences, where speed and resources are very important.

repository: pykrishna/numpy

repo_name: pykrishna/numpy


text:
# puppet-roles

 [![Build Status](https://secure.travis-ci.org/icalvete/puppet-roles.png)](http://travis-ci.org/icalvete/puppet-roles)

#Actions

* Roles usefull for https://github.com/icalvete?tab=repositories

## Requires:

* [hiera](http://docs.puppetlabs.com/hiera/1/index.html)

### syslog_remote_server

Enable AWS Glacier backup require https://github.com/icalvete/puppet-awstools (included in https://github.com/icalvete?tab=repositories)

## Example:

```puppet
node 'zoth-ommog.smartpurposes.net' inherits sp_defaults {
  include roles::puppet_master
  include roles::puppet_db
  include roles::puppet_dashboard
  include roles::syslog_sender_server
  include roles::mysql_server
}
```

## Authors:

 Israel Calvete Talavera<EMAIL_ADDRESS>

repository: icalvete/puppet-roles

gha_language: Puppet

repo_name: icalvete/puppet-roles


text:
My Bash Profile
===============

Grabbed from https://github.com/matschaffer/profile, and slowly but surely ripped to pieces from there.

Installation
------------
On a brand new mac, to set things up the way *Mike* wants things, try this:

    curl -O https://raw.githubusercontent.com/sixty4k/profile/master/bin/rebuild_mac.sh && bash ./rebuild_mac.sh

    curl -s https://raw.githubusercontent.com/sixty4k/profile/master/install | bash

From here you can now put any `*.conf` in `~/.profile.d` and it'll get loaded in alphabetical order. You can also make directories that end in `.conf` which profile will then load all the `.conf` files in that directory in order.

Load order
----------

This profile loads `init` first, followed by all the `.conf` files in `.profile.d/core`, then any `.conf` files in `.profile.d`. This way you can put any general use configs in `core` and anything personal in `.profile.d`. If you want to keep your personal profile information in git, just set up `core` as a submodule to your personal repository.

Features
--------


text:
# Convert ER Diagrams To Schemas

## Step 1: Mapping of Regular Entity Types
For each regular entity type create a relation (table) that includes all the simple attributes of that entity.

## Step 2: Mapping of Weak Entity Types
For each weak entity type create a relation (table) that includes all simple attributes of the weak entity.

The primary keys of the weak entity should also include the primary key of the owner of this weak entity.

## Step 3: Mapping of Binary 1:1 Relationship Types
Include one side of the relationship as a foreign key in the other Favor total participation.

The entity that has total participation should keep the primary key of the other entity as a foreign key in it.

## Step 4: Mapping of Binary 1:N  Relationship Types
Include the 1 side's primary key as a foreign key on the N side relation (table).

## Step 5: Mapping of Binary N:M Relationship Types
Create a new relation (table) who's primary key is a combination of both entites's primary keys. Also include any relationship attributes.

## Company Database Schema
![Company Database Schema](https://www.mikedane.com/databases/sql/company-relations.png)
Source: [SQL Tutorial by Mike Dane](https://www.mikedane.com/databases/sql/company-relations.png)

repository: ZhongkaiLiu/learnsql

repo_name: ZhongkaiLiu/learnsql


text:
# Debugging

## How it works

Should the worst happend and an unexpected error raises, ApplicationStarter provides a global *Error* handler that logs to stderr the error message if present, and dumps a fuel file (by default _logs/prefix-timestamp.fuel_) with `thisContext` this can be then loaded using fuel and debugged later.

## How to debug it

You need:

- An image with your code loaded.
  - _If you can use the same image where the error generated better, otherwise, you need to have all the code loaded, otherwise fuel will not be able to deserialize the context_.
- The dump file **the-file.fuel** in this example.
- Materialize the fuel file by any of the following methods:
  - By dragging it to the image
  - Opening it from the file browser
  - Execute the following code on a Playground

```smalltalk
| session |

session := Processor activeProcess
  newDebugSessionNamed: 'External stack'
  startedAt: (FLMaterializer materializeFromFileNamed: 'the-file.fuel').

Smalltalk tools debugger openOn: session.
```

repository: ba-st/ApplicationStarter

branch_name: refs/heads/release-candidate

gha_language: Smalltalk

repo_name: ba-st/ApplicationStarter


text:
# gpx-creator
Generate a GPX which can be imported into c:geo.
One use case is to be able to create a PMO cache entry in c:geo without being a PM if you have access to the cache coordinates at least (e.g. though a PM friend).
The rest of the information (cache name, type and size, D/T rating, owner) can be retrieved from geocaching.com.


text:
#### Component Accessibility

In our efforts for our front-ends to meet accessibility requirements, we implement supplemental automated accessibility testing via `jest-axe`, a custom jest matcher for testing accessibility. Since it is a jest plugin, this allows us to write tests for components where we would be conerned about any accessibility requirements being met. These sort of tests can be placed right alongside existing unit tests, and they can be parameterized to run the same test over our provided screen sizes by using the `jest-each` plugin, which in general enables parameterized tests.

Automated accessibility provide a safeguard around common issues and can catch ~30% of issues.

Packages
- jest-axe
- jest-each

```typescript
import { axe, toHaveNoViolations } from 'jest-axe';
import each from 'jest-each';
expect.extend(toHaveNoViolations)

describe('component: MenuBar', () => {

    // ... other tests

    describe("accessibility", () => {
        each([
            ["tiny", SIZES.tiny],
            ["small", SIZES.small],
            ["medium", SIZES.medium],
            ["large", SIZES.large],
            ["extraLarge", SIZES.extraLarge]
        ]).test("has no violations on screen size '%s'", async (text, screen_size) => {
            const { container } = render(
                <StyleProvider>
                    <ScreenSizeContext.Provider value={screen_size}>
                        <Menu />
                    </ScreenSizeContext.Provider>
                </StyleProvider>
            );

            const results = await axe(container);

            expect(results).toHaveNoViolations();
        });
    });
});

```

You will get output
```bash
component: MenuBar
    ...
    accessibility
      ✕ has no violations on 'tiny' (667 ms)
      ✕ has no violations on 'small' (715 ms)
      ✕ has no violations on 'medium' (723 ms)
      ✕ has no violations on 'large' (655 ms)
      ✕ has no violations on 'extraLarge' (692 ms)

  ● component: MenuBar › accessibility › is accessible on 'tiny'

    expect(received).toHaveNoViolations(expected)

    Expected the HTML found at $('#mb-button') to have no violations:

    <button id="mb-button" role="button" aria-haspopup="menu" aria-controls="mb-popup" type="button" class="sc-gyUeRy fUyNFh">

    Received:

    "Buttons must have discernible text (button-name)"

    Fix any of the following:
      Element does not have inner text that is visible to screen readers
      aria-label attribute does not exist or is empty
      aria-labelledby attribute does not exist, references elements that do not exist or references elements that are empty
      Element has no title attribute
      Element's default semantics were not overridden with role="none" or role="presentation"

    You can find more information on this issue here:
    https://dequeuniversity.com/rules/axe/4.2/button-name?application=axeAPI

      202 |       const results = await axe(container);
      203 |
    > 204 |       expect(results).toHaveNoViolations();
          |                       ^
      205 |     })
      206 |
      207 |   });

      at src/MenuBar/MenuBar.test.tsx:204:23

    ...
```

Which gives a good indication on first steps to improving accessibility.

Basically, we want to put some components together, audit them, and fix as necessary. Improving this process will help mitigate issues as the framework itself is adopted across the organization.

One more example but with specifically the MenuButton component,
```typescript
describe('component: MenuButton', (): void => {

  // ... other tests

  describe("accessibility", () => {
    each([
      ["tiny", SIZES.tiny],
      ["small", SIZES.small],
      ["medium", SIZES.medium],
      ["large", SIZES.large],
      ["extraLarge", SIZES.extraLarge]
    ]).test("is accessible on '%s'", async (text, screen_size) => {
      const actionOne = jest.fn()
      const actionTwo = jest.fn()

      const { container} = render(
        <StyleProvider>
          <ScreenSizeContext.Provider value={screen_size}>
          <MenuButton>
            <MenuButton.Item onSelect={actionOne}>Action One</MenuButton.Item>
            <MenuButton.Item onSelect={actionTwo}>Action Two</MenuButton.Item>
            <MenuButton.Link href="#">Action Three</MenuButton.Link>
          </MenuButton>
          </ScreenSizeContext.Provider>
        </StyleProvider>
      )

      const results = await axe(container);

      expect(results).toHaveNoViolations();
    })

  });
})

```

With output (which is condensed to two of the failures, which show different issues on screen size)
```bash
  component: MenuButton
    ...
    accessibility
      ✕ is accessible on 'tiny' (206 ms)
      ✕ is accessible on 'small' (158 ms)
      ✕ is accessible on 'medium' (134 ms)
      ✕ is accessible on 'large' (154 ms)
      ✕ is accessible on 'extraLarge' (144 ms)

   ● With theme changes  › accessibility › is accessible on 'tiny'

    expect(received).toHaveNoViolations(expected)

    Expected the HTML found at $('#menu-button--menu--38') to have no violations:

    <button aria-haspopup="true" aria-controls="menu--38" variant="filled" class="sc-fHYxKZ itpwSe" data-reach-menu-button="" type="button" id="menu-button--menu--38">

    Received:

    "Buttons must have discernible text (button-name)"

    Fix any of the following:
      Element does not have inner text that is visible to screen readers
      aria-label attribute does not exist or is empty
      aria-labelledby attribute does not exist, references elements that do not exist or references elements that are empty
      Element has no title attribute
      Element's default semantics were not overridden with role="none" or role="presentation"

    You can find more information on this issue here:
    https://dequeuniversity.com/rules/axe/4.2/button-name?application=axeAPI

    ────────

    Expected the HTML found at $('div[role="none"]') to have no violations:

    <div role="none" tabindex="-1"><a role="menuitem" tabindex="-1" href="#" data-reach-menu-link="" data-reach-menu-item="" data-valuetext="Action Three" id="option-2--menu--38">Action Three</a></div>

    Received:

    "Elements of role none or presentation should be flagged (presentation-role-conflict)"

    Fix all of the following:
      Element is not focusable.

    You can find more information on this issue here:
    https://dequeuniversity.com/rules/axe/4.2/presentation-role-conflict?application=axeAPI

      184 |       const results = await axe(container);
      185 |
    > 186 |       expect(results).toHaveNoViolations();
          |                       ^
      187 |     })
      188 |
      189 |   });

      at src/MenuButton/MenuButton.test.tsx:186:23
```

repository: repaygithub/cactus

path: /docs/How-tos/Accessibility Testing.md

filename: Accessibility Testing.md

gha_language: TypeScript

repo_name: repaygithub/cactus


text:
---
layout: page
title: 2010-2011 Board
---
### Chair: Jen King

Jen adores making long "to do" lists and convincing people to help her check things off.  Next on her list is persuading you to come to all of our ACM events, and then locating Hermione's time-turner to fit even more activities into a day.

### Vice Chair: Emily Lancaster

Emily is a computer science major.  She studies computer science because she recognizes how the world is dependent on technology and wants to learn more about software and computers so she can exploit them.  Upon graduation, Emily plans to become a super villain and take over the world.  She wants you to be a CS major too so you can join her.

### Secretary/Treasurer: Patrick Wendell

Patrick was a late round draft pick for the Princeton ACM's senior leadership. He is widely recognizes as being the only ACM officer who owns a car.

### Competitions Chair: Jarett Schwartz

Jarett really wants you to join the ACM, the Programming Team, and the COS department. Seriously.

### Webmaster: Dana Hoffman

*[No Bio]*

### Fundraising Chair: Lavanya Jose

*[No Bio]*

branch_name: refs/heads/develop

gha_language: HTML


text:
---
layout: post
title: Multi Armed Bandits and Exploration Strategies
published: true
project: false
---

This blog post is about the Multi Armed Bandit(MAB) problem and about the Exploration-Exploitation dilemma faced in reinforcement learning. MABs find applications in areas such as advertising, drug trials, website optimization, packet routing and resource allocation.

A Multi Armed Bandit consists of $$K$$ arms, $$K\ge2$$ numbered from $$1$$ to $$K$$. Each arm $$a$$ is associated with an unknown probability distribution $$P_a$$ whose mean is $$\mu_a$$. Pulling the $$a$$th arm produces a reward $$r$$ which is sampled from $$P_a$$. There is an agent which has a budget of $$T$$ arm pulls. The agent has to pull these arms in some sequence so as to maximize the accumulated reward after $$T$$ arm pulls. The reward for pulling arm $$a_i$$ at the $$t$$'th step be $$r_{a_i,t}$$ which is sampled from $$P_{a_i}$$. The agent's task is to maximise $$\sum_{t=1}^T r_{a_i,t}$$.

$$Maximize \quad \sum_{t=1}^T r_{a_i,t} \quad \mbox{where } r_{a_i,t} \sim P_{a_i}$$

Since the arm rewards are stochastic, we concentrate on maximising the expected total reward.

$$\mbox{Total Reward}= \sum_{t=1}^T r_{a_i,t}$$

$$\begin{align*}
E[\mbox{Total Reward}] &= E\big[\sum_{t=1}^T r_{a_i,t}\big]\\
&=\sum_{t=1}^T E[r_{a_i,t}]\\
&=\sum_{t=1}^T\mu_{x_t}
\end{align*}$$

Where $$x_t \in \{a_1,a_2,..,a_K\}$$ for $$t=1,2,..T$$ are the sequence of arm pulls.

The arm which has the highest mean reward is called as the optimal arm. Let $$a^*$$ be the optimal arm and $$\mu^*$$ be its mean reward. Another way of maximizing the cumulative reward is by minimizing the cumulative expected Regret.

$$Regret = \sum_{t=1}^Tr_{a^*,t}-r_{a_i,t}$$

$$\begin{align*}
E[Regret]&=E\big[\sum_{t=1}^Tr_{a^*,t}-r_{a_i,t}\big]\\
&=\sum_{t=1}^TE[r_{a^*,t}] - \sum_{t=1}^TE[r_{a_i,t}] \\
&= T\mu^* - \sum_{t=1}^T\mu_{x_t}
\end{align*}$$

So the agent can minimize the cumulative expected regret if it can identify the optimal arm. Initially the agent does not know anything about the distribution of the rewards of each arm. It has to explore by pulling the arms in some sequence and has to learn these distributions. Also the agent has to exploit by pulling the arm which it believes to be the most rewarding, as its goal is to maximize the cumulative reward. But how does the agent decide when to explore and when to exploit? This is the exploration-exploitation dilemma. Explore too much and the agent will accumulate less reward. Exploit too much and the agent may not discover the optimal arm and still accumulate less reward.

Let $$Q_a$$ represent the empirical mean of the rewards received by pulling arm $$a$$. $$Q_a$$ is an unbiased estimate of $$\mu_a$$

$$Q_a = \frac{\mbox{Sum of rewards received from arm }a}{\mbox{Number of times arm }a \mbox{ was pulled} }$$

The kind of MAB we consider here is the Stochastic multi armed bandit. In this case, the support of the probability distribution $$P_a$$ is $$[0,1]$$, so naturally $$\mu_a$$ and the rewards for each arm are bounded between $$[0,1]$$. A sub case of Stochastic bandits are Bernoulli bandits, in which the rewards are either $$0$$ or $$1$$ and the probability distribution $$P_a$$ is a Bernoulli distribution with unknown success probability $$\mu_a$$.

The algorithm for Bernoulli MAB is as follows:
![enter image description here](https://raw.githubusercontent.com/sudeepraja/sudeepraja.github.io/master/images/bandit_alg.PNG)

Here $$Q[a]$$ is the empirical average reward of pulling arm $$a$$, $$N[a]$$ is the number of times arm $$a$$ was pulled, $$S[a]$$ is the number of times a reward of $$1$$ was received when arm $$a$$ was pulled and $$F[a]$$ is the number of times a reward of $$0$$ was received when arm $$a$$ was pulled.

Now we have to decide how to pick an arm so as to balance exploration and exploitation so that the cumulative reward is maximized.

### Random Selection
Pick the arm completely at random. This is not an effective strategy as it completely disregards the history of the arm pulls. We only consider pulling random arms to form a baseline to compare it with other strategies. The expected cumulative regret for random arm selection would be:

$$E[Regret] = T\mu^* - \sum_{t=1}^TE[r_{a_i,t}] = T(\mu^* - \bar{\mu})$$

Here $$\bar{\mu}$$ is the mean of $$\mu_1,\mu_2,..,\mu_K$$. The regret is linear in the number of arm pulls.

### Greedy Selection
Pick the arm which has the highest empirical average reward breaking ties randomly.

$$A = argmax_a (Q[a])$$

Note that initially, all $$Q[a]$$ are $$0$$, so arms are selected randomly until one of the arms give a non zero reward. Form then on only that arm will be chosen. This strategy does no exploration at all and so it is very unlikely that the optimal arm is selected. The expected cumulative regret for greedy arm selection would be:

$$ E[Regret] = T\mu^* - \sum_{t=1}^TE[r_{a_i,t}] = T(\mu^* - \mu_{a_{i'}}) $$

Here $$a_{i'}$$ is the first arm which produces a non zero reward. Since any of the $$K$$ arms could be the first to give a non zero reward, we take the expectation over them.  This gives the same linear regret as random selection.

### $$\epsilon$$-Greedy Selection
With some probability $$\epsilon$$, $$0<\epsilon<1$$, pick a random arm. Otherwise with probability $$1-\epsilon$$ pick the greedy arm. So the probability of picking the greedy arm would be $$1-\epsilon +\frac{\epsilon}{K}$$ and any other arm is picked with probability $$\frac{\epsilon}{K}$$

$$
P(a_i)=\begin{cases}
      1-\epsilon +\frac{\epsilon}{K} & \mbox{, if } a_i = argmax_a (Q[a])\\
      \frac{\epsilon}{K} &\mbox{, otherwise}
 \end{cases}
$$

Consider the case where $$\epsilon$$ is constant. The expected cumulative regret would be:

$$\begin{align*}
E[Regret] &= \sum_{t=1}^TE[r_{a^*,t}] - \sum_{t=1}^TE[r_{a_i,t}] \\
&\ge\sum_{t=1}^T \mu^*-[(1-\epsilon)\mu^* +\epsilon\bar{\mu}]\\
&= T\epsilon(\mu^* - \bar{\mu})
\end{align*}$$

The regret is again linear in the number of arm pulls.

We could change $$\epsilon$$ through time so that the agent acts greedily in the limit as $$T\to \infty $$. Initially the agent should behave randomly to encourage exploration and, as time progresses, it should act more greedily. It is possible to achieve logarithmic regret using a decaying $$\epsilon$$-greedy strategy, but decay schedule requires the knowledge of $$P_i$$s.

### Boltzmann Exploration

A problem with the $$\epsilon$$-greedy strategy is that it treats all of the arms, apart from the best arm, equivalently. We could select arm $$a$$ with a probability depending on the value of $$Q[a]$$. This is known as a soft-max action selection. A common method is to use a Boltzmann distribution, where the probability of selecting arm $$a$$ is proportional to $$\exp(Q[a]/\tau)$$. That is, the agent selects arm $$a$$ with probability

$$P(a) = \frac{\exp(Q[a]/\tau)}{\sum_a \exp(Q[a]/\tau)}$$

where $$\tau>0$$ is the temperature specifying how randomly arms should be chosen. When $$\tau$$ is high, the arms are chosen in almost equal amounts. As the temperature is reduced, the highest-valued arms are more likely to be chosen and, in the limit as $$\tau\to0$$, the best arm is always chosen. The expected cumulative regret for Boltzmann Exploration is also linear.

### Upper-Confidence-Bound Arm Selection
This strategy is based on the Optimism in the Face of Uncertainty principle. We know that $$Q[a]$$ is an unbiased estimate of $$\mu_a$$. After some $$N[a]$$ pulls of arm $$a$$ we can be fairly certain about how close $$Q[a]$$ is to $$\mu_a$$. Using Hoeffding's Inequality which we used in my previous post we can derive the following bound:

$$\Pr(\mid Q[a] - \mu_a \mid \ge \epsilon)\le 2 \exp(-2N[a]\epsilon^2)$$

Using a one sided version of this inequality we get:

$$\Pr(\mu_a \ge Q[a] + \epsilon) \le \exp(-2N[a]\epsilon^2)$$

So for arm $$a$$, whose average reward is $$Q[a]$$ after it has been pulled $$N[a]$$ times, $$\mu_a$$ exceeds the upper confidence bound(UCB) with probability $$p = \exp(-2N[a]\epsilon^2)$$

We want the probability that $$\mu_a$$ exceeds UCB to decrease with $$t$$, the number of arm pulls so far. We could use $$p=t^{-4}$$, which ensures that we select the optimal action in the limit as $$t \to \infty$$.

$$\epsilon = \sqrt{\frac{-\log p}{2N[a]}} = \sqrt{\frac{2\log t}{N[a]}}$$

The UCB strategy is as follows:

$$A = argmax_a\bigg[ Q[a] +\sqrt{\frac{2\log t}{N[a]}} \bigg]$$

The UCB strategy could also be seen as an Incentive Based exploration strategy. Here the extra incentive apart from the expected reward for pulling arm $$a$$ is $$\sqrt{\frac{2\log t}{N[a]}}$$ which could be interpreted as the reward for gaining confidence about arm $$a$$'s reward. This incentive may cause the agent to pull non-greedy arms when it thinks it can gain more information about an arm's reward.

The expected cumulative regret for UCB strategy is logarithmic. The proof for this is a little involved and can be found in this paper: [Finite-time analysis of the multiarmed bandit problem.](https://homes.di.unimi.it/~cesabian/Pubblicazioni/ml-02.pdf)

### Thompson Sampling Strategy

Thompson sampling works by maintaining a prior on the the mean rewards of the arms $$\mu_i$$. It samples values for each arm from its prior and picks the arm with the highest value. When an arm $$a$$ is pulled and a Bernoulli reward $$r$$ is observed, it modifies the prior based on the reward. This procedure is repeated for the next arm pull. Beta distribution is a convenient choice of priors for Bernoulli rewards. The probability density function of a Beta distribution with parameters $$\alpha$$ and $$\beta$$ is:

$$\frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)}x^{\alpha-1}(1-x)^{\beta-1}$$

The Thompson sampling algorithm initially assumes arm $$a$$ to have a prior $$Beta(1,1)$$ on $$\mu_a$$, which is the uniform distribution on $$(0,1)$$. Beta distribution is useful for Bernoulli rewards because, if the prior is a $$Beta(\alpha,\beta)$$ distribution, then after observing a Bernoulli trial, the posterior distribution is $$Beta(\alpha+1,\beta)$$ if the trial was a success or $$Beta(\alpha,\beta+1)$$, if it was a failure.

At time $$t$$, having observed $$S[a]$$ successes and $$F[a]$$ failures out of $$N[a]$$ pulls of arm $$a$$, the algorithm updates the distribution on $$a$$ as $$Beta(S[a] + 1, F[a] + 1)$$. The algorithm then samples from these posterior distributions of the $$\mu_a$$'s, and plays an arm according to the probability of its mean being the largest. The algorithm is as follows:

$$\mbox{For } a \mbox{ in }1..K:\quad \theta[a] \sim Beta(S[a] + 1, F[a] +1)$$

$$A=argmax_a(\theta[a])$$

The expected cumulative regret for Thompson sampling strategy is logarithmic. Though this strategy has been around since the 1930s, its regret bound has been proved only recently in this paper: [Analysis of Thompson Sampling for the Multi-armed Bandit Problem](https://jmlr.org/proceedings/papers/v23/agrawal12/agrawal12.pdf)

## Empirical Evaluation

We test the above strategies using several randomly generated 10 arm Bernoulli Bandit instances with the mean arm rewards generated by sampling the Uniform distribution between $$[0,1]$$. We evaluate these strategies by plotting the average Percentage of optimal arm pulls vs Number of arm pulls. The average is taken over the randomly generated instances.

![enter image description here](https://raw.githubusercontent.com/sudeepraja/sudeepraja.github.io/master/images/Bandits.png)

Since we have 10-arms, the Random strategy pulls the optimal arm in only 10% of pulls. Greedy strategy locks onto the optimal arm in only 20% of pulls. The $$\epsilon$$-Greedy strategy quickly finds the optimal arm but only pulls it 60% of the time. UCB is slow to find the optimal arm but then eventually overtakes the  $$\epsilon$$-Greedy strategy. Thompson sampling is by far the best strategy, pulling the optimal arm almost 100% of the times.  The code for generating this graph and for playing around with Multi armed bandits can be found in this [gist](https://gist.github.com/sudeepraja/6f57e29f2854828793fb75ba5a09d152).

References:

 1. Chapter 2 in [Reinforcement Learning: An Introduction](https://www.dropbox.com/s/b3psxv2r0ccmf80/book2015oct.pdf?dl=0)
 2. [Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems](https://arxiv.org/abs/1204.5721)
 3. [Optimism in the Face of Uncertainty: the UCB1 Algorithm](https://jeremykun.com/2013/10/28/optimism-in-the-face-of-uncertainty-the-ucb1-algorithm/)


text:
##ReadMe for Readme Auto Generator

  [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)

  # Table of Contents:

  [Description](#description)
  [License](#license)
  [Installation](#installation)
  [Usage](#usage)
  [License](#license)
  [Contribution](#contribution)
  [Testing](#testing)
  [Questions](#questions)

  ## Description

  This app makes for an easy way to auto-generate readmes for the apps you create

  ## License

  Apache 2.0

  ## Installation

  You need to install npm inquirer for this app to work

  ## Usage

  Simply run index.js from your terminal and answer the prompts.

  ## Contribution

  Feel free to check out the repo and make whatever changes you'd like.

  ## Testing

  Run the app in your terminal

  ## Questions

  For questions you may email me at<EMAIL_ADDRESS>
  GitHub Profile Link: https://github.com/DanGillette25


text:
# Scribe
A Chrome extension that converts raw text URLs in Gmail to embedded images or to friendly hyperlinks with page title text, like Quora's text editor. I'm doing this because it's fun, and I really want to improve system designs.

*This project is now effectively dead*

# Back-End

- A server that contains a Python crawler. On receipt of a URL, the crawler fetches the associated page and parses it for important human-readable information (a title tag or similar). If the URL points to an image file, it simply downloads the image.
- Current design of system employs use of a self-designed HTTP server with support for only a GET request for handling communications. An HTTPS wrapper provided by Mashaope's convenient API functionality allows us to access this point from an HTTPS endpoint.

# Front-End

A Chrome extension that encodes an event handler on pressing the F2 key in the document window (rather than a compose window). An XMLHttpRequest is carried out synchronously, and the result replaces the URLs.

# Current Status

**This Chrome extension is able to reproduce all the desired functionality.** Pressing the F2 key in the document window once converts all raw URLs to human-readable links and actual images. All that's left is to iron out a few kinks that would subtract from user experience (see Issues).

The back-end server is complete, with an HTTPS middleman ensuring security. Current example of API endpoint definition can be seen in `server.py`. One can view examples within a favourite browser - the server has been successfully daemonised, and runs continuously on port 8080 of my domain.

The front-end component is nearing completion. It uses [Autolinker.js](https://github.com/gregjacobs/Autolinker.js) to grab all raw URLS and implements a simple XMLHttpRequest to communicate with the server. Currently this XMLHttpPRequest works synchronously (i.e. the document will wait until the request is handled), something I plan to fix soon (this does mimic Quora's behaviour, but that is no excuse for embracing deprecated functionality). It replaces the text in a compose or reply window with nice hyperlinks and images.

# Current Issues

* The F2 can only be pressed in the document window - it will not work if focus is on the composition or reply window. One should ideally focus on the compose or reply window that the user is on, and allow the application to affect that window.

Efforts are being made to resolve this.

repository: AkshatM/Scribe

repo_name: AkshatM/Scribe


text:
# Reignite

Reignite is an opinionated rapid prototyping environment for ReactJS.

# Contribute

- `yarn debug` will start the node server
- `yarn ui` will start the UI build task in watch mode

repository: Raathigesh/reignite

repo_name: Raathigesh/reignite


text:
# Connect to NuoDB Database Externally

## Introduction

NuoDB supports SQL clients via a NuoDB-provided client driver such as JDBC.
NuoDB provides a configurable internal load balancer that can direct clients to Transaction Engines (TEs) according to user-specified load balancing rules.
This enables the administrator to direct different client workloads to different pools of TEs in order to maximize cache locality and execute certain workloads on a TE with the best configuration for that workload.

When a SQL client connects to a database, the driver first establishes a connection with the NuoDB Admin, which will perform load balancing and redirect the client to an appropriate TE in that database.
Load balancing can be configured using the powerful `LBQuery` language to define the database connection strategy for routing SQL clients to the most appropriate TEs.

For more information on NuoDB client connections, see [Client Development](https://doc.nuodb.com/nuodb/latest/client-development/).

For more information on the LBQuery language syntax, see [Load Balancer Policies](https://doc.nuodb.com/nuodb/latest/client-development/load-balancer-policies/).

A _direct_ connection to TEs is also supported by NuoDB, however, in this case, the load balancing must be performed by the client application.

SQL clients and applications running in the same Kubernetes cluster as the NuoDB domain can connect to the database with the default NuoDB Helm Charts configuration using the NuoDB Admin `ClusterIP` service or directly using the TE `ClusterIP` service. This document focuses on external client applications running outside of the Kubernetes cluster where the NuoDB database is hosted. Allowing external access to the NuoDB database is not enabled by default and requires additional configuration.

## Transaction Engine Groups

A TE group consists of TEs with the same configuration which is part of the same database most often used to serve specific SQL workload.
NuoDB Helm Charts 3.4.0+ supports the deployment of one or more TE groups per database.

Multiple Helm releases of the `database` chart for the same NuoDB database can be installed in the same Kubernetes namespace where only one of them is _primary_.
One or more _secondary_ Helm releases are used to deploy additional TE groups for the same database with different configuration options.
This allows flexible configuration of each TE group including but not limited to the number of TEs in a group, their resource requirements, process labels, and scheduling rules.
SQL clients can be configured to target each TE group separately using NuoDB Admin load balancer rules.
Specifying the type of the Helm `database` release is controlled by the  `database.primaryRelease` option (`true` by default).

## External Access for TE Groups

External access for the NuoDB database is _not_ enabled by default.
To enable external access to the NuoDB domain and database, set the `admin.externalAccess.enabled=true` and `database.te.externalAccess.enabled=true` options.
Configuring external database access via the NuoDB Admin with TE groups is supported with NuoDB Helm Charts 3.4.0+ and NuoDB 4.2.4+.
In earlier versions, it is only possible to connect to the provisioned TEs directly and bypass the NuoDB Admin.

A Kubernetes service of type `LoadBalancer` or `NodePort` is created per TE group.
This should allow the external SQL clients to connect to the TEs backing the service by uniquely targeting each TE group.
Most of the cloud vendors provide Kubernetes Load Balancer controllers that support different service annotations used to control the properties and configuration of the provisioned cloud load balancer.
The Kubernetes cluster should be properly configured so that the external network (Layer4) cloud load balancer is provisioned automatically.
For more information on additional cluster configuration, see [Cloud Provider Specifics](#cloud-provider-specifics).

> **NOTE**: When external access is enabled, the NuoDB Helm Charts will create Internet-facing load balancers by default.  Be sure to understand the difference between _internet facing_ load balancers (allowing connectivity external to the cloud and Kubernetes cluster) and  _internal_ load balancers (allowing connectivity external to the Kubernetes cluster but within the cloud virtual network).

If the SQL clients are located outside of the Kubernetes cluster where NuoDB is deployed but in the same cloud provider or virtual network, then _internal_ load balancer can be used.
This is configured by setting the `admin.externalAccess.internalIP=true` and `database.te.externalAccess.internalIP=true` or further customized by explicitly setting custom annotations for the Kubernetes services using the `admin.externalAccess.annotations` and `database.te.externalAccess.annotations` options.
The user-provided custom annotations will overwrite the default annotations for the services.

> **IMPORTANT**: The customer is responsible for correctly configuring security rules and restricting access to the cloud load balancers.

Services of type `NodePort` can be created as well by setting the `admin.externalAccess.type=NodePort` and `database.te.externalAccess.type=NodePort` options.
For this type of deployment, a Layer4 load balancer must be manually provisioned. It should also be configured to load balance traffic across all worker nodes in the Kubernetes cluster.
The different TE groups will be reachable on different node ports.

The external address and port for TEs are configured using the `external-address` and `external-port` process labels.
If supplied, they will be advertised by NuoDB Admin to the SQL clients during the second stage of the client connection protocol.
For more information, see [Use External Address](https://doc.nuodb.com/nuodb/latest/client-development/load-balancer-policies/#_use_external_address).

Obtaining and configuring the address of the L4 load balancers can be tedious and error-prone as they are provisioned asynchronously by the Kubernetes controllers.
NuoDB can inspect the Kubernetes services and configure the TE database processes with the `external-address` and `external-port` process labels automatically if the `--enable-external-access` argument is provided.
This simplifies the deployment and ensures the correct configuration of the TE database processes.

- For services of type `LoadBalancer`, NuoDB will configure the `external-address` process label with the service ingress IP or hostname as a value.
- For services of type `NodePort` are provisioned, the customer must configure the  `external-address` and NuoDB will configure the `external-port` process label with the service node port as a value.
If the `--enable-external-access` argument is supplied but the `external-address` process label is not defined, then external access won’t be enabled and a warning message will be logged.

> **NOTE**: If either the hostname or the IP address value(s) of the provisioned cloud load balancer change, the TE database process must be restarted for the new value(s) to take effect.

## Examples

To demonstrate the external access using TE groups, let's consider a working example with the following requirements:

- A NuoDB database is deployed in a single Kubernetes cluster.
- Transactional SQL workload should be processed by TEs in _group 1_.
- Reporting SQL workload should be processed by TEs in _group 2_.
- A set of the applications are installed in the same Kubernetes cluster as NuoDB.
- Another set of the applications are installed in a different Kubernetes cluster, on bare metal, or in a different cloud.

The resource requirements for the different TE groups may be different as there is a direct dependency on the type of SQL workload that TEs will serve.
There will be 2 _smaller_ TEs dedicated for the _transactional_ and 1 _bigger_ TE dedicated for the _reporting_ workload deployed in _nuodb_ namespace.

To satisfy the requirement of having several workloads targeting a different set of TEs, the `LBQuery` connection property will be used.
Alternatively, load balancer policies can be configured by an administrator and SQL clients can reference them using the `LBPolicy` connection property.
For more information, see [Registering Named Load Balancer Policies](https://doc.nuodb.com/nuodb/latest/client-development/load-balancer-policies/#_registering_named_load_balancer_policies_lbpolicy).

For simplicity, the `tx-type` database process label will be used to filter which workload is served by a set of TEs using the `LBQuery` connection property.
The label value is either `transactional` or `reporting`.

NuoDB supports multi-tenant, multi-cluster, and multi-cloud database deployments using TE groups, however, for simplicity a single-cluster single-tenant deployment will be demonstrated here.
The below diagram illustrates the deployed resources and SQL clients along with the `LBQuery` syntax.

![External Access with TE Groups](../images/database-groups.png)

### Deployment

The steps below will deploy a NuoDB database with 2 TE groups in Google Kubernetes Engine (GKE).
If you are deploying in a different environment, make sure that the correct cloud provider is set in the `cloud.provider` option.

> **NOTE**: Use the `nuodb.image.tag` option to specify the NuoDB product version.
NuoDB 4.2.4+ docker image should be used.

Install the [admin](../stable/admin/README.md) chart and enable external access.
Service of type `LoadBalancer` will be provisioned by default.

```shell
helm install admin nuodb/admin \
    --namespace nuodb \
    --cloud.provider=google \
    --set admin.externalAccess.enabled=true
```

Install the [database](../stable/database/README.md) chart for the primary Helm release which deploys TE _group 1_.
Configure the `tx-type=transactional` label for the TEs in this group.

```shell
helm install database-group1 nuodb/database \
    --namespace nuodb \
    --set cloud.provider=google \
    --set database.name=demo \
    --set database.te.externalAccess.enabled=true \
    --set database.te.otherOptions.enable-external-access=true \
    --set database.te.replicas=2 \
    --set database.te.resources.limits.cpu=4 \
    --set database.te.resources.limits.memory=8Gi \
    --set database.te.resources.requests.cpu=4 \
    --set database.te.resources.requests.memory=8Gi \
    --set database.te.labels.tx-type=transactional
```

Install the [database](../stable/database/README.md) chart for the secondary Helm release which deploys TE _group 2_.
Configure the `tx-type=reporting` label for the TEs in this group.

```shell
helm install database-group2 nuodb/database \
    --namespace nuodb \
    --set cloud.provider=google \
    --set database.name=demo \
    --set database.primaryRelease=false \
    --set database.te.externalAccess.enabled=true \
    --set database.te.otherOptions.enable-external-access=true \
    --set database.te.replicas=1 \
    --set database.te.resources.limits.cpu=4 \
    --set database.te.resources.limits.memory=16Gi \
    --set database.te.resources.requests.cpu=4 \
    --set database.te.resources.requests.memory=16Gi \
    --set database.te.labels.tx-type=reporting
```

Wait for the NuoDB database to become ready:

```shell
kubectl exec -ti admin-nuodb-cluster0-0 -n nuodb -- nuocmd check database \
    --db-name demo \
    --check-running \
    --num-processes 4 \
    --wait-forever
```

### Verification

Obtain the external address for the NuoDB Admin service:

```shell
kubectl config set-context --current --namespace=nuodb
DOMAIN_ADDRESS=$(kubectl get services nuodb-balancer -o jsonpath='{.status.loadBalancer.ingress[].ip}')
```

Check database processes node IDs:

```shell
kubectl exec -ti admin-nuodb-cluster0-0 -- nuocmd show database \
    --db-name demo \
    --process-format '{type} {hostname} {node_id}'
```

Use `nuosql`, which can be found in the [NuoDB Client-only Package](https://github.com/nuodb/nuodb-client), to connect to the NuoDB database from the local machine.
Repeat the command several times to ensure that each time the expected node ID is printed.

```shell
echo 'select GETNODEID() from dual;' |  nuosql demo@${DOMAIN_ADDRESS} \
    --user dba --password secret \
    --connection-property 'LBQuery=round_robin(first(label(tx-type transactional) any))'
```

Repeat the steps for the _reporting_ workload.

```shell
echo 'select GETNODEID() from dual;' |  nuosql demo@${DOMAIN_ADDRESS} \
    --user dba --password secret \
    --connection-property 'LBQuery=round_robin(first(label(tx-type reporting) any))'
```

Verify that internal SQL clients can still connect to the database by setting `PreferInternalAddress=true` connection property.

```shell
kubectl exec -ti admin-nuodb-cluster0-0 -- bash -c \
    "echo 'select GETNODEID() from dual;' |  nuosql demo@nuodb-clusterip \
        --user dba --password secret \
        --connection-property 'LBQuery=round_robin(first(label(tx-type transactional) any))' \
        --connection-property 'PreferInternalAddress=true'"
```

## Cloud Provider Specifics

### Native CNI

All of the managed Kubernetes offerings from cloud providers supported by the NuoDB Helm Charts (EKS, AKS, GKE) have the ability to deploy Kubernetes with a native Container Network Interface (CNI) plugin.
This causes Kubernetes pods to be assigned IP addresses from the underlying virtual network CIDR, making them addressable by SQL clients running in the same VPC (or peered VPCs / VPN connections with the correct routing configured).
As such, the TE IP address returned by the second stage of the client connection protocol is already addressable by the client, and it is enough to make only the Admin layer available external to the Kubernetes cluster using the `admin.externalAccess.*` values, to receive client connections.

> **NOTE**: The node security configuration may also need to explicitly allow the NuoDB TE port (48006).

### GCP

No additional configuration is needed to enable NuoDB database external access in GKE.
For more information, check [Configuring TCP/UDP load balancing](https://cloud.google.com/kubernetes-engine/docs/how-to/service-parameters).

### AWS

Kubernetes load balancer controller will automatically provision AWS Network Load Balancer (NLB) for Kubernetes services of type `LoadBalancer`, as described in the [Network load balancing on Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/network-load-balancing.html) guide.
Amazon Elastic Kubernetes Service (EKS) can be configured with several different load balancer controllers:

- [AWS Load Balancer Controller](https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html)
- [legacy AWS cloud provider load balancer controller](https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer)

The annotation `service.beta.kubernetes.io/aws-load-balancer-type` is used to determine which controller reconciles the service.
For more information, check [Legacy Cloud Provider](https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.4/guide/service/annotations/#legacy-cloud-provider).

The default configuration for external access varies over different versions of NuoDB Helm Charts.
The below table describes the AWS load balancer controller that will be used by default.

|                     | Helm Charts 3.3.x | Helm Charts 3.4.x | Helm Charts 3.5.x and later |
|---------------------|-------------------|-------------------|-----------------------------|
| `internalIP`: true  | Legacy LB         | Legacy LB         | Legacy LB                   |
| `internalIP`: false | Legacy LB         | AWS LB            | Legacy LB                   |

The default Kubernetes service annotations can be modified using the `admin.externalAccess.annotations` and `database.te.externalAccess.annotations` values.
For more information on how to customize the provisioned NLB, check [Network Load Balancer](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/service/nlb/).

### Azure

No additional configuration is needed to enable NuoDB database external access in Azure Kubernetes Service (AKS).
For more information, see [Use a public Standard Load Balancer](https://docs.microsoft.com/en-us/azure/aks/load-balancer-standard).

## Troubleshooting

### Unable to connect

There may be different reasons for client connectivity problems such as:

- not _Ready_ NuoDB Admin Pods
- not _Ready_ TE pods
- incorrect external access configuration
- incorrect NuoDB Load Balancer configuration
- incorrect NLB configuration
- network connectivity problems including lack of routing, firewall configuration, and many more

> **ACTION**: You can rule out any of the points above one by one.
Start by checking the overall Pod status for the NuoDB domain and database.
Some of the common troubleshooting steps are listed below, however, there might be additional verifications specific to your deployment.

1. Verify that all AP, TE, and SM pods are reported _Ready_ using `kubectl get pods`.
2. Check the NuoDB domain and database using `nuocmd show domain`.
3. Verify that the database is available to SQL clients within the Kubernetes cluster by using the same connection properties that the application uses.
To verify this, use the `nuosql` tool inside an AP Pod.
4. Make sure that `external-address` and/or `external-port` process labels are configured correctly on the TE database processes using `nuocmd --show-json-fields hostname,labels get processes`.
If you are using the `--enable-external-access` argument, verify that the configured values are the same as the `EXTERNAL-IP` shown for the Kubernetes service in `kubectl get services` output.
If the value is not the same, restart the TE database process and verify again.
5. Verify that the `LBQuery` or `LBPolicy` syntax is correct.
The expected TE process should be shown when using `nuocmd get sql-connection`.
If using `LBPolicy`, verify that the expected policies are configured in the NuoDB Admin using `nuocmd get load-balancers` and `nuocmd get load-balancer-config`.
6. Check Kubernetes events for issues deploying cloud load balancers and external addresses.
For more information on this, see the [TE does not start](#te-does-not-start) section.
7. Verify that the cloud load balancer is provisioned and forwards traffic to the correct Kubernetes cluster.
Check that its configuration is correct and modify the Kubernetes service annotations if needed.
8. Verify that the configured security rules allow external access.

### TE does not start

TEs started with the `--enable-external-access` argument will wait for the `LoadBalancer` service IP address or hostname to be available before they start.
In a case of a problem during NLB provisioning, the IP address will never be populated and the _engine_ container will fail.
The following errors can be seen in the TE container logs:

```text
2021-12-15T07:43:05.015+0000 INFO  [admin-nuodb-cluster0-0:te-database-nuodb-cluster0-demo-55d664c8d7-bn57p] CustomAdminCommands Found service name=database-nuodb-cluster0-demo-balancer, type=LoadBalancer, selector={u'app': u'database-nuodb-cluster0-demo', u'component': u'te'}
2021-12-15T07:43:05.015+0000 INFO  [admin-nuodb-cluster0-0:te-database-nuodb-cluster0-demo-55d664c8d7-bn57p] CustomAdminCommands Waiting for load balancer service database-nuodb-cluster0-demo-balancer ingress address...
'start te' failed: Timeout after 120.0 sec waiting for ingress hostname in service database-nuodb-cluster0-demo-balancer
```

> **ACTION**: Verify that the `EXTERNAL-IP` for the service is available using `kubectl get services`.
Check the events for the service for this TE group using `kubectl describe service database-nuodb-cluster0-demo-balancer`.
Refer to your cloud provider's documentation on how to troubleshoot the load balancer controller.

repository: nuodb/nuodb-helm-charts

gha_language: Go

repo_name: nuodb/nuodb-helm-charts


text:
Actions
=======

The following actions are provided by the Api Plugin:

## Api.Index

This will give you a (full) collection of items.

## Api.View

This will return a single item.

## Api.Add

This will add an item.

> Not implemented properly yet...

## Api.Edit

This will modify/edit an item.

> Not implemented properly yet...

## Api.Delete

This will delete the item.

> Not implemented properly yet...

## Further Reading

Read more about actions [here](http://crud.readthedocs.io/en/latest/actions.html)

repository: jabit/api

repo_name: jabit/api


text:
---
layout: post
title: How to configure App Engine for your Go application
categories: []
tags: ["Google", "AppEngine"]
fullview: true
twittertitle: How to configure App Engine for your Go application
twitterdescription: Quick How-to with the basic App Engine configuration for an applicaiton using REST API and static files
twitterimage: http://www.appscale.com/wp-content/uploads/2013/10/google-app-engine-logo.png
---

A new blog post since a long long… very long time. I promise I will write articles more often than in the past. For this rebirth, I will start with the first article of a serie about Google technologies.

I have started last week a project in which I will only use Google technologies. It will help myself to learn new stuff. I won't explain what this application will be used for, I will let you discover with these blog posts.

What I have in mind for the moment, is an application using :

- The **Go** language for the server-side hosted on **App Engine**
- **Angular2** and **Material Design Lite** for the front-end

If you have other ideas for the technical stack, do not hesitate to comment one of my posts, or create issues on the Github repository. I will be very glad to have a look to your ideas, and write some articles about them.

In a daily bases, I am working with front-end technologies (HTML5, AngularJS, jQuery, ...). All articles I will write about server-side components will be maybe too simple for some of you. I am sorry for that. But the main goal of this serie of posts is the discover of Google products.

Let start with the beginning… with the quick configuration of my **App Engine** instance, defined in a app.yaml file.

The application will be composed of two parts : an REST API (implemented in **Go**) and an interface (using **Angular2**). The UI part should be available at “/” and the API at “/api/”. In the app.yaml configuration file of **App Engine**, I have to configure differents handlers, in a `handler` block,  based on an URL pattern.

Here is the basic version of this file, defining the technology of the application hosted on **App Engine**.

{% highlight yaml %}
application: xxxxxxx
version: 1
runtime: go
api_version: go1

handlers:
#handlers
{% endhighlight %}

Here is the description of these properties :

- `application` : the ID of the application you have created on App Engine
- `version` : the version of your application
- `runtime` : the App Engine runtime needed for your application
- `app_version` : the version of the runtime

The first handler we will define is for the REST API. We have to define two properties :

- `url` : the pattern of the URL. In this regular expression, you can define capture group (more informations about these groups later).
- `script` : the name of the script App Engine has to execute for this URL pattern. For **Go** application, you have to use the magic string `_go_app`.

{% highlight yaml %}
handlers:
- url: /api/.*
  script: _go_app
{% endhighlight %}

When your API is implemented, if you launch your application with the **goapp** utility (maybe a specific article about this command line tool ?), your API should be available !

So we have our API, let’s configure our static files (CSS, JS and HTML for the moment). We will define the same kind of configuration described previously.

For each URL pattern (based on the file extension) , we will define at least three properties :

- `mime_types` : the mime-type of the corresponding static file
- `static_files` : the path of the files matched by the URL pattern. You can access to capture groups defined in the URL pattern ("\1" refer to the first group, "\2" the second one...).
- `upload` : a regexp corresponding to the path of all files served by this handler

Other configuration properties are available, I let you read [the corresponding documentation](https://cloud.google.com/appengine/docs/python/config/appconfig#Python_app_yaml_Static_file_pattern_handlers).

{% highlight yaml %}
handlers:
- url: /(.*\.css)
  mime_type: text/css
  static_files: static/\1
  upload: static/(.*\.css)

- url: /(.*\.js)
  mime_type: text/javascript
  static_files: static/\1
  upload: static/(.*\.js)

- url: /(.*\.html)
  mime_type: text/html
  static_files: static/\1
  upload: static/(.*\.html)

- url: /
  static_files: static/index.html
  upload: static/index.html
{% endhighlight %}

Of course, this small configuration is not enough. You should add for every static files your App Engine instance has to serve, the corresponding configuration block (XML, font, …). You can have a look to the file I have created for the application [here](https://github.com/Gillespie59/google-project/blob/master/app.yaml).

And this first post is now drawing to a close. You should have now an up-and-running application you can deploy on App Engine.

See you soon...


text:
---
id: 464
title: Building my homelab – part 4
base: Blog
base_url: /blog
author: Tzeny
layout: blog_post
guid: http://<IP_ADDRESS>:8000/?p=464
thumbnail: 2019/01/IMG_20181231_213214-360x210.jpg
categories:
  - engineering
tags:
  - docker
  - server
  - xfce
---
I changed my networking setup and switched from Proxmox to Ubuntu, moving most of my services to containers.

**Part 3:** [**Building my homelab – part 3**](https://tzeny.com/2017/11/30/building-my-homelab-part-3/)

{% include figure_caption.html url="/assets/img/posts/2019/01/IMG_20181231_213214-768x1024.jpg" description="" %}

## Networking

Parts used:

  * [N3150M-E Asus motherboard](https://www.asus.com/Motherboards/N3150ME/) with an integrated [Intel Celeron N3150](https://ark.intel.com/products/87258/Intel-Celeron-Processor-N3150-2M-Cache-up-to-2-08-GHz-)
  * 4GB of DDR3 RAM
  * Old computer case and power supply
  * 2 x Gigabit PCI 1x network cards
  * 5 port Gigabit ethernet switch

I wanted to try using a [pfSense](https://www.pfsense.org/) machine as my router / firewall, and use my existing wireless router as an access point. So I bought a cheap Asus motherboard with an integrated Intel CPU supporting AES NI (a requirement for newer versions of pfSense) and built a system around it.

The CPU is a low power quad core, which I figured would be more than enough for my needs.

After installing and configuring pfSense I had some issues with it after a couple of weeks: I could no longer access the web interface and I had rare, random internet failures.

Although it was interesting playing around with it, I reverted back to my wireless router, as it offers all the functionality that I need. In the future I plan on trying pfSense again, but with a different hardware configuration.

{% include figure_caption.html url="/assets/img/posts/2018/12/IMG_20181231_213356-1024x768.jpg" description="Here you can see Farcaster, the pfSense machine and a black gigabit ethernet switch" %}

The other addition to the Homelab is a 5 port ethernet gigabit switch. It connects all the computers (only Tesla and my laptop) to the network, and is really useful when transferring files between my laptop and Tesla.

Although I have a 300 mb/s symmetric internet connection, with my current router (a [TL-WR740N](https://www.tp-link.com/au/products/details/TL-WR740N.html)) all I can get is 100 mb/s. That is part of the next upgrade.

## Tesla

Parts used:

  * Old 120GB SSD

Software used:

  * [Ubuntu 18.04 LTS](https://www.ubuntu.com/download/desktop)
  * [Docker](https://www.docker.com/)
      * [Nginx Reverse Proxy](https://github.com/jwilder/nginx-proxy)
      * [Let’s Encrypt Nginx Proxy Companion](https://github.com/JrCs/docker-letsencrypt-nginx-proxy-companion)
      * [Media Wiki](https://hub.docker.com/_/mediawiki/)
      * [MariaDB](https://hub.docker.com/_/mariadb/)
      * [Samba Server](https://hub.docker.com/r/sixeyed/samba/)
      * [OpenVPN](https://github.com/kylemanna/docker-openvpn)
  * [obfsproxy](https://blog.torproject.org/obfsproxy-next-step-censorship-arms-race)
  * [TightVNC](https://www.tightvnc.com/)

I added an old 120 Gb SSD to Tesla, which now acts as the OS drive, speeding everything up.

I installed Ubuntu 18.04 Desktop edition as I had planned to use Gnome together with [VNC Viewer](https://www.realvnc.com/en/connect/download/viewer/), but when I used it like that the maximum resolution was that of the attached display; I ended up using [TightVNC](https://www.tightvnc.com/) and [Xfce](https://www.xfce.org/) instead, as shown in [Digital Ocean’s tutorial](https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-vnc-on-ubuntu-18-04).

{% include figure_caption.html url="/assets/img/posts/2018/12/homelab.png" description="Sketch of my current network architecture" %}

### MediaWiki

I started looking into ML some time ago, and I decided to use a wiki in order to take notes. You can access it here if you want: [https://tzeny.ddns.net/](https://tzeny.ddns.net/index.php/Main_Page)

MediaWiki is “a free and open source software wiki package written in PHP, originally for use on Wikipedia. It is now also used by several other projects of the non-profit Wikimedia Foundation and by many other wikis, including this website, the home of MediaWiki.
Use the links below to explore the site contents. You’ll find some content translated into other languages, but the primary documentation language is English.”

I decided to use it since it is simple, yet powerful and also open source.

The wiki itself consists of 2 parts: the PHP server, and the SQL database. They are hosted in separate containers, and linked together using a Docker network.

The next step is adding SSL support. An easy way to do this is to follow
[this guide](https://cloud.google.com/community/tutorials/nginx-reverse-proxy-docker) from Google. It details how to use Nginx as a reverse proxy with [Let’s Encrypt](https://letsencrypt.org/) certificates that are auto renewed every month.

### Samba

I have a basic Samba server running which gives me easy access to the files on the RAID 1 array (which I use as a NAS).

### VPN

For the VPN I used a [Docker container](https://github.com/kylemanna/docker-openvpn) which supports both UDP and TCP and [obfsproxy](https://blog.torproject.org/obfsproxy-next-step-censorship-arms-race), which obfuscates the traffic so that Deep Packet Inspection cannot figure out which protocol is used and thus can not selectively block it.

Generally I connect directly to the UDP container, but when that doesn’t work I use obfsproxy + OpenVPN TCP.

This is a good tutorial explaining how to use obfsproxy and Open VPN: [How to hide OpenVPN traffic using Obfsproxy on a Windows PC and Linux EC2 server](https://www.comparitech.com/blog/vpn-privacy/hide-openvpn-traffic-obfsproxy-on-windows-and-linux-ec2/).

### TightVNC

I use a TightVNC server to be able to launch long running visual processes. Here is a great guide on how to install it and Xfce in Ubunut: [Digital Ocean guide](https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-vnc-on-ubuntu-18-04).

## Main changes

  * Tried pfSense, returned to router
  * Gigabit ethernet switch
  * Dockerized most servicies
  * Switched to Ubuntu 18.04
  * Added 120GB system drive to Tesla
  * Got a new low power quad core Celeron server that is currently unused

gha_language: CSS