text:
# Mainflux Core Server

[![License](https://img.shields.io/badge/license-Apache%20v2.0-blue.svg)](LICENSE)
[![Build Status](https://travis-ci.org/Mainflux/mainflux-core-server.svg?branch=master)](https://travis-ci.org/Mainflux/mainflux-core-server)
[![Go Report Card](https://goreportcard.com/badge/github.com/Mainflux/mainflux-core-server)](https://goreportcard.com/report/github.com/Mainflux/mainflux-core-server)
[![Join the chat at https://gitter.im/Mainflux/mainflux](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/Mainflux/mainflux?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

Mainflux Core Microservice for Mainflux IoT Platform.

 For example:
```bash
mkdir -p ~/go
export GOPATH=~/go
export GOBIN=$GOPATH/bin
```

#### Get the code
Use [`go`](https://golang.org/cmd/go/) tool to "get" (i.e. fetch and build) `mainflux-core-server` package:
```bash
go get github.com/mainflux/mainflux-core-server
```

This will download the code to `$GOPATH/src/github.com/mainflux/mainflux-core-server` directory,
and then compile it and install the binary in `$GOBIN` directory.

Now you can run the server:
```bash
$GOBIN/mainflux-core-server
```

Please note that the binary `mainflux-core-server` expects to find configuration file `config.yml` in
direcotry provided by `MAINFLUX_CORE_SERVER_CONFIG_DIR` if this variable is set. Otherwise it looks for `config.yml`
in `$GOPATH/src/github.com/mainflux/mainflux-core-server`.

Note also that using `go get` is prefered than out-of-gopath code fetch by cloning the git repo like this:
```
git clone https://github.com/Mainflux/mainflux-core-server && cd mainflux-core-server
go get
go build
MAINFLUX_CORE_SERVER_CONFIG_DIR=. ./mainflux-core-server
```
#### Dependencies
Mainflux Core Server is connected to `NATS` on northbound interface, and to `MongoDB` and `InfluxDB` southbound.

Following diagram illustrates the architecture:
![Mainflux Arch](https://github.com/Mainflux/mainflux-doc/blob/master/mermaid/arch.png)

This is why to run Mainflux Core Server you have to have running:
- [NATS](https://github.com/nats-io/gnatsd)
- [MongoDB](https://github.com/mongodb/mongo)
- [InfluxDB](https://github.com/influxdata/influxdb)

Installation and start of these services depends the operating system running on host (e.g. for Debian you can use `apt-get` to fetch and install these), so you must follow the installation instructions for each of the project.

However, each of these projects provides an official Docker image which can be pulled from DockerHub and started in a separate container (each in separate terminal if not detached):
```bash
docker run -p 4222:4222 -it apcera/gnatsd
docker run -p 27017:27017 -it mongo
docker run -p 8086:8086 -it influxdb
```
Now you can run `mainflux-core-server`:
```bash
MAINFLUX_CORE_SERVER_CONFIG_DIR=. ./mainflux-core-server
```

Note that when running services in this way (weather they are installed in the localhost system or run and mapped on localhost ports) you will need to change [`config.yml`](config.yml) and replace `influx`, `mongo` and `nats` hostnames by `localhost`

### Documentation
Development documentation can be found on our [Mainflux GitHub Wiki](https://github.com/Mainflux/mainflux/wiki).

Swagger-generated API reference can be foud at [http://mainflux.com/apidoc](http://mainflux.com/apidoc).

### Community
#### Mailing lists
- [mainflux-dev](https://groups.google.com/forum/#!forum/mainflux-dev) - developers related. This is discussion about development of Mainflux IoT cloud itself.
- [mainflux-user](https://groups.google.com/forum/#!forum/mainflux-user) - general discussion and support. If you do not participate in development of Mainflux cloud infrastructure, this is probably what you're looking for.

#### IRC
[Mainflux Gitter](https://gitter.im/Mainflux/mainflux?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

#### Twitter
[@mainflux](https://twitter.com/mainflux)

### License
[Apache License, version 2.0](LICENSE)

repository: AtticFantastic/mainflux-core-server

repo_name: AtticFantastic/mainflux-core-server


text:
# Climber

A Flask application for handling a fussball ladder tournament system.

## Docker

In order to get started with Docker, you first have to build the image by running the following command inside the project folder:

    docker build --tag climber .

After that you will be able to start a container with the image by running:

    docker run --detach --publish-all --name=climber climber

Then check the port it is available on by using:

    docker port climber 5000

license: MIT

repository: Tenzer/climber

branch_name: refs/heads/master

extension: md

license_type: permissive

repo_name: Tenzer/climber


text:
# Projecter

[![Code Climate](https://codeclimate.com/github/gray-industries/projecter/badges/gpa.svg)](https://codeclimate.com/github/gray-industries/projecter)
[![Build Status](https://travis-ci.org/gray-industries/projecter.svg?branch=master)](https://travis-ci.org/gray-industries/projecter)

Projecter is a simple CLI application generator.

## Installation

    $ gem install projecter

## Usage

Using projecter's quite a bit like running `bundle gem gemname`.

When you

	$ projecter create $PROJECT

you get a new gem in $PROJECT, complete with its own fresh git repo.

### Directory Layout

If you create an app called `myapp`, projecter creates the following directories:

- `lib/commands/`: `myapp`'s subcommands.
- `lib/myapp/`: `myapp`'s library.
- `spec/fixtures`: test fixtures.
- `spec/resources`: configs, other files needed to run tests.
- `spec/unit/lib/{commands,myapp}/`: unit tests.
- `spec/integration/lib/{commands,myapp}/`: integration tests.
- `spec/acceptance/lib/{commands,myapp}/`: large scale system or subsystem tests; stress tests; performance tests; interface conformance tests for demonstrating the adherence of APIs, CLIs, etc. to established standards.

Each leaf directory contains an empty `.gitignore` file. This is because git does not track directories, and without an empty file the directory would not appear in the repo.

## Contributing

1. Fork it
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Added some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
5. Create new Pull Request

repository: gray-industries/projecter

gha_language: Ruby

repo_name: gray-industries/projecter


text:
# Cymbella Pat. GENUS

#### Status
SYNONYM

#### According to
Index Fungorum

#### Published in
in Doassans & Patouillard, Revue mycol. , Toulouse 8: 27 (1886)

#### Original name
Cymbella Pat.

### Remarks
null

repository: mdoering/backbone

repo_name: mdoering/backbone


text:
# Story

Airbyte was founded by Michel and John in December 2019.

Our first idea was to help companies exchange data, starting by building a customer portal for data providers so their clients can easily evaluate and pull data however and wherever they want.

During our time at YC, we pivoted to a data unblocking product which allowed website owners to get 100% of their customer traffic data for analytics, even when visitors are using ad blockers. The product was getting good traction until COVID-19 hit, and marketing priorities shifted, making this product a nice-to-have rather than a must.

At the time, we were meeting with a lot of companies. As we were discussing their data problems, a pattern started to emerge. All were building in some way their own data integration pipelines, even though there were already strong cloud-based solutions for it. We understood that data integration was not a solved problem. So we decided to go back to our roots, making moving data easy and address this issue.

There are more and more data products out there, event-based customer data pipelines a la Segment and batch data ingestion pipelines a la Fivetran. But they are cloud-based and entail moving your data out of your infrastructure. In addition to adding unnecessary costs, this poses a data privacy risk for more and more companies. In the end, engineers end up having to develop and maintain these connections themselves.

That’s why we decided to build Airbyte, a new self-hosted, open-source data integration engine.

Open sourcing the core engine is a prerequisite to building a top class engine so that anyone can add their own integrations and extend the project with the features that serve their use cases.

And that’s how we started!

repository: MinjaMiladinovic/airbyte

gha_language: Java

gha_license_id: MIT

repo_name: MinjaMiladinovic/airbyte


text:
# Raw Data for the `Correlated Errors` Case

## `proto-covariance.xml`

- This is a mock VOTable.
- It has been inspired from the use case of high accuracy astrometric data set e.g. Gaia.
- Both data and meta-data have been named and set in a way that facilitates the client code testing.

 - The data table contains the following quantities:
   - A position (2D vector)
   - A proper motion (2D vector)
   - A radial velocity (1D)
   - A parallax (1D)

-  All quantities have their proper error
   - Scalar error for the 1D values
   - 2D error + correlation for the 2D values

- Errors on different quantities can be correlated
   - Position and proper motion errors are correlated (4 values)
   - Position and radial velocity errors are correlated (2 values)
   - Position and parallax errors are correlated (2 values)
   - Proper motion and parallax errors are correlated (2 values)

license: Unlicense

repository: mcdittmar/dm-usecases

branch_name: refs/heads/main

gha_language: Awk

gha_license_id: Unlicense

repo_name: mcdittmar/dm-usecases


text:
# Chatbot-ML
It is a chatbot project created using machine learning models

It is an intent based chatbot,which predict the class of the question by using nlp and logistic regression,naive bayes and decision tree algorithms and then gives the response based on the class predicted.
It also have static restuarant database which provide the information asked by the user

After running gui.py you can directly ask questions and if your question are identified by "restaurant" intent then chatbot will enter restuarant-based mode and provide restaurant specific information.

repository: vash-ashutosh/Chatbot-ML

repo_name: vash-ashutosh/Chatbot-ML


text:
[![license](https://img.shields.io/badge/license-Apache%20License%202.0-blue.svg?style=flat)](http://www.apache.org/licenses/LICENSE-2.0)

# CUBA Greek Translation Add-on

Greek translation of CUBA Platform (plus commercial components) implemented as an add-on.

## Installation
This add-on's repository is officially linked to the main CUBA repository, so you can install it from the Marketplace
or by adding the repository manually to your project.

The following table shows which version of the add-on is compatible with which version of the platform:

| Platform Version | Add-on Version | Coordinates
| ---------------- | -------------- | ------------
| 7.2.*            | 1.0.0          | gr.netmechanics.cuba.translationel:translationel-global:1.0.0

The latest stable version is: `1.0.0`

### From Marketplace
You can use **CUBA Studio** / **IntelliJ IDEA** to add it to your project: choose the `CUBA -> Marketplace...` menu item,
find the *Greek Translation* add-on, then click on the `Install` button.

### Adding the repository manually
Modify the `build.gradle` of your CUBA application. First add the repository `https://dl.bintray.com/netmechanics/cuba-components`
to the buildscript.
```gradle
buildscript {
    ...
    repositories {
        ...
        maven {
            url 'https://dl.bintray.com/netmechanics/cuba-components'
        }
        ...
    }
    ...
}
...
```
Then select the version of the add-on which is compatible with the platform version of your project
and add custom application component to your project
```gradle
...
dependencies {
    ...
    appComponent('gr.netmechanics.cuba.translationel:translationel-global:1.0.0')
    ...
}
...
```

## Enable Greek Language
From the `CUBA -> Project Properties...` menu item, click the edit button next to `Available locales`.
Then click the add button and fill the field `Language` with "Greek". The field `Code` should be auto filled with "el".

 ![](cuba-translation-el.png)

repository: pbaris/cuba-translation-el

repo_name: pbaris/cuba-translation-el


text:
@function can-ejs/tags.templated <%% CODE %>
@parent can-ejs/tags 3

@signature `<%% CODE %>`

Renders <% CODE %> as text in result of the template rather than running CODE itself. This is useful for generators.

The following results in "<%= 'hello world' %>" rather than the string "hello world."

         <%%= 'hello world' %>

repository: canjs/can-ejs

gha_language: JavaScript

repo_name: canjs/can-ejs


text:
This project was bootstrapped with [Create React App](https://github.com/facebook/create-react-app).

The repository contains code for a multimodal annotation preprocessing tool built using React and PHP.

GSoC report links:
First Evaluation Report: https://github.com/FrameNetBrasil/webtool/blob/gsoc2020_2/GSoC2020/weekly_reports/FirstEvaluationReport_idea2.md
Second Evaluation Report: https://github.com/FrameNetBrasil/webtool/blob/gsoc2020_2/GSoC2020/weekly_reports/SecondEvaluationReport_idea2.md
Final Report<EMAIL_ADDRESS>

This distribution is running as a Docker container. In order to run this container you'll need docker installed:

* [Docker Windows](https://docs.docker.com/windows/started)
* [Docker OS X](https://docs.docker.com/mac/started/)
* [Docker Linux](https://docs.docker.com/linux/started/)

### Usage

* Clone this repository at an accessible folder

```sh
$ git clone https://github.com/FrameNetBrasil/charon.git
```
* Download the trained model file from this google drive link, and save it in the public/server folder
https://drive.google.com/file/d/1iw7VIoBspqgChl-4nDU809A6rvwZqKk5/view?usp=sharing

* Start the container

```sh
$ cd charon/devops/charon
$ docker-compose up
```

* Build frontend app

```sh
$ cd charon
$ docker exec -it fnbr-webserver-charon bash

// inside the container

# npm install
# npm run build
```

* Access the app at http://localhost:8600

* To start the flask server follow these steps inside the root directory of the container:

```sh
$ cd public/server
$ apt-get install python3.7
$ apt-get install python3-pip python3-dev python3-venv
$ python3 -m pip install --upgrade pip
$ python3 -m pip install numpy matplotlib keras opencv-python scikit-learn scikit-image scipy argparse mysql-connector-python Pillow pickle-mixin glob3 flask opencv_contrib_python mpld3 moviepy
$ python3 -m pip install flask_cors, requests
$ apt update && apt install -y libsm6 libxext6
$ apt-get install -y libxrender-dev
$ python3 -m pip install --upgrade pip
$ python3 -m pip install tensorflow==2.2.0 --no-cache-dir
$ python3 -m pip install elementpath
$ apt-get install python3-tk
$ python3 track_objects.py
```
The server will start running on http://<IP_ADDRESS>:5000/

#### Container Parameters

Parameters are available to the container (mainly the webserver port) at

```shell
/charon/devops/charon/.env
```

Also adjust React env at (baseURL for API)

```shell
/charon/.env
```
Note: Since frame element updates are not being made currently in the ObjectMM table, to avoid referential integrity errors, temporarily remove the foreign keys by executing the following sql commands in the ObjectMM table from phpmyadmin:

```shell
ALTER TABLE objectmm DROP FOREIGN KEY fk_ObjectMM_AnnotationSetMM
ALTER TABLE objectmm DROP FOREIGN KEY fk_ObjectMM_FrameElement
```

## Built With

* PHP 7.4
* MariaDb 10.4
* PhpMyAdmin 5.0.1
* React
* IBM Cloud Speech-to-Text API Service Credentials (User Account- Prishita Ray):

{
  "apikey": "jrdLqCqvqz9JU8Eu8Ls7c40_uXTmCFrb3iWbLk77KgvJ",
  "iam_apikey_description": "Auto-generated for key d372a095-0998-4f7f-8557-afcf7301ca4a",
  "iam_apikey_name": "Service credentials-1",
  "iam_role_crn": "crn:v1:bluemix:public:iam::::serviceRole:Manager",
  "iam_serviceid_crn": "crn:v1:bluemix:public:iam-identity::a/6de8c018fee0fb1865ab45744f2f9483::serviceid:ServiceId-fac502ab-3ee9-44df-b3d5-bcb2158a69cd",
  "url": "https://api.us-south.speech-to-text.watson.cloud.ibm.com/instances/0c057c58-d790-44ba-97b5-2094825614fa"
}

Service Access URL: https://stream.watsonplatform.net/speech-to-text/api/v1/recognize?end_of_phrase_silence_time=2.0&split_transcript_at_phrase_end=true&speaker_labels=true

<!--
### Steps to reproduce:
1. Install React, npm and Node js on your system.
2. Install Xampp to run the development server.
3. Add the paths of these packages to the environment variables of your system.
4. Clone this repository using Github Desktop.
5. Copy the server and vendor folders listed under the src directory to your local xampp/htdocs/ folder.
6. Install composer on your system, and copy the composer.json file present in src to your xampp/htdocs folder.
7. Open another command prompt or terminal, move to the directory xampp/htdocs and then run `composer require php-ffmpeg/php-ffmpeg`
8. Run the add_paths.bat file if using a Windows System or add_paths.sh file if using a Linux system, to add the paths to environment variables.
9. Open xampp control panel as administrator and start apache and mysql.
10. Access the phpmyadmin page at `localhost/phpmyadmin` and import the webtool mysql database dump located at mariadb/webtool_github_bkp.sql.gz (You may have to update the entry:`$cfg['ExecTimeLimit'] = 0;` of the config.default.php file in the phpmyadmin/libraries folder of xampp to import the whole dump without problems of time limit being exceeded which can break the import.). Also change the `max_execution_time` entry in php.ini under xampp/php/ to 0 , i.e. `max_execution_time=0`
9. Open the folder of the cloned repository on your machine using Command Prompt or Terminal eg: `cd Desktop/charon`
10. Run `npm install` to install all the dependencies to run the app.
11. Run `npm start` to start the app.
12. The app will open at localhost:3000 in a tab on your browser
13. Select a video file from your computer or enter a URL. You will receive a few notifications from the app. Then select your language, corpus and document from the dropdown lists. Finally click on the Upload Files button.
14. Wait for the server to run, as it may take some time to generate the audio transcripts. The video file, extracted audio file, generated thumbnail and audio transcript text files will be stored under `Video_Store/full/`, `Audio_Store/audio/`, `Images_Store/thumbnails/` and `Text_Store/transcripts` folders under src respectively, if all the size constraints, duplicate checks, URL validation, etc. checks are successful.
15. A new entry in the documentmm table will be generated, that can be accessed using phpmyadmin.
-->


text:
---
date: 2022-02-04
title: 'Best Practices for working with Dockerfile'
shortTitle: 'Best Practices for working with Dockerfile'
description: 'Best Practices for working with Dockerfile'
template: post
featuredImage: '../thumbnails/docker.png'
thumbnail: '../thumbnails/docker.png'
slug: best-practices-of-using-dockerfile
categories:
  - docker
  - microservice
  - aws
tags:
  - docker
  - k8s
  - microservice
---

# Best Practices for working with Dockerfile

This guide assumes you are familiar with Docker and its build environment. Let's review some of the basic concepts before you start to put them into practice.

### What is a Docker image?
A Docker image is a template that allows you to instantiate running containers. It is represented as a list of instructions (known as layers) in a filesystem.

### What is a Dockerfile?
A Dockerfile is just a blueprint that contains the instructions to build a Docker image. Currently, more than a million Dockerfiles are on GitHub.

### What does "Docker build" mean?
The process of building a Docker image from a Dockerfile is known as a Docker build.

Find detailed information in the Dockerfile reference.

### What is a Docker layer?
Each layer in a Docker context represents an instruction included in a Docker image's Dockerfile. The layers can also be referred to as "build steps".

### What is the Docker build cache?
Every time you build a Docker image, each build step is cached. Reuse cached layers that do not change in the image rebuild process to improve the build time.

### Concerns when building images
These are the main areas of improvement I will cover in this guide:

- Consistency: If you are consistent designing your images, they are easier to maintain and you will reduce the time spent when developing new images.
Build Time: Especially when your builds are integrated in a Continuous Integration pipeline (CI), reducing the build time can significantly reduce your apps' development cost.
Image Size: Reduce the size of your images to improve the security, performance, efficiency, and maintainability of your containers.
Security: Critical for production environments, securing your containers is very important to protect your applications from external threats and attacks.

Image Source and Credits: http://pierre-jean.baraud.fr/blog/2014/05/14/fist-look-dockerfile/

Dockerfiles allow users to define the exact actions needed to create a new container image. This allows users to write the execution environment as if it were code, storing it in version control if desirable.

The same Docker file built in the same environment will always produce an identical container image. Dockerfiles helps in automating the building of container images and establishes a repeatable process.

Some of the benefits Dockerfiles provide are:

* Easy versioning: Dockerfiles can be committed and maintained via version control to track changes and revert any mistakes.

* Predicbility: Building images from a Dockerfile helps remove human error from the image creation process.

* Accountability: If you plan on sharing your images, it is often a good idea to provide the Dockerfile that created the image as a way for other users to audit the process.

* Flexibility: Creating images from a Dockerfile allows you to override the defaults that interactive builds are given. This means that you do not have to provide as many runtime options to get the image to function as intended.
* Docker images have intermediate layers that increase reusability, decrease disk usage, and speed up docker build by allowing each step to be cached.

![Image Source and Credits: [http://blog.bigstep.com/developers-love-docker/](http://blog.bigstep.com/developers-love-docker/)](https://cdn-images-1.medium.com/max/2000/1*rCBBYVVJBlnNdo2f3OVyWw.png)*Image Source and Credits: [http://blog.bigstep.com/developers-love-docker/](http://blog.bigstep.com/developers-love-docker/)*

## Best Practices for writing Dockerfiles:

### Use a .dockerignore file

The best way is to put the Dockerfile inside the empty directory and then add only the application and configuration files required for building the docker image. To increase the build’s performance, you can exclude files and directories by adding a .dockerignore file to that directory as well.

### Containers should be immutable & ephemeral

The container created with the image produced by Dockerfile should be ephemeral and immutable. In other words, the container should be destroyed and a new one built and put in place with an absolute minimum set-up and configuration.

### Minimize the number of layers / Consolidate instructions

Each instruction in the Dockerfile adds an extra layer to the docker image. The number of instructions and layers should be kept to a minimum as this ultimately affects build performance and time.

### Avoid installing unnecessary packages

In order to reduce complexity, dependencies, file sizes, and build times, avoid installing unnecessary packages.

### Sort multi-line arguments

Sorting multiline arguments alphanumerically will help avoid duplication of packages and make the list much easier to update.

    RUN  yum update -y && \
         yum install -y apache2 \
                        git \
                        java \
                        python

### Build cache

While building an image, Docker will step through the instructions mentioned in the Dockerfile, executing them in chronological order. As each instruction is examined Docker will look for an existing image layer in its cache that it can reuse, rather than creating a new image layer.
- If you do not want to use the cache at all, then use the--no-cache=true option with the docker build command.

However, when Docker is not allowed to use its cache, then the basic rules Docker will follow to find a matching image are mentioned below:

* Starting with a base image that is already in the cache, the next instruction is compared against all child images derived from that base image to see if one of them was built using the exact same instruction. If not, the cache is invalidated.

* For the ADD and COPY instructions, the contents of the file(s) in the image are examined and a checksum is calculated for each file. During the cache lookup, the checksum is compared against the checksum in the existing images. If anything has changed in the file(s), such as the contents and metadata, then the cache is invalidated.

* Aside from the ADD and COPY commands, cache checking will not look at the files in the container to determine a cache match. For example, when processing a RUN apt-get -y update command the files updated in the container will not be examined to determine if a cache hit exists. In that case just the command string itself will be used to find a match.

Once the cache is invalidated, all subsequent Dockerfile commands will generate new images and the cache will not be used.

### Build every time

Building docker images is very fast as docker makes use of previously cached build steps (default). By building the image every time, one can use containers as reliable artifacts. For example, one can go back and run a container from previous docker image to inspect a problem, or can run long tests on the previous version image while editing the code.

### Dockerfile for Development Environment

For a development environment, map your source code on the host to a container using a volume. This enables to choose the editor of your choice on the host and test the application right away in the container. This is done by mounting the application build folder as a volume rather than copying the build artifact using the ADD command in the Dockerfile.

### Understand CMD and ENTRYPOINT

CMD simply sets a command to run in the image if no arguments are passed to docker run, while ENTRYPOINT is meant to make your image behave like a binary.

* If the Dockerfile uses only CMD, the specified command is executed if no arguments are passed to docker run.

* If the Dockerfile uses only ENTRYPOINT, the arguments passed to docker run are always passed to the entrypoint; the entrypoint is executed if no arguments are passed to docker run.

* If the Dockerfile declares both ENTRYPOINT and CMD and no arguments are passed to docker run, then the argument(s) to CMD are passed to the declared entrypoint.

Source: ~ Docker Docs

Disclaimer: Content and Image source has been mentioned


text:
# Challenge Generator

This is a Challenge Generator For Binding Of Isaac - Afterbirth+.

Try it out here:
[Binding of Isaac - Challenge Generator](https://josh-cook.github.io/challenge-generator/)

A daily seed is automatically generated based on the days current date.
If a different seed is needed there is the ability to generate a new seed which will refresh upon generation.

There is also the ability to share the seed by copying it to clipboard and pasting wherever necessary.

## Building the generator

- Checkout the project
- Inside the project folder you will need npm to build this first `npm install`
- Then build the project `npm run build`
- After build has complete you can then spin the project up: `npm start`
- Go to the server the generator has been built on
- It will look something like the following:

![Alt text](/readme-assets/build-image.png "Title")

repository: jackwilsdon/challenge-generator

repo_name: jackwilsdon/challenge-generator


text:
# TestFlight iOS Native Extension
Adobe Air Native Extension (ANE) for the TestFlight SDK on iOS

## About
This extension provides a wrapper around the TestFlight SDK. The goal is to keep the API as close to the Objective-C TestFlight API as possible.

## Building
The native extension is using ruby's rake for it's build system and therefore requires ruby 1.9.3, this is installed by default on OS X Lion.  Additionally, you must have Xcode and the iOS SDK's installed on your system.

To build simply type `rake build` into your terminal from within the root directory of the native extension.  If this is your first time building the build system will generate the file `config/build.yml` and prompt you to edit the file for your system.  The relevant changes are the path to your Adobe Air SDK and what version of the iOS SDK you are using.

After editing your `config/build.yml` file, simply type `rake build` again.  If all goes well you will see the `TestFlight.ane` file sitting in your `bin` directory.

## Usage
The api is a single class called TestFlight in the `com.thejustinwalsh.ane` package. The class consists entirely of static methods that wrap the TestFlight SDK.

Call `TestFlight.takeOff("app_id")` with your application token to initialize TestFlight. You should ensure that you call the `takeOff` method after the `NativeApplication` signals the `ACTIVATE` event, calling `takeOff` too soon will result in a bad time. If this is a debug build you may want to call `TestFlight.setDeviceIdentifier()` before calling `TestFlight.takeOff` so that your sessions are bound to the proper user.

## Considerations
I am not certain if Apple will approve your ANE using this SDK with the included deprecated `[[UIDevice currentDevice] uniqueIdentifier]]` call. If you would like to use this TestFlight ANE in an App Store build, it is advised that you comment out the code in `TestFlight.m` that uses the deprecated function call.

## Donating
Support this project and [others by thejustinwalsh][gittip-thejustinwalsh] via [gittip][gittip-thejustinwalsh].

[![Support via Gittip][gittip-badge]][gittip-thejustinwalsh]

[gittip-badge]: https://rawgithub.com/twolfson/gittip-badge/master/dist/gittip.png
[gittip-thejustinwalsh]: https://www.gittip.com/thejustinwalsh/

## License
TestFlightANE is license under a permissive MIT source license. Fork well my friends.

	Copyright (c) 2012 Justin Walsh, http://thejustinwalsh.com/

	Permission is hereby granted, free of charge, to any person obtaining
	a copy of this software and associated documentation files (the
	"Software"), to deal in the Software without restriction, including
	without limitation the rights to use, copy, modify, merge, publish,
	distribute, sublicense, and/or sell copies of the Software, and to
	permit persons to whom the Software is furnished to do so, subject to
	the following conditions:

	The above copyright notice and this permission notice shall be
	included in all copies or substantial portions of the Software.

	 IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
	LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
	OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
	WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

repository: tconkling/TestFlightANE

repo_name: tconkling/TestFlightANE


text:
22 December 2017

- Resolves an issue which led to very brief loss of connectivity
    after upgrading to v3.0.0 from v2.6.4 when using an etcd datastore.
    [https://github.com/projectcalico/felix/pull/1676]

- Certain configuration changes no longer cause Felix to restart, allowing
    upgrades to Calico v3.0.0. [felix #1631](https://github.com/projectcalico/felix/pull/1631) (@fasaxc)

- Calico now assigns the host side of veth pairs a MAC address of `ee:ee:ee:ee:ee:ee`.
    If this fails, it uses a kernel-generated MAC address as before. For more information,
    refer to the [Troubleshooting FAQ](https://docs.projectcalico.org/v2.6/usage/troubleshooting/). [cni-plugin #436](https://github.com/projectcalico/cni-plugin/pull/436) (@tmjd)

- `calicoctl` now includes a new environment variable called `CALICO_LIBNETWORK_VETH_MTU`
    that allows you to configure the MTU of veth endpoints when using the Docker orchestrator.
    [libnetwork-plugin #164](https://github.com/projectcalico/libnetwork-plugin/pull/164) (@ti-mo)

repository: daiyinliang/calico

repo_name: daiyinliang/calico


text:
Open-PDKs (open_pdks)
-----------------------------------------

Summary:

Automatic setup of PDKs for open-source tools from foundry sources.

Builds out and populates a new set of directories and subdirectories in
the efabless format, with the efabless name for the PDK at the top,
followed by categories "libs.ref" (IP) and "libs.tech" (EDA tool setup),
each with subcategories corresponding to layout, abstract views,
netlists, etc. for the IP;  and magic, netgen, qflow, etc., for the
EDA tool setup.

The populated PDK directories can contain either copies of files from
the foundry sources, links to the foundry sources, or links back to
another PDK.

Generates magic layout views for all vendor IP for which either a GDS
view or a LEF view exists.  Annotates the views as needed to handle
ports, bounding boxes, etc.

-----------------------------------------

This distribution contains sources for building out the SkyWater SKY130
130nm process.  Sources for the foundry process data must be obtained
separately.  Read the README file in subdirectory sky130/ for instructions
on obtaining and building the SkyWater sky130A PDK.

-----------------------------------------

License:

Open_PDKs is open-source software distributed under the Apache-2.0 license.
See file LICENSE for the complete license text.

-----------------------------------------

Instructions:

    git clone https://github.com/RTimothyEdwards/open_pdks.git
    cd open_pdks
    ./configure [options]
    make
    [sudo] make install

where the configure [options] are specific to each foundry PDK supported
by open_pdks and can be queried using

    ./configure --help

At a minimum you will want to pass a configure option to declare the location
of the PDK vendor source distribution (which can be obtained separately),

    ./configure --enable-[PDK_name]-pdk=[path]

The open_pdks version of the PDK can be built locally within open_pdks without
the need to declare an install target or run "make install".  However, it is
recommended to declare a target location and install there, using

    ./configure --prefix=[path] --enable-[PDK_name]-pdk=[path]

followed by "make" and "make install".

Also see the website at http://opencircuitdesign.com/open_pdks/.  The "Install"
page has full instructions for configuring and installing open_pdks.


text:
---
layout: page
title: "ProteinDF software package"
description: ""
---

{% include JB/setup %}

[English](./index.html)
[Japanese](./index_j.html)

![ProteinDF logo](img/PDFLOGO.jpg "ProteinDF logo")

## About

ProteinDF is a free and open source quantum chemistry calculation software package.

## News

  {% for post in site.posts %}

{{ post.date | date_to_string }} » {{ post.title }}
  {% endfor %}

## GitHub Repository

### ProteinDF

[https://github.com/ProteinDF/ProteinDF](https://github.com/ProteinDF/ProteinDF)

### ProteinDF_bridge

[https://github.com/ProteinDF/ProteinDF_bridge](https://github.com/ProteinDF/ProteinDF_bridge)

### ProteinDF_pytools

[https://github.com/ProteinDF/ProteinDF_pytools](https://github.com/ProteinDF/ProteinDF_pytools)

### QCLObot

[https://github.com/ProteinDF/QCLObot](https://github.com/ProteinDF/QCLObot)

## manual

- [ProteinDF user manual](http://proteindf.github.io/ProteinDF_userman/en/index.html)
- [QCLObot user manual](http://proteindf.github.io/QCLObot/)

## Cite

### ProteinDF

- F. Sato, Y. Shigemitsu, I. Okazaki, S. Yahiro, M. Fukue, S. Kozuru, H. Kashiwagi, “Development of a new density functional program for all-electron calculation of proteins”, Int. J. Quant. Chem., 63, 245-246 (1997).
- I. Okazaki, F. Sato, T. Yoshihiro, T. Ueno, H. Kashiwagi, "Development of a restricted open shell Kohn–Sham program and its application to a model heme complex", THEOCHEM, 451, 109-119 (1998).
- T. Yoshihiro, F. Sato, H. Kashiwagi, "Distributed parallel processing by using the object-oriented technology in ProteinDF program for all-electron calculations on proteins", Chem. Phys. Lett., 346, 313–321 (2001).
- T. Inaba, F. Sato, "Development of parallel density functional program using distributed matrix to calculate all-electron canonical wavefunction of large molecules", J. Comput. Chem., 28, 984–995 (2007).
- T. Hirano, F. Sato, "A third-generation density-functional-theory-based method for calculating canonical molecular orbitals of large molecules", PCCP, 16, 14496-14503 (2014).

### QCLO

- H. Kashiwagi, H. Iwai, K. Tokieda, M. Era, T. Sumita, T. Yoshihiro, F. Sato, "Convergence process with quasi-canonical localized orbital in all-electron SCF calculation on proteins", Mol. Phys., 101, 81-86 (2003).
- N. Nishino-Uemura, T. Hirano, F. Sato, "Study of the quasicanonical localized orbital method based on protein structures", J. Chem. Phys., 127, 184106 (2007).

## References using the ProteinDF

- T.Hirano, F.Sato, “Theoretical study of the receptor-binding domain of spike protein of SARS-CoV-2 by canonical molecular orbital calculation”, AIP Conference Proceedings, 2611, 020005 (2022).
- R.Nakaoka, T.Hirano, F.Sato, “Study on mechanism of electronic states of Interferon alpha2”, Seisan Kenkyu, 74, 153-159 (2022).
- T.Wang, T.Hirano, F.Sato, “Study on the Role in the Catalytic Reaction Occurrence of the Active Site in PETase by Canonical Molecular Orbital Method", Seisan Kenkyu, 74, 161-167 (2022).
- T.Hirano, F.Sato, “Interaction Energy Analysis Based on Canonical Kohn-Sham Molecular Orbitals Calculation of Protein”, AIP Conference Proceedings, 2343, 020007 (2021).
- D.Maeda, T.Hirano, F.Sato, “Analysis of Electronic Structure of Photoactive Yellow Protein by Canonical Molecular Orbitals Calculation", Seisan Kenkyu, 73, 157-163 (2021).
- T.Hirano, F.Sato, "Interaction energy analysis based on canonical Kohn-Sham molecular orbital calculation of protein", DFT2020 (2020).
- H.Sasaki, T.Hirano, F.Sato, "Comparison of Electronic Structures between Insulin-like Growth Factor 1 and Insulin", Seisan Kenkyu, 72, 239-246 (2020).
- T.Hirano, F.Sato, “Recent progress of protein canonical molecular orbital calculation by the third generation density functional method”, AIP Conference Proceedings 2186, 030009 (2019).
- H.Eguchi, T.Hirano, F.Sato, "Study on the Cloud-like Visualization Technique of Protein Molecular Orbitals by the Rejection Method", Seisan Kenkyu, 71, 769-773, (2019).
- T.Hirano, F.Sato, "The electronic structure of the active site in the glucose oxidase", HPCI Research Report, 3, 11-16 (2018).
- H.Eguchi, T.Hirano, F.Sato, "Study on the Cloud-like Visualization Model of Protein Molecule Orbitals by the Rejection Method", J. Comput. Chem., J., 17, 189-192 (2018).
- T.Hirano, F.Sato, "Study of High-Performance Canonical Molecular Orbitals Calculation for Proteins", AIP Conference Proceedings, 1906, 030019 (2017).
- S.Kihira, T.Hirano, F.Sato, "Study on Stable Structures of Oxytocin by Full-Quantum Chemical Calculation", Seisan Kenkyu, 68, 219-223, (2016).
- T.Kim, T.Hirano, F.Sato, "Development of atomic charges for proteins by using the linear regression method based on canonical molecular orbital calculation", Seisan Kenkyu, 68, 213-217, (2016).
- J.Yoshida, T.Hirano, F.Sato, "Electronic state calculation of the active center of Glucose Oxidase by QCLO method", Seisan Kenkyu, 67(3), 265-272 (2015).
- T.Hirano, F.Sato, “A third-generation density-functional-theory-based method for calculating canonical molecular orbitals of large molecules”, Phys. Chem. Chem. Phys., 16, 14496-14503 (2014).
- H.Higuchi, T.Hirano, F.Sato, "Study of Practical Approach for Grid-free XC Computational Method", Seisan Kenkyu, 66, 273 (2014).
- K.Chiba, T.Hirano, F.Sato, M.Okamoto, “Clarification of the role of protein in carbonmonoxy myoglobin by investigating electronic states”, Int. J. Quant. Chem., 113, 2345-2354, (2013).
- T.Shimomukai, K.Chiba, J.Matsuda, T.Hirano, F.Sato, "All-Electron Wave Functions and the Density-of-States Analyses of Proteins", J. Vis. Soc. Jpn., 32, 132-137 (2013).
- T.Hirano, F.Sato, "Cholesky decomposition method for canonical molecular orbital calculations of proteins", 31th JSST Annual Conference proc., (2012).
- T.Kuroda, T.Hirano, F.Sato, "Study of the Electronic States of the ［Ni-Fe］ and ［Ni-Fe-Se］ Hydrogenase Active Center", Seisan Kenkyu, 64, 345 (2012).
- M.Hori, T.Hirano, F.Sato, "Computational Study of Key Steps of RuBisCO Carboxylase Reaction and Roles of Active-Site Residues", Seisan Kenkyu, 64, 351 (2012).
- T. Hirano, F. Sato, "Massively parallel computation strategies for canonical molecular orbital calculations of proteins", 30th JSST Annual Conference proc., (2011).
- Y.Tokita, S.Yamada, W.Luo, Y.Goto, N.Bouley-Ford, H.Nakajima, Y.Watanabe, "Protein photoconductors and photodiodes", Angew. Chem., 50, 11663-6 (2011)
- K.Ishikawa, T.HIRANO, F.SATO, "Study on Model Chemistry for All-Electron Calculation of Proteins", Seisan Kenkyu, 62, 241, (2010).
- T.Hirano, K.Chiba, F.Sato, "ProteinDF: An All-electron Density Functional Program Package for Proteins", Trans. Jpn. Soc. Sim. Tech., 4, 50-59 (2009).
- T.Hirano, F.Sato, "Recommendation of All-Electron Calculation in the Theoretical Study of Polypeptides", 46th Japanese peptide symp. proc., (2009).
- H.Yukawa, T.HIirano, Y.Nishimura, F.Sato, "High Performance Computing of Precise Electrostatic Potential on Proteins by GPU", Seisan Kenkyu, 61, 103 (2009).
- K.Chiba, T.Hirano, F.Sato, M.Okamoto, “A Density Functional Study on Reaction Center Models of Horse Carbonmonoxy Myoglobin – Effect of distal histidine to the electronic states -”, J. Comput. Chem. Jpn. (2008).
- Y.Tokita, J.Shimura, H.Nakajima, "Mechanism of Intramolecular Electron Transfer in the Photoexcited Zn-Substituted Cytochrome c: Theoretical and Experimental Perspective", J. Am. Chem. Soc., 130, 5302–5310 (2008).
- T.Inaba, N.Tsunekawa, T.Hirano, T.Yoshihiro, H.Kashiwagi, F.Sato, "Density functional calculation of the electronic structure on insulin hexamer", Chem. Phys. Lett., 434, 331-335 (2007).
- N.Nishino-Uemura, T.Hirano, F.Sato, “Study of the quasi-canonical localized orbital method based on protein structures”, J. Chem. Phys., 127, 184106 (2007).
- T.Inaba, S.Tahara, N.Nisikawa, H.Kashiwagi, F.Sato, "All-electron density functional calculation on insulin with quasi-canonical localized orbitals.", J. Comput. Chem., 26, 987-93 (2005).

If you know the other unlisted articles using the ProteinDF, please let me know.

## License

ProteinDF is licensed under the GNU GPL v3.
The source code can be found on the Github.

## Bugs

If you find any bugs, please let me know.
And if you have a suggestion for improvement, please let me know.


text:
# CppTemplateProject
Template for C++ projects

## App

`App` (main project) can be compiled from both VisualStudio (`.vcxproj`) and `build.sh` script.

## AppTest (test project)

`AppTest` (test project) requires `googletest` and (for now) it's only configured to be compiled using VisualStudio.

Existing settings expect `googletest` to be located at `C:/External/gtest`

A compiled version of it can be found [here](https://github.com/eduherminio/googletest). Cloning it into mentioned path should be enough to start working

repository: eduherminio/CppTemplateProject

repo_name: eduherminio/CppTemplateProject


text:
# izlude
Ansible playbooks for building simple private cloud services

## Included tools

- [Linx Server](https://github.com/andreimarcu/linx-server) - Pastebin service - `paste.yourdomain.tld`
- ... (to be added)

## Installation

```
# switch your virtualenv as needed
pip install -r requirements.txt
ansible-playbook server.yml
```

### Testing on Vagrant

```
vagrant up
```

## References

- [Awesome Selfhosted](https://github.com/Kickball/awesome-selfhosted)
- [Sovereign Playbooks](https://github.com/sovereign/sovereign)

repository: akhy/izlude

repo_name: akhy/izlude


text:
# SensorML Registry and Repository (SRR)
The primary interface of the SensorML Registry and Repository (SRR) as deployed on https://xdomes.tamucc.edu/srr/.

## Direct Access of a Record

The SensorML files records can be accessed directly using the following syntax:

https://xdomes.org/srr/sensorML.php?urn={enter the URN of the record on file}

*Example:*

https://xdomes.org/srr/sensorML.php?urn=urn:whoi:mvco:mvco_workhorse_1200

Alternatively, if the direct filename is available (the ':' characters are replaced with '-'), a direct HTTP call (i.e. API-less) can also be used to access the SensorML file.

*Example:*

https://xdomes.org/srr/sensorML/urn-whoi-mvco-mvco_workhorse_1200.xml

## Query the Registry

The SensorML files in the registry can be queries using the following syntax:

https://xdomes.org/srr/sensorML.php?query={enter query term}&attribute={enter attribute to search (e.g. description)}

*Example:*

https://xdomes.org/srr/sensorML.php?query=RD&attribute=description

repository: xdomes/srr

repo_name: xdomes/srr


text:
# klog wrapper of zap

inspired by [istio/klog](https://github.com/istio/klog).

## usage

### import

import this package by `github.com/xial-thu/klog`

### initialization

for example, in main.go

```golang
// Unchanged
klog.InitFlags(nil)
flag.StringVar(&kubeconfig, "kubeconfig", "", "path to Kubernetes config file")
flag.Parse()

// Append a new API
klog.Singleton()
```

If `Singleton()` is not called, the default global no-ops logger will work, which means you are not able to see any real log.

Due to some gaps between klog and zap, parameters shall be converted, and the conversion must be done after `flag.Parse()`. `klog.Singleton()` inits an unique global logger whose configuration is slightly different from default zap production configuration at:

1. `Timekey` is set to "time"
2. `EncodeTime` is set to `ISO8601TimeEncoder`

### flags

Not all flags defined in klog is supported, or rather say, not all the flags still make sense. Only `alsologtostderr` and `v` is supported currently.

* `v`: still supports `klog.V(2).Info()` syntax. But the level of `klog.Info()` is **INFO**; that `klog.V(3).Info()` is **DEBUG**. If v is set to zero, zap **DEBUG** log will be ignored. Default to **0**
* `alsologtostderr`: default to true. If set to false, INFO and DEBUG log will only output to stdout

### structured logging

There're 3 APIs:

* `With()`: parse each field and value from input. `WithFields(struct{A string}{"hi"})` will output `"A":"hi"`. If you care the fields in your struct and hope to extract them, use `With()`
* `WithAll()`: sugar of `zap.Any()`. e.g. `WithFields(struct{A string}{"hi"})` will output `"":{"A":"hi"}`. If you want to record the name of your struct, use `WithAll()`
* `WithFields()`: e.g. `WithFields("ID", 1, "name": "hi")`, just another sugar of `sugar.With()`

Tips of `With()`:

1. Only struct or map will be accepted
2. If arg is a struct, only exported field(which means "FieldName", not "fieldname") will be logged
3. If arg is a map, only accept maps whose key is string
4. If you have nested structs or maps, consider spliting them into several anomony parts
5. If you don't want some fields to be logged automatically, unexport them

Some examples of `With()`:

```golang
type S struct {
	A int
	B string
}
type Q struct {
	D ID
}
s := S{
	A: 10,
	B: "abc",
}
q := Q{
	D: ID(1),
}
c := ""

// struct args
With(s).Info(c) // "A":10,"B":"abc"
With(s, q).Info(c) // "A":10,"B":"abc","D":1
// anomony
With(struct {
	A int
	B int
}{1, 2}).Info(c) // "A":1,"B":2
// map args
With(map[string]int{"A":1}).Info(c) // "A":1
```

### performance of `With()`

|  case | ns/op  | B/op | allocs/op |
|  ---- | ----  | ---- | ---- |
| With every time | 3121 |2884 | 20|
| WithField every time | 2239 |1568|9|
|WithAll|3215|2642|14|
|With once|573|7|1|

Due to reflect, `With()` is slow. It indicates us that it's better to write like this instead of parsing interfaces every time.

```golang
newLogger := klog.With(something)
newLogger.Info(something)
```

## limitation

1. default field is empty.
2. structured logging still needs modification in business logic.
3. support of flags needs more work.

repository: xial-thu/klog

gha_language: Go

repo_name: xial-thu/klog


text:
# Caching

## Why needs caching

- Hard disk or SSD read/write speed is slow
- Read heavy applications/services need a mechanism to speed up
- Read heavy applications/services need a mechanism to reduce load to backend
- Use in memory cache layer to store subset of data (capacity trade off performance)

When local caches are used, they only benefit the local application consuming the data. In a distributed caching
environment, the data can span multiple cache servers and be stored in a central location for the benefit of all the
consumers of that data.

There are different levels of caching:

- Client side caching: browser cache, OS cache
- CDN caching: used to cache static files, like images, videos, etc.
- Database caching: used to increase the data retrieval performance
- Caching layer between client and datastore: in-memory key-value store like `Redis` and `Memcached`

## Cache write policies

Reference: [Open CAS cache mode](https://open-cas.github.io/cache_configuration.html#cache-mode)

- [Write-through Cache](https://www.youtube.com/watch?v=ptFn7f_SgSM&ab_channel=ASmallBug)
  - In Write-Through mode, the cache engine writes data to the cache storage and simultaneously writes the same data
    “through” to the backend storage

- [Write-around Cache](https://www.youtube.com/watch?v=mA5D48POAww&ab_channel=ASmallBug),
  the same idea of [look-aside](#look-aside) mentioned above.
  - In Write-Around mode, the client writes data to the cache if and only if that cache line is already mapped into the
    cache and simultaneously writes the same data “through” to the backend storage. If the key does not exist in the cache,
    the writes will only happen to backend storage.
  - Read from cache first
    - if cache hits then return the value to client
    - if cache misses then read from the db. Return the data back to client, update cache simultaneously

- [Write-back Cache](https://www.youtube.com/watch?v=-ucqTc1eDuI&ab_channel=ASmallBug) (risky of data loss)
  - The cache engine writes the data first to the cache storage and acknowledges to the application that the write is
    completed before the data is written to the backend storage
  - The data is written back to DB when the key is evicted in cache

- [Write-invalidate Cache](https://open-cas.github.io/cache_configuration.html#write-invalidate)

## Types of cache architecture

### Look-aside

![look-aside-cache](resources/look-aside-cache.png)

#### Look-aside On read

- client first checks the cache
- if cache-hit, cache returns the result
- if cache-miss, client gets the value from storage, then updates cache(Write-around)

---
Pros:

- client handles the logic, storage and cache are decoupled
- if cache server fails, client still work

Cons:

- Performance on cache-miss is low, three trips are involved
- Adding new cache server will cause surge traffic to backend
  - [Cold start solution from "memcached at Facebook" paper](https://youtu.be/Myp8z0ybdzM?t=3239). The idea is to have a
    flag to mark the new cache server/clusters as cold, so that reads will be redirected to some warm cache/clusters to
    get the data instead of going to db directly. Once the cold cache gets the data, it will update as well.
  - Routing tier does not route request to it until it catches up

#### Look-aside On write

- client writes to storage
- client deletes the entry in cache(Write-invalidate)

---
Pros:

- client deletes the entry in cache could make sure the data consistency

Cons:

- Extra step to deletes the entry could cause some performance issue on write

### Look-through

#### Look-through On read

![look-through-cache](resources/look-through-cache.png)

- client first checks the cache
- if cache-hit, cache returns the result
- if cache-miss, cache gets the value from storage, then updates itself

---
Pros:

- client's logic is easy to be implemented, it does not need to handle the logic of cache-miss

Cons:

- cache needs to hanle the cache-miss logic now, and to expose generic API for both upstream client and downstream storage
- the cache miss performance is still low

#### Look-through On write

##### Sync writes

![look-through-cache-sync-write](resources/look-through-cache-sync-write.png)

- client writes to cache
- cache writes to storage in sync

---
Pros:

- client's logic is easy to be implemented

Cons:

- writes in sync has low performance
- not all writes need to be stored in cache (not the hot key)
- having cache layer to handle the distributed concurrent writes is complicated
- usually caching layer is deployed seperately in a different cluster, writes to cache increases the latency
- complicated error handling if db writes fail

##### Async writes

![look-through-cache-async-write](resources/look-through-cache-asyc-write.png)

---
Pros:

- It probably has a better performance comparing with sync-write

Cons:

- not all writes need to be stored in cache (not the hot key)
- way too complicated !!!

## Granularity of caching key

We know that the caching usually is a key-value store, and what data to be stored in cache is case by case. Usually
there are two types I could think of:

- **Hash of the queries as the caching key**: The value could be a result of complicated query with filters
  - Pros:
    - Delegate the computation of the query to data store layer
    - Application just tracks the static query
  - Cons:
    - If a small piece of data is updated, there could be multiple queries need to be invalidated. And how to track
      which queries are impacted ?

- **Hash of the object as the caching key (mirror the subset of data store)**: If the data store is key-value already,
  we could cache a subset of those key-value pairs
  - Pros:
    - cache invalidation is easy, since we know which key is updated and where to find it in cache
  - Cons:
    - Application needs to handle the computation of the final query result, e.g. filtering.

## How cache eviction work

- LRU: Evict the least recently used item. E.g. Netflix video caching
- MRU: Evict the most recently used item. E.g. Tinder does not want to show the most recent items.
- LFU: Evict the least frequently used item. E.g. Phone typing prediction

## Distributed cache

### How distributed caching(hash table) works

Having the local cache cluster within each region. And have it replicated to all other regions.

#### Netflix EVCache Reading

![reading](resources/distributed-cache-netflix-reading.png)

- Read from the local cache first, if cache-miss then read from another cluster in different region

#### Netflix EVCache writing

![writing](resources/distributed-cache-netflix-writing.png)

- Write to all regions

#### Facebook Memcache Reading

![reading](resources/distributed-cache-fb-read.png)

- There is one region is master region which holds the master databases for both reads and writes. Replica regions are
  read only
- If read from replicate region and cache miss
  - If `isUpToDate==true`, then read from local database. Because it means that local database has
  replicated the latest data from master region
  - If `isUpToDate==false`, then redirect the reads to master region to read from the cache first. If still cache miss
  then read from the database in master region

#### Facebook Memcache Writing

##### Write from master region

- Write to the database
- Invalidate the key in all cache replicas by using `mcsquel`. We want the data to be replicated then invalidate the cache
  in non-master region. Because that if we invalidate the cache first while the data has not yet been available, any
  subsequent queries will cache-miss and read the stale data from local database then update the cache.(It causes data
  inconsistency)

![mcsquel-pipeline](resources/mcsqueal-pipeline.png)

##### Write from non-master region

- Invalidate cache in local cluster
- Write data to master database
- Let the `mcsquel` to handle the cache invalidation broadcast

### How distributed cache replica cross region

#### Netflix cross region replication

![cross-region-replication-netflix](resources/cross-region-replication-netflix.png)

- Application mutate the cache data
- Application will send the mutation metadata to `Kafka` as producer
- An agent called replication relay poll the message from `Kafka` or get the data from cache directly
- The agent sends the data to remote replication proxy via `https`
- The remote replication proxy in turns mutate the cache

### How to scale cache

#### To have more shards

Not all keys reside in a single node of cache cluster, keys are sharded into multiple nodes and each of the node holds
a subset of keys. Redis has 16384 hash slots, it uses `CRC16 of the key modulo 16384` to find the location of a key. When
a new shard is added with empty, we need to transfer keys from other nodes to the new nodes. Once the new shard catches
up we could clean up them from original nodes. Redis has to run `resharding` cmd manually to move keys, we could also use
consistent hash ring to minimize the data move.

#### To have more replicas of a particular shard

Adding a new replica of a shard is easy, we just need to copy data to it from the main/master shard. Once it catches up
we could allow that node to do consensus voting.

#### To put some cold data on SSD

![use-ssd-for-cache](resources/use-ssd-for-cache-netflix.png)

## How K8S client-go caching works

[How custom controller works](https://app.gitbook.com/@daniel-guo/s/notebook/k8s-related/extend-k8s/crds/custom-controllers)

![client-go-controller-interaction](resources/client-go-controller-interaction.png)

There could be multiple controllers reconcile a same set of resources, it would be a huge load if all controllers talk
to api server to ask for the state of resources. So caching is really important.

The [`Thread Safe Store`](https://github.com/kubernetes/client-go/blob/fb61a7c88cb9f599363919a34b7c54a605455ffc/tools/cache/thread_safe_store.go)
is the cache where [informer](https://github.com/kubernetes/client-go/blob/fb61a7c88cb9f599363919a34b7c54a605455ffc/tools/cache/controller.go#L371)
will add the object.

## Comparision between Redis and Memcached

- `redis` provides a superset of features comparing to `memcached`
- To implement `memcached` cluster, client needs to solve traffic rounting, e.g. using consistent caching
- They have different implementations on memory management

More details could be found from the links below in references section.

## References

- [What is caching (from medium)](https://medium.com/system-design-blog/what-is-caching-1492abb92143)
- [Caching overview (by AWS)](https://aws.amazon.com/caching/)
- [Everything you need to know about caching](https://levelup.gitconnected.com/everything-you-need-to-know-about-caching-system-design-932a6bdf3334)
- [Redis vs Memcached (by AWS)](https://aws.amazon.com/elasticache/redis-vs-memcached/)
- [Redis vs Memcached (by Alibaba Cloud)](https://alibaba-cloud.medium.com/redis-vs-memcached-in-memory-data-storage-systems-3395279b0941)
- [Cache Consistency: Memcached at Facebook (MIT Lecture)](https://www.youtube.com/watch?v=Myp8z0ybdzM&ab_channel=MIT6.824%3ADistributedSystems)
- [Paper: Scaling memcache at Facebook](resources/memcache-fb.pdf)
- [Youtube: Caching at Netflix](https://www.youtube.com/watch?v=Rzdxgx3RC0Q&ab_channel=StrangeLoop)
- [Medium Netflix blog](https://netflixtechblog.com/ephemeral-volatile-caching-in-the-cloud-8eba7b124589)

repository: leoleochi/tech-notes

repo_name: leoleochi/tech-notes


text:
---
title: Using Xamarin and NFC for IOT devices.
date: 2018/12/14 01:01:01
---

One of the largest challenges when we are talking about IOT it is the cpability of giving maitenance of the devices. Most of the time this can be done over the air or the devices come in pre programed from the manufacturer. But when that is not the case havinv  device that can accpet nfc as a way to program it can be really useful.

This blog post is to explain how can you use Xamarin and IOThub to build an app that can reprogram those devices and provision thousands of them without the need to pay extra for the manufactured to built in your specifications on it.

The full source code demonstrated here can be found at [Elsys Xamarin deployer](http://www.com)

<!-- more -->

## Architecture and deployment

The idea it is really simple we are going to have a bunch of devices comminucating through LoraWan to a gateway that will be passing this reading to IOTHub you can find the code used by Azure Functions to decode this messages in our case at: [Functions provisioner decoder][1].

<!-- Drawning of the architecture -->

In order to setup our devices when we are in the fild we are going to place the device them tap it with a phone. after the device has been tapped it will try to joing our network and start sending data.

## Building the app

First step it is to create a xamaring app we are going to be using the Monodroid solution since Apple does not expose their [NFC for third party developers][2].

After creatign our one page app we need to add NFC to it so you need to include in your manifext file.

repository: Nepomuceno/blog

gha_language: CSS

repo_name: Nepomuceno/blog


text:
BaseAdminBundle
=====================

**Getting started**
| [CrudController](CrudController.md#BaseAdminBundle)
| [Router](Router.md#BaseAdminBundle)
| [AbstractVoter](AbstractVoter.md#BaseAdminBundle)
| [AbstractDatatable](AbstractDatatable.md#BaseAdminBundle)
| [TwigComponents](TwigComponents.md#BaseAdminBundle)

> ## Note: this bundle is under development and lots of changes may be still going on.

## Getting Started

Run `composer require k3ssen/base-admin:dev-master --dev` in your console.

Symfony Flex should add the bundle automatically to your `config/bundles.php`.

If installation fails due to minumum-stability, you could add the
following settings to your composer.json file first:

    "minimum-stability": "dev",
    "prefer-stable": true

### Quick start with example template

If you're creating a new project, you could use the following line
in your `templates/base.html.twig`:

    {% extends<EMAIL_ADDRESS>%}

This layout contains the basics for what is needed in the crud application.

Most certainly you should use your own layout, but it can be helpful to
get started with some basis if you don't have a layout ready yet.

### Required/Recommended bundles

No other bundles are strictly required for using this bundle, but to
take full advantage of this bundle, you may need
the following bundles:

- [DatatablesBundle](https://github.com/stwe/DatatablesBundle)
for datatables.
The `AbstractDatatable` needs this bundle to work.
- [StofDoctrineExtensionsBundle](http://symfony.com/doc/master/bundles/StofDoctrineExtensionsBundle/index.html).
The `CrudController` takes SoftDeleteable into account. It won't break
without, so it's fully optional to use this bundle.
- [ExtendedGeneratorBundle](https://github.com/k3ssen/ExtendedGeneratorBundle)
The ExtendedGeneratorBundle is built together with this bundle. It
lets you generate files for quickly creating your application.
Using the ExtendedGeneratorBundle is optional.


text:
# Chapter 4 Exercises

This repository contains programming exercises for formatting documents in Markdown,
based on Chapter 4 of [_Programming Skills for Data Science_](https://programming-for-data-science.github.io/).

Solutions can be found in the `solution` branch.

repository: sparkoparko/book-exercises

repo_name: sparkoparko/book-exercises


text:
---
title: "Gatsby Build Automation"
tags:
  - gatsby
  - git
  - hooks
  - blog
date: "2018-12-01T04:01:00.000Z"
---

What I wanted to simplify for myself is the following process:
- write a blog post on Sublime Text on my laptop
- push it to the repository's master branch
- push it to the production server
- Gatsby builds all the static files and deploys it

Thanks to [DigitalOcean's guide on automating with Git hooks](https://www.digitalocean.com/community/tutorials/how-to-use-git-hooks-to-automate-development-and-deployment-tasks), I was able to set everything up the way I want it.

Several locations need to be established.
- My local machine:
  - `/home/localuser/git/blog/`
- My remote production server:
  - `/home/remoteuser/blog`
  - `/var/www/blog/prod`

### Nginx setup

On my remote machine, Nginx is installed and is statically serving my blog domain, `blog.jyntran.ca`.

If it's not already, you may need to go through the [installation process](https://www.nginx.com/resources/wiki/start/topics/tutorials/install/):

```
sudo apt-get update
sudo apt-get install nginx
```

Inside a new file called `/etc/nginx/sites-available/blog`:

```
server {
        listen 80;
        listen [::]:80;

        server_name blog.jyntran.ca;
        root /var/www/blog/prod/public;
        index index.html;
        ...
```

Notice how the root is going to the subdirectory `public`? It's the place where the static files, including `index.html` will reside after a build.

Now make a symlink to `sites-enabled`, check that your Nginx configurations are ok, then restart Nginx. Note: you may need `sudo` for these commands.

```
ln -s /etc/nginx/sites-available/blog /etc/nginx/sites-enabled/blog
nginx -t
service nginx reload
```

### Production directory

I set up a space for the production blog at `/var/www/blog`.

Inside it I set up a directory where the push happens.

```
cd /var/www/blog
mkdir prod
cd prod
```

Make sure the user has proper privileges to write to the folder:

```
sudo chown -R `whoami`:`id -gn` /var/www/blog/prod
```

### Git hook

Now go to the home directory, and create a directory for the blog's Git hooks will be.

```
cd ~/blog
git init --bare
```

With the `--bare` argument, this creates a Git instance with no working directory. Since my working directory is on my local machine, this is fine with me.

Now, for the creation of the hook itself:

```
vim hooks/post-receive
```

And the contents, mostly ripped from the guide above:
```
#!/bin/bash

while read oldrev newrev ref
  do
    # ref = refs/head/master
    if [[ $ref =~ .*/master$ ]]; then
      echo "Master ref received. Deploying master branch to production..."
      git --work-tree=/var/www/blog/prod --git-dir=/home/remoteuser/blog checkout -f
      cd /var/www/blog/prod
      yarn install && gatsby build
    elseh
      echo "Ref $ref successfully received. Doing nothing: only the master branch may be deployed on this server."
    fi
  done
```

What's happening here? Read in the received data: if the ref is referring to the master branch, checkout the repo in the production folder, and once in there, use Yarn to install any packages that's changed, and use Gatsby to build.

### Local setup

Now to set up the client side, that is, my local machine.

I have my remote origin set up to my GitHub repository. We need to do the same for the production repository.

Let's go to the working directory:

```
cd /home/localuser/git/blog
```

Before running these commands, make sure you replace some of these values: `remoteuser` and `server_domain_or_IP`, and `blog` if your home subfolder is named something else.

For pushing to both GitHub and to production:

```
git remote set-url --add --push origin git@github.com:jyntran/gatsby-blog.git
git remote set-url --add --push origin remoteuser@server_domain_or_IP:blog

```

Check them with the flag verbose:

```
git remote -v
```

And the result:

```
origin	git@github.com:jyntran/gatsby-blog.git (fetch)
origin	git@github.com:jyntran/gatsby-blog.git (push)
origin	remoteuser@server_domain_or_IP:blog (push)
```

### The final product

Now to test the push!

```
git push origin master
```

You might be prompted for the password of your production server if SSH keys are not set up. If you read the message that returns, you will now see the client message from the `post-receive` hook, as well as output from `yarn install` and `gatsby build`:

```
Counting objects: 6, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (6/6), 3.01 KiB | 0 bytes/s, done.
Total 6 (delta 3), reused 0 (delta 0)
remote: Resolving deltas: 100% (3/3), completed with 3 local objects.
To git@github.com:jyntran/gatsby-blog.git
 + a0c5a49...08aa6cb master -> master
remoteuser@server's password:
Counting objects: 6, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (6/6), 3.01 KiB | 0 bytes/s, done.
Total 6 (delta 3), reused 0 (delta 0)
remote: Master ref received. Deploying master branch to production...
remote: yarn install v1.12.3
remote: success Already up-to-date.
remote: Done in 1.59s.
remote: Running gatsby build...
remote: success open and validate gatsby-configs — 0.013 s
remote: success load plugins — 0.377 s
remote: success onPreInit — 0.375 s
remote: success delete html and css files from previous builds — 0.076 s
remote: success initialize cache — 0.010 s
remote: success copy gatsby files — 0.085 s
remote: success onPreBootstrap — 0.021 s
remote: success source and transform nodes — 0.218 s
remote: success building schema — 0.656 s
remote: success createPages — 0.113 s
remote: success createPagesStatefully — 0.046 s
remote: success onPreExtractQueries — 0.000 s
remote: success update schema — 0.243 s
remote: success extract queries from components — 0.093 s
remote: success run graphql queries — 0.194 s — 6/6 31.22 queries/second
remote: success write out page data — 0.006 s
remote: success write out redirect data — 0.001 s
remote: success onPostBootstrap — 0.001 s
remote:
remote: info bootstrap finished - 5.049 s
remote:
remote: success Building production JavaScript and CSS bundles — 13.028 s
remote: success Building static HTML for pages — 2.542 s — 32/32 49.69 pages/second
remote: info Done building in 20.632 sec
To remoteuser@server:blog
 + a0c5a49...08aa6cb master -> master
```

Now that Gatsby has finished the build, the `/var/www/blog/prod/public` directory is populated with the static files for Nginx to find.

Finally, it's all set - an automated Gatsby build process for the production server. Hurray!

repository: jyntran/gatsby-blog

repo_name: jyntran/gatsby-blog


text:
---
layout: default
---

# 1994-02-02 - system logging

## Header Data

From<EMAIL_ADDRESS>(PGP Slave Key Server)

To<EMAIL_ADDRESS>Message Hash: a15d9aa81ddd2f60155650e26de452a924c3d1751c627221dc14eac5c8edecba

Message ID<EMAIL_ADDRESS>Reply To: _N/A_

UTC Datetime: 1994-02-02 18:20:56 UTC

Raw Date: Wed, 2 Feb 94 10:20:56 PST

## Raw message

```
{% raw  %}From<EMAIL_ADDRESS>(PGP Slave Key Server)
Date: Wed, 2 Feb 94 10:20:56 PST
To<EMAIL_ADDRESS>Subject: system logging
Message-ID<EMAIL_ADDRESS>MIME-Version: 1.0
Content-Type: text/plain

> Greetings.
>
> New remailer<EMAIL_ADDRESS>>
> No logs. Only a "counter" that works by appending the word "R" or "ER" to a
> text file so I can get an idea if anyone is using it. However, I'm sure the
> Netcom and other site's mail logs will be enough to track serious abusers
> of anonymity down, without my help. This remailer is dedicated to honest
> people who desire PRIVACY.

People should be aware that whether Niko makes personal logs on his
qwerty account or not, the public logs on netcom show more than enough
info to trivially track people down.

By the way it's very bad practice to forge From: lines, especially
with completely non-existant site names like qwerty.org...perhaps
you should ask netcom to register it for you.  Or if they charge real
money for it, your postmaster at columbia.edu might do it for free if
you asked him nicely.

{% endraw %}
```

## Thread

+ Return to [February 1994](/archive/1994/02)

+ Return to<EMAIL_ADDRESS>(PGP Slave Key Server)](/authors/pgpkeys_at_wasabi_io_com_pgp_slave_key_server_)"

+ 1994-02-02 (Wed, 2 Feb 94 10:20:56 PST) - system logging -<EMAIL_ADDRESS>(PGP Slave Key Server)_


text:
The datasets are shown in this directory. Limited to the upload size of cmt3, they contain 1% of all data for example.

Each dataset contains 6 numpy file, including train.npy, test.npy, train_label.npy, test_label.npy, train_complexity and test_complexity. train.npy and test.npy are the data in training set and testing set. train_label.npy and test_label.npy are the label for training set and testing set. train_compexity.npy and test_complexity is the file size of data in training set and testing set, compressed by distinct compressor: PNG, JPEG and FLIF.

| Dataset      | Original size | Reshape Size | Num. Class | Num. Image |
| ------------ | ------------- | ------------ | ---------- | ---------- |
| Constant     | 32x32x3       | 32x32x3      | 10         | 70000      |
| Constant28   | 28x28x1       | 28x28x1      | 10         | 70000      |
| Noise        | 32x32x3       | 32x32x3      | 10         | 70000      |
| Noise28      | 28x28x1       | 28x28x1      | 10         | 70000      |
| Omniglot     | 105x105x1     | 28x28x1      | 55         | 32415      |
| MNIST        | 28x28x1       | 28x28x1      | 10         | 60000      |
| FashionMNIST | 28x28x1       | 28x28x1      | 10         | 60000      |
| KMNIST       | 28x28x1       | 28x28x1      | 10         | 60000      |
| NOTMNIST     | 28x28x1       | 28x28x1      | 10         | 60000      |
| CIFAR-10     | 32x32x3       | 32x32x3      | 10         | 60000      |
| CIFAR-100    | 32x32x3       | 32x32x3      | 100        | 60000      |
| SVHN         | 32x32x3       | 32x32x3      | 10         | 99289      |
| CelebA       | 178x218x3     | 32x32x3      | 10         | 182732     |
| TinyImageNet | 64x64x3       | 32x32x3      | 200        | 100,000    |
| LSUN         | 64x64x3       | 32x32x3      | 10         | 10000      |
| iSUN         | 64x64x3       | 32x32x3      | 10         | 8925       |

repository: chenwenxiao/DOI

repo_name: chenwenxiao/DOI


text:
## rosa download openshift-client

Download OpenShift client tools

### Synopsis

Downloads to latest compatible version of the OpenShift client tools.

```
rosa download openshift-client [flags]
```

### Examples

```
  # Download oc client tools
  rosa download oc
```

### Options

```
  -h, --help   help for openshift-client
```

### Options inherited from parent commands

```
      --debug            Enable debug mode.
      --profile string   Use a specific AWS profile from your credential file.
  -v, --v Level          log level for V logs
```

### SEE ALSO

* [rosa download](rosa_download.md)	 - Download necessary tools for using your cluster

repository: mikenairn/rosa

repo_name: mikenairn/rosa


text:
---
date: '2011-03-09'
title: SXSW 2011 Intro
---

[SXSW][1] is right around the corner and anyone who
has ever been before knows that they spend an inordinate amount of time getting
ready. Probably a considerable amount of time more, than let's say, Keisha
trying to figure out how to spell her name at the DMV.  Anyway, sudo-funny dude [Alex Blagg][1] has compiled this video to
help you out:

<iframe title="YouTube video player" width="550" height="390" src="http://www.youtube.com/embed/hx3FC_DWsGQ" frameborder="0" allowfullscreen></iframe>

[1]: http://sxsw.com/

repository: braidn/blrrrg

gha_language: Liquid

repo_name: braidn/blrrrg


text:
## 7.4-1

> Show that in the recurrence
>
> $$
> \begin{aligned}
> T(n) & = \max\limits_{0 \le q \le n - 1} (T(q) + T(n - q - 1)) + \Theta(n), \\\\
> T(n) & = \Omega(n^2).
> \end{aligned}
> $$

We guess $T(n) \ge cn^2 - 2n$,

$$
\begin{aligned}
T(n) & =   \max_{0 \le q \le n - 1} (T(q) + T(n - q - 1)) + \Theta(n) \\\\
     & \ge \max_{0 \le q \le n - 1} (cq^2 - 2q + c(n - q - 1)^2 - 2n - 2q -1) + \Theta(n) \\\\
     & \ge c\max_{0 \le q \le n - 1} (q^2 + (n - q - 1)^2 - (2n + 4q + 1) / c) + \Theta(n) \\\\
     & \ge cn^2 - c(2n - 1) + \Theta(n) \\\\
     & \ge cn^2 - 2cn + 2c & (c \le 1) \\\\
     & \ge cn^2 - 2n.
\end{aligned}
$$

## 7.4-2

> Show that quicksort's best-case running time is $\Omega(n\lg n)$.

We'll use the substitution method to show that the best-case running time is $\Omega(n\lg n)$. Let $T(n)$ be the best-case time for the procedure $\text{QUICKSORT}$ on an input of size $n$. We have

$$T(n) = \min _{1 \le q \le n - 1} (T(q) + T(n - q - 1)) + \Theta(n).$$

Suppose that $T(n) \ge c(n\lg n + 2n)$ for some constant $c$. Substituting this guess into the recurrence gives

$$
\begin{aligned}
T(n) & \ge \min _{1 \le q \le n - 1}(cq\lg q + 2cq + c(n - q - 1) \lg(n - q - 1) + 2c(n - q - 1)) + \Theta(n) \\\\
     & =   (cn / 2)\lg(n / 2) + cn + c(n / 2 - 1)\lg(n / 2 - 1) + cn - 2c + \Theta(n) \\\\
     & \ge (cn / 2)\lg n - cn / 2 + c(n / 2 - 1)(\lg n - 2) + 2cn - 2c\Theta(n) \\\\
     & =   (cn / 2)\lg n - cn / 2 + (cn / 2) \lg n - cn - c\lg n + 2c + 2cn - 2c\Theta(n) \\\\
     & =   cn\lg n + cn / 2 - c\lg n + 2c - 2c\Theta(n).
\end{aligned}
$$

Taking a derivative with respect to $q$ shows that the minimum is obtained when $q = n / 2$. Taking $c$ large enough to dominate the $−\lg n + 2 − 2c + \Theta(n)$ term makes this greater than $cn\lg n$, proving the bound.

## 7.4-3

> Show that the expression $q^2 + (n - q - 1)^2$ achieves a maximum over $q = 0, 1, \ldots, n - 1$ when $q = 0$ and $q = n - 1$.

$$
\begin{aligned}
  f(q) & = q^2 + (n - q - 1)^2 \\\\
 f'(q) & = 2q - 2(n - q - 1) = 4q - 2n + 2 \\\\
f''(q) & = 4. \\\\
\end{aligned}
$$

$f'(q) = 0$ when $q = \frac{1}{2}n - \frac{1}{2}$. $f'(q)$ is also continious. $\forall q: f''(q) > 0$, which means that $f'(q)$ is negative left of $f'(q) = 0$ and positive right of it, which means that this is a local minima. In this case, $f(q)$ is decreasing in the beginning of the interval and increasing in the end, which means that those two points are the only candidates for a maximum in the interval.

$$
\begin{aligned}
    f(0) & = (n - 1)^2 \\\\
f(n - 1) & = (n - 1)^2 + 0^2.
\end{aligned}
$$

## 7.4-4

> Show that $\text{RANDOMIZED-QUICKSORT}$'s expected running time is $\Omega(n\lg n)$.

We use the same reasoning for the expected number of comparisons, we just take in a different direction.

$$
\begin{aligned}
\text E[X]
    & =   \sum_{i = 1}^{n - 1} \sum_{j = i + 1}^n \frac{2}{j - i + 1} \\\\
    & =   \sum_{i = 1}^{n - 1} \sum_{k = 1}^{n - i} \frac{2}{k + 1} & (k \ge 1) \\\\
    & \ge \sum_{i = 1}^{n - 1} \sum_{k = 1}^{n - i} \frac{2}{2k} \\\\
    & \ge \sum_{i = 1}^{n - 1} \Omega(\lg n) \\\\
    & =   \Omega(n\lg n).
\end{aligned}
$$

Using the master method, we get the solution $\Theta(n\lg n)$.

## 7.4-5

> We can improve the running time of quicksort in practice by taking advantage of the fast running time of insertion sort when its input is "nearly" sorted. Upon calling quicksort on a subarray with fewer than $k$ elements, let it simply return without sorting the subarray. After the top-level call to quicksort returns, run insertion sort on the entire array to finish the sorting process. Argue that this sorting algorithm runs in $O(nk + n\lg(n / k))$ expected time. How should we pick $k$, both in theory and practice?

In the quicksort part of the proposed algorithm, the recursion stops at level $\lg(n / k)$, which makes the expected running time $O(n\lg(n / k))$. However, this leaves $n / k$ non-sorted, non - intersecting subarrays of (maximum) length $k$.

Because of the nature of the insertion sort algorithm, it will first sort fully one such subarray before consider the next one. Thus, it has the same complexity as sorting each of those arrays, that is $\frac{n}{k}O(k^2) = O(nk)$.

In theory, if we ignore the constant factors, we need to solve

$$
\begin{aligned}
            & n\lg n \ge nk + n\lg{n / k} \\\\
\Rightarrow & \lg n \ge k + \lg n - \lg k \\\\
\Rightarrow & \lg k \ge k.
\end{aligned}
$$

Which is not possible.

If we add the constant factors, we get

$$
\begin{aligned}
            & c_qn\lg n \ge c_ink + c_qn\lg(n / k) \\\\
\Rightarrow & c_q\lg n \ge c_ik + c_q\lg n - c_q\lg k \\\\
\Rightarrow & \lg k \ge \frac{c_i}{c_q}k.
\end{aligned}
$$

Which indicates that there might be a good candidate. Furthermore, the lower-order terms should be taken into consideration too.

In practice, $k$ should be chosed by experiment.

## 7.4-6 $\star$

> Consider modifying the $\text{PARTITION}$ procedure by randomly picking three elements from array $A$ and partitioning about their median (the middle value of the three elements). Approximate the probability of getting at worst an $\alpha$-to-$(1 - \alpha)$ split, as a function of $\alpha$ in the range $0 < \alpha < 1$.

First, for simplicity's sake, let's assume that we can pick the same element twice. Let's also assume that $0 < \alpha \le 1 / 2$.

In order to get such a split, two out of three elements need to be in the smallest $\alpha n$ elements. The probability of having one is $\alpha n / n = \alpha$. The probability of having exactly two is $\alpha^2 - \alpha^3$. There are three ways in which two elements can be in the smallest $\alpha n$ and one way in which all three can be in the smallest $\alpha n$ so the probability of getting such a median is $3\alpha^2 - 2\alpha^3$. We will get the same split if the median is in the largest $\alpha n$. Since the two events are mutually exclusive, the probability is

$$\Pr\\{\text{OK split}\\} = 6\alpha^2 - 4\alpha^3 = 2\alpha^2(3 - 2\alpha).$$

repository: walkccc/CLRS

gha_language: Markdown

repo_name: walkccc/CLRS


text:
% Nordic-RSE : I/O profiling and optimization
% Simo Tuomisto
% 26.10.2021

## Contents of this talk

- What kinds of IO problems occur in HPC environments?
- How do we spot them?
- How can they be solved?

## Disclaimer

- I have worked a lot with I/O problems, but don't take me for an authority on solving them.
  I suggest looking at what big players are doing.
- I/O frameworks move constantly, I might have missed some good tools.
  Let me know if you have good suggestions.

## Why I/O problems are important?

- Shared filesystems have to deal with increased load, which can result in various problems
- Job execution slows down as CPU/RAM/GPU waits for I/O
- Bad I/O hampers interactive usage and waiting for I/O is very annoying as an user as it does not feel productive

## What kinds of problems are there?

- Most common problems in my opinion (in no particular order):

. . .

1. "ab != ba"-problem

. . .

2. "Jenga"-problem

. . .

3. "I hope I didn't forget anything"-problem

. . .

4. "She'll have the steak"-problem

## "ab != ba"-problem

. . .

In many fields of mathematics, all operations do not have the commutative property.

E.g. for matrices usually: $$ AB \neq BA $$

. . .

How is this related to I/O?

## "ab != ba"-problem

- From coding perspective, both of these can produce the same results:

```python
for parameter in parameters:
    for datafile in datafiles:
        data = load_data(datafile)
        calculate_model(data, parameter)
```
vs.
```python
for datafile in datafiles:
    data = load_data(datafile)
    for parameter in parameters:
        calculate_model(data, parameter)
```

## "ab != ba"-problem

- From I/O perspective, the code is not commutative:

```python
for parameter in parameters:
    for datafile in datafiles:
        data = load_data(datafile) # <- I/O is multiplied
        calculate_model(data, parameter)
```
vs.
```python
for datafile in datafiles:
    data = load_data(datafile) # <- I/O is not multiplied
    for parameter in parameters:
        calculate_model(data, parameter)
```

## "ab != ba"-problem

- The problem might look trivial, but it is surprisingly hard to spot!
- Usually requires unraveling most of the workflow.
- When working with small data, this might not cause any problems.

## "ab != ba"-problem

- One way of spotting this is with `strace`:
  ```sh
  strace -c -e trace=%file,read,write ./command
  ```
- If the number of IO operations scales with the number of parameters, something might be amiss.

## "Jenga"-problem

. . .

- Closely related to the "ab != ba"-problem.
- First steps do not cause problems, but as time goes on the situation becomes more problematic due to constant I/O.

![Jenga](https://upload.wikimedia.org/wikipedia/commons/6/6b/Jenga_distorted.jpg?download){ width=150px }

[Source: Wikipedia](https://commons.wikimedia.org/w/index.php?curid=17999924)

## "Jenga"-problem

- I/O in machine learning is especially suspect to this problem as data will be iterated over multiple times
- Having a bad I/O pattern doesn't matter for one epoch, but as training can consist of thousands of epochs, the overall result is bad
- Bad I/O patterns can result in bad GPU utilization in deep learning

## "Jenga"-problem

- Using good data formats (e.g. [Parquet](https://parquet.apache.org/documentation/latest/), [webdataset](https://github.com/webdataset/webdataset)) and following best practice guides usually helps (e.g. [Keras](https://keras.io/getting_started/intro_to_keras_for_engineers/), [Tensorflow](https://www.tensorflow.org/guide/data_performance) and [PyTorch](https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html) have good guides)

## "Jenga"-problem

Quick demo:

https://github.com/simo-tuomisto/data-format-tests

## "I hope I didn't forget anything"-problem

. . .

- It is quite common practice to carry everything that was in the initial data throughout the whole analysis
- Reducing the data size would reduce the needed resources (space, ram, etc.)
- Usually having a mapping between initial data and processed data would be sufficient

## "I hope I didn't forget anything"-problem

- This is especially problematic when working remotely as transferring full datasets is slower than transferring increments
- Problem can be minimized when using data analysis libraries such as pandas that allow for easy data merging from multiple sources
- This is closely related to the "She'll have the steak"-problem

## "She'll have the steak"-problem

. . .

Quite often workflows happen in the following way:

. . .

  1. Data inspection tools are written to inspect data.
  2. Coding framework & data format is chosen based on how easy data is to **load** for inspection.
  3. Data needs to be used with a model / solver, but data format has already been chosen.
  4. Data fits to the model / solver poorly.

## "She'll have the steak"-problem

A better solution would be to look at the problem from the other end:

. . .

  1. Model / solver is chosen.
  2. Coding framework & data format is chosen based on how easy data is to **use** with a model.
  3. Data inspection tools are written based on the framework.
  4. Data fits well into the model / solver.

## "She'll have the steak"-problem

- Major frameworks usually have a preferred way of working with data (e.g.
  [tidy data](https://www.jstatsoft.org/index.php/jss/article/view/v059i10/v59i10.pdf) with Pandas,
  data loaders for deep learning toolkits, NetCDF for physics simulations, ...)

. . .

- Downside is that one might need to write visualization tools with respect to the data formats needed by these frameworks

. . .

- Upside is other people are doing the same thing: there are lots of already existing tools

## Profiling vs understanding

## Profiling vs understanding

- Quite often when talking about I/O we'll use terms such as `Megabytes per second` or `I/O operations per second`

. . .

- Profiling is good, but more often **understanding what we're trying to accomplish with the data** is more important

. . .

- Often we need to ask how the computer perceives what we're telling it to do

## Example in understanding a problem

Common problem in deep learning is related to **randomness**:

We want to randomize our data ordering for each epoch

## Example in understanding a problem

- This is problematic, as random access is much slower than sequential access
- To fix this, we give up some degree of randomness for increased efficiency

## Example in understanding a problem

Consider shuffling of a deck of playing cards:

![](graphs/playing-cards-1.svg)

## Example in understanding a problem

Instead of shuffling the deck, we split the deck into multiple batches:

![](graphs/playing-cards-2.svg)

## Example in understanding a problem

We shuffle the batches randomly:

![](graphs/playing-cards-3.svg)

## Example in understanding a problem

We shuffle the data within each batch:

![](graphs/playing-cards-4.svg)

## Example in understanding a problem

Is this ordering random enough?

![](graphs/playing-cards-5.svg)

. . .

More often than not, it is.

## Example in understanding a problem

- This kind of IO can be done sequentially.
- We still get randomization, but not **complete randomization**.
- Vast majority of big data analysis uses this.
- This was used in the demo as well.

## Conclusions

## Conclusions

- I/O problems are often tied to the way we're working
- To solve them, we need to look at our workflows
- Looking at the problem from the computer's point of view can help

repository: simo-tuomisto/data-format-tests

repo_name: simo-tuomisto/data-format-tests


text:
# NumPY
#### This assigment is to investigate the Numpy random package

The first part looks at the various functions within the random sampling section

Found it best to just take a function and running it to see what might be happening

There is some overlapp between the various function and that can present some confusion

Next looked at random and shuffle

Distribution functions: looked at a number of these
 did find I had to go back to school hugely

can see how powerful these functions are which is the kernal of the package, it is hugely powerful

Looked at the seed generator, again its about the testing and trying and playing around with the
 funciton and
 introducing elements that can show what this is about.

To run the note book

-  One of the requirements here is Python, either Python 3.3 or greater or Python 2.7. The general recommendation is that you use the Anaconda distribution to install both Python and the notebook application.

repository: theamurtagh/Numpy

repo_name: theamurtagh/Numpy


text:
[atom-bitcoin](https://atom.io/packages/atom-bitcoin)
============

####Watch the latest selling price for bitcoin while you code!

![](https://dl.dropboxusercontent.com/s/4lvxmytpicoz9o1/screenshot.png?dl=1&token_hash=AAEBoVYT77ey684_qqX5ltFAhUOfRdqmxjMUcnuUNxnFTA)

####To install

`` apm install atom-bitcoin `` or search for ``atom-bitcoin`` in Settings > Packages

This uses the [BitcoinAverage API](https://bitcoinaverage.com/api.htm), an open source project designed to provide weighted average bitcoin price calculation. It utilises all exchanges where price and volume data is available.

####Features

- Update interval defaults to 5 seconds but any second-based interval can be entered in the settings

- Has colour updates to show increase/decrease/stable in price

- Currency defaults to USD but any currency from the list below can be entered in the settings

  - AUD
  - BRL
  - CAD
  - CHF
  - CNY
  - EUR
  - GBP
  - HKD
  - ILS
  - JPY
  - NOK
  - NZD
  - PLN
  - RUB
  - SEK
  - SGD
  - TRY
  - USD
  - ZAR


text:
# Vanilla Policy Gradient (VPG)

If you wanted to explore Policy Gradient algorithms in RL, there is a high chance you would've heard of PPO, DDPG, etc. but understanding them can be tricky if you're just starting.

VPG is arguably one of the easiest to understand policy gradient algorithms while still performing to a good enough level.

Let's understand policy gradient at a high level, unlike the classical algorithms like Q-Learning, Monte Carlo where you try to optimise the outputs of the action-value function of the agent which are then used to determine the optimal policy. In policy gradient, as one would like to say we go directly for the kill shot, basically we optimise the thing we want to use at the end, i.e. the Policy.

So that explains the "Policy" part of Policy Gradient, so what about "Gradient", so gradient comes from the fact that we try to optimise the policy by gradient ascent (unlike the popular gradient descent, here we want to increase the values, hence ascent). So that explains the name, but how does it even work.

For that, have a look at the following Psuedo Code (source: [OpenAI](https://spinningup.openai.com))

![Psuedo Code](https://spinningup.openai.com/en/latest/_images/math/262538f3077a7be8ce89066abbab523575132996.svg)

For a more fundamental understanding [this](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html) spinningup article is a good resource

Now that we have an understanding of how VPG works at a high level let's jump into the code to see it in action\
This is a very minimal way to run a VPG agent on **GenRL**

### VPG agent on a Cartpole Environment

```python
import gym  # OpenAI Gym

from genrl import VPG
from genrl.deep.common import OnPolicyTrainer
from genrl.environments import VectorEnv

env = VectorEnv("CartPole-v1")
agent = VPG('mlp', env)
trainer = OnPolicyTrainer(agent, env, epochs=200)
trainer.train()
```

This will run a VPG agent `agent` which will interact with the `CartPole-v1` [gym environment](https://gym.openai.com/) \
Let's understand the output on running this (your individual values may differ),

```sh
timestep         Episode          loss             mean_reward
0                0                8.022            19.8835
20480            10               25.969           75.2941
40960            20               29.2478          144.2254
61440            30               25.5711          129.6203
81920            40               19.8718          96.6038
102400           50               19.2585          106.9452
122880           60               17.7781          99.9024
143360           70               23.6839          121.543
163840           80               24.4362          129.2114
184320           90               28.1183          156.3359
204800           100              26.6074          155.1515
225280           110              27.2012          178.8646
245760           120              26.4612          164.498
266240           130              22.8618          148.4058
286720           140              23.465           153.4082
307200           150              21.9764          151.1439
327680           160              22.445           151.1439
348160           170              22.9925          155.7414
368640           180              22.6605          165.1613
389120           190              23.4676          177.316
```

`timestep`: It is basically the units of time the agent has interacted with the environment since the start of training\
`Episode`: It is one complete rollout of the agent, to put it simply it is one complete run until the agent ends up winning or losing\
`loss`: The loss encountered in that episode\
`mean_reward`: The mean reward accumulated in that episode

Now if you look closely the agent will not converge to the max reward even if you increase the epochs to say 5000, it is because that during training the agent is behaving according to a stochastic policy (Meaning when you try to pick from an action given a state from the policy it doesn't simply take the one with the maximum return, rather it samples an action from a probability distribution, so in other words, the policy isn't just like a lookup table, it's function which outputs a probability distribution over the actions which we sample from when using it to pick our optimal action).\
So even if the agent has figured out the optimal policy it is not taking the most optimal action at every step there is an inherent stochasticity to it.\
If we want the agent to make full use of the learnt policy we can add the following line of code at after the training

```python
trainer.evaluate(render=True)
```

This will not only make the agent follow a deterministic policy and thus help you achieve the maximun reward possible reward attainable from the learnt policy but also allow you to see your agent perform by passing `render=True`

For more information on the VPG implementation and the various hyperparameters available have a look at the official **GenRL** docs [here](https://genrl.readthedocs.io/en/latest/api/algorithms/genrl.agents.deep.vpg.html)

Some more implementations

### VPG agent on an Atari Environment

```python
env = VectorEnv("Pong-v0", env_type = "atari")
agent = VPG('cnn', env)
trainer = OnPolicyTrainer(agent, env, epochs=200)
trainer.train()
```

repository: threewisemonkeys-as/genrl

gha_language: Python

repo_name: threewisemonkeys-as/genrl