text:
---
author: kubu4
comments: true
date: 2016-07-14 20:20:33+00:00
layout: post
slug: computing-a-very-quick-guide-to-amazon-ec2-continued
title: Computing - A Very Quick "Guide" to Amazon EC2 Continued
wordpress_id: 2197
author:
  - kubu4
categories:
  - Miscellaneous
tags:
  - EBS
  - EC2
  - s3fs-fuse
  - scp
---

[Yesterday's post](https://robertslab.github.io/sams-notebook/2016/07/13/computing-the-very-quick-guide-to-amazon-web-services-cloud-computing-instances-ec2.html) ended with me trying to mount a S3 bucket to my EC2 instance using [s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse).

Waited for the 36GB of data to copy over to new bucket with proper naming (i.e. no capital letters in name). Copying took hours; left lab before copying completed.

**Mount S3 bucket: kubu4**

    s3fs kubu4 /mnt/s3bucket/ -o passwd_file=/home/ubuntu/creds_s3fs

![](https://eagle.fish.washington.edu/Arabidopsis/20160714_s3f2_mount_error.jpg)(http://eagle.fish.washington.edu/Arabidopsis/20160714_s3f2_mount_error.jpg)

So, that didn't work. The reason that it doesn't work is that I uploaded the files to the S3 bucket via the Amazon AWS command line (awscli). Apparently, s3fs-fuse can't mount S3 buckets that contain data uploaded via awscli [[see this GitHub Issue for s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse/issues/333)! However, I had to upload them via awscli because the web interface kept failing!

That means I need to upload the data directly to my EC2 instance, but my EC2 instance is set with the default storage capacity of 8GB so I need to increase the capacity to accommodate my two large files, as well as the anticipated intermediate files that will be generated by the types of analysis I plan on running. I'm guessing I'll need at least 100GB to be safe. To do this, I have to [expand the Elastic Block Storage (EBS) volume of my instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-expand-volume.html). The rest of stuff below is fully explained and covered very well in the EBS expansion link I have in the previous sentence.

Don't be fooled into thinking I figured any of this out on my own!

**Expanding the EC2 Instance**

The initial part of the process is creating a Snapshot of my instance. This took a long time (2.5hrs). However, I did finally decide to refresh the page when I noticed that the "Status" progress bar hadn't moved beyond 46% for well over an hour. After refreshing, the "Status" showed "Complete." Maybe this actually was ready to go much faster, but the page didn't automatically refresh? Regardless, in retrospect, since this EC2 instance is pretty much brand new and doesn't have too many changes from when it was initialized, I probably should've just created a brand new EC2 instance with the desired amount of EBS...

Created volume from that Snapshot with 150GB of magnetic storage.

Attached volume to the EC2 instance at /dev/sda1 (the default setting /dev/sdf resulted in an error message about the instance not having a root volume) and SSH'd into the instance. Odd, it seems to show that I still only have 8GB of storage (see the "Usage of..." in the screenshot below):

![](https://eagle.fish.washington.edu/Arabidopsis/20160714_ec2_expanded_volume_01.png)(http://eagle.fish.washington.edu/Arabidopsis/20160714_ec2_expanded_volume_01.png)

Check to see if I actually have the expanded storage volume or not. It turns out, I do! (notice that the only drive listed is "xvda" and its partition, "xvda/xvda1" AND they are equal in size; 150G):

![](https://eagle.fish.washington.edu/Arabidopsis/20160714_ec2_expanded_volume_03.jpg)(http://eagle.fish.washington.edu/Arabidopsis/20160714_ec2_expanded_volume_03.jpg)

Time to upload (via the secure copy command) the files to my EC2 instance! The following commands upload the files to a folder called "data" in my /home directory. I also ran the "time" command at the beginning to get an idea of how long it takes to upload each of these files.

    time scp -i ~/Dropbox/Lab/Sam/bioinformatics.pem /Volumes/web/nightingales/O_lurida/20160223_gbs/160123_I132_FCH3YHMBBXX_L4_OYSzenG1AAD96FAAPEI-109_1.fq.gz ubuntu@ec2.ip.address:~/data

    time scp -i ~/Dropbox/Lab/Sam/bioinformatics.pem /Volumes/web/nightingales/O_lurida/20160223_gbs/160123_I132_FCH3YHMBBXX_L4_OYSzenG1AAD96FAAPEI-109_2.fq.gz ubuntu@ec2.ip.address:~/data

Details on upload times and file sizes:

![](https://eagle.fish.washington.edu/Arabidopsis/20160714_ec2_upload_times.png)(http://eagle.fish.washington.edu/Arabidopsis/20160714_ec2_upload_times.png)

Confim the files now reside in my EC2 instance:

![](https://eagle.fish.washington.edu/Arabidopsis/20160714_ec2_confirm_transfer.jpg)(http://eagle.fish.washington.edu/Arabidopsis/20160714_ec2_confirm_transfer.jpg)

Alas, I should've captured all of this in a Jupyter Notebook. However, I didn't because I thought I would need to enter passwords (which you can't do with a Jupyter Notebook). It turns out, I didn't need a password for anything; even when using "sudo" on the EC2 instance. Oh well, it's set up and running with my data finally accessible. That's all that really matters here.

Alrighty, time to get rolling on some data analysis with a fancy new Amazon EC2 instance!!!

repository: RobertsLab/sams-notebook

extension: markdown

gha_language: Shell

repo_name: RobertsLab/sams-notebook


text:
# About

Calls the AccuWeather API to get a 1 hour forecast of the weather in Singapore.

## Configuration

Create a file named `config`. This file should contain only 1 line with the AccuWeather API key.

To get an AccuWeather API key:

1. Go to https://developer.accuweather.com/ and sign up for free
2. Go to [My Apps](https://developer.accuweather.com/user/me/apps) and create a new app. Once done, you should be able to retrieve the API key for that app.

## Build and run

    ./build-and-run

## Just run (after building)

    ./run

repository: yanhan/scratchpad

branch_name: refs/heads/master

extension: md

license_type: permissive

repo_name: yanhan/scratchpad


text:
CSC417 Project: Exploring the Tech of Noita
===========================================

## References
All references are from [this](https://www.youtube.com/watch?v=prXuyMCgbTc) gcd talk.

[Youtube Explanation](https://youtu.be/ixqOd76E_bU)

[![Recreating Noita](https://img.youtube.com/vi/ixqOd76E_bU/0.jpg)](https://www.youtube.com/watch?v=ixqOd76E_bU)

## User Guide
This project has been test-built on Windows using Visual Studio 2019. I'm not sure if they work on other platforms.

### Setup (Windows)
1. Clone this repository using `git clone --recursive` to get all of the submodules as well.
1. Create and `cd` into the `build` directory at the base of the repository.
1. Run `cmake ..`
1. Open the VS `RecreatingNoita.sln` file.
1. Under solution explorer, right click `main` > select Properties > Change `Output Directory` to the `build/` directory you created in step 2.
1. Build and run the executable using `RELEASE` mode!

### Controls
* Use the number keys to select a material (the name of the material is outputted into the console).
* Left click to place material, right click to erase material
* Scroll wheel to change the radius of the brush
* Space bar to start/stop the simulation. By default, the simulation starts paused.
* Tab to spawn rigid body bouncy balls (only if `SIMULATE_RIGID_BODIES` is set)

### Additional Configurations
Additional configuration is under the `USER SETTINGS` section of `main.cpp`.

| Variable | Use | Default |
| -------- | --- | ------- |
| `SIMULATE_RIGID_BODIES`   | Set this compile flag if you want to simulate rigid bodies. | SET |
| `SPAWN_BODY`              | Set this value to one of `NONE`, `PENDULUM`, `BOUNCE`, or `CAR` to spawn initial rigid bodies. | NONE |
| `DOUGLAS_PEUCKER`         | Set this compile flag if you want to approximate particle contours with the Douglas-Peucker algorithm. Greatly increases performance | SET |
| `DEBUG_DRAW`              | Set this compile flag if you want to show calculated contours. | UNSET |
| `LOAD_FROM_FILE`          | Set this compile flag if you want to load a file from disk as initial falling sand state | UNSET |
| `TEXTURE_FILE`            | Set this value to a `.b` file name under the `/assets/` folder. This is the initial state of the falling sand simulation | |
| `SIM_WIDTH`, `SIM_HEIGHT` | The dimensions of the simulation. Lower this if the simulation runs too slow. | 400, 300 |
| `RENDER_WIDTH`, `RENDER_HEIGHT` | The dimensions of the render window. Leave this as a whole number multiple of `SIM_WIDTH`, `SIM_HEIGHT` | 1200, 900 |

license: MIT


text:
# Gloeocystidium contiguum (P. Karst.) Bourdot & Galzin, 1913 SPECIES

#### Status
SYNONYM

#### According to
The Catalogue of Life, 3rd January 2011

#### Published in
Bull. Soc. mycol. Fr. 28(4): 362 (1913)

#### Original name
null

### Remarks
null

repository: mdoering/backbone

path: /life/Fungi/Basidiomycota/Agaricomycetes/Polyporales/Meruliaceae/Intextomyces/Intextomyces contiguus/ Syn. Gloeocystidium contiguum/README.md

repo_name: mdoering/backbone


text:
# Important things

To learn how to add commands to the bot I recommend you to watch [An Idiot's Guide](https://www.youtube.com/playlist?list=PLR2_rarYLHfg6ZJqq0WTMmI9uLcd7_GRO) on Discord.JS!

Look through all the comments so you don't miss out anything!

# Configuring the bot
After you "fork" / download the files you will need to install [NodeJS](https://nodejs.org/en/), open a command prompt in the folder of the bot (Shift + Right Click => Open command window here)

After you are done with this, just do ```npm i``` in the console and it will install all the things the bot needs to use!

After this things, you are ready to add commands to it!

**Don't forget to add in your token and the prefix in the config.json file!**

Why? Because some people are lazy, that's why :)


text:
# Stock Market Prediction Using Scala, Akka, Play and Spark

## Running

```bash
sbt run
```

And then go to <http://localhost:9000> to see the running web application.

## Controllers

- HomeController.scala:

  Shows how to handle simple HTTP requests.

## Components

## Filters

gha_language: Scala


text:
---
coverImageUri: ""
title: "Install .NET 3.5 on Windows 8"
datetime: "2012-10-01T12:59:47.000Z"
categories: "technology"
tags: ".net,tips,windows,windows 8"
---

If you're having troubles installing .NET 3.5 on to Windows 8, here's a quick tip that may solve your problem.

Make sure your Windows 8 installer disc is in your drive (or mounted/copied somewhere), then from an elevated command prompt run this command:

```bash
dism.exe /online /enable-feature /featurename:NetFX3 /Source:d:\\sources\\sxs /LimitAccess
```

Replace _d:\\_ with the drive/path to your local sources.

Solution via: ["I am unable to install .net 3.5 on windows 8" on Technet Social](http://social.technet.microsoft.com/Forums/en-US/W8ITProPreRel/thread/f4217b5c-4341-4c6a-99be-46a178b34132/#1f4263a3-c528-42c7-b916-bb671db42845)

_As a sidenote, I needed .NET 3.5 for SQL Server 2012._

repository: brandonmartinez/brandonmartinez

repo_name: brandonmartinez/brandonmartinez


text:
---
title: Layouts
description: Using Laravel Blade layouts to build emails with Maizzle PHP
page_order: 5
navigation:
  group: Developing
  order: 5
---

# Layouts

The foundation of any email template in Maizzle PHP is the Layout.
These are 'master' pages that contain the boilerplate of your emails.
They pull in (or `yield()`) named content sections from any page that extends them.

In Maizzle PHP, layouts usually contain the `doctype`, `<head>` and `<body>` tags, with all the necessary child tags, like `<meta>` or other markup that doesn't need to change.

---

## Creating A Layout

Layout files are Laravel Blade files, stored in the `source/_layouts` directory.
For example, here is the `master.blade.php` layout in Maizzle PHP:

```php
<!DOCTYPE @{!! $page->doctype ?? 'html' !!}>
<html lang="@{{ $page->language ?? 'en' }}" xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office">
<head>
  <meta charset="@{{ $page->charset ?? 'utf8' }}">
  <meta http-equiv="x-ua-compatible" content="ie=edge">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <meta name="x-apple-disable-message-reformatting">
  @@if($page->title)<title>@{{ $page->title }}</title>@@endif

  @@if(is_object($page->googleFonts) && $page->googleFonts->isNotEmpty())
  <!--[if !mso]><!--><link rel="stylesheet" href="https://fonts.googleapis.com/css?family=@{{ $page->googleFontsString() }}"><!--<![endif]-->
  @@endif

  @@include('_partials.css.email')
  @@stack('head')
</head>
<body class="@{{ $page->bodyClasses ?? '' }}">

  @@if($page->preheader)
  <div class="hidden text-0 leading-0" lang="@{{ $page->language ?? 'en' }}">@{!! $page->preheader !!}</div>
  @@endif

  @@yield('content')

</body>
</html>
```

As you can see, it's an email-tailored HTML boilerplate. It uses Laravel Blade syntax to insert variables and partials,
which are being pulled in from either your environment config, or from the template's Front Matter.

Notice the `@@yield('content')` Blade directive. This pulls in a section named `content` from any template that extends this layout.
More on sections, in the [Templates documentation](/docs/templates/#defining-sections) and in the [Laravel Blade docs](https://laravel.com/docs/5.6/blade#defining-a-layout).

## Referencing Variables

Variables, like those defined in the config or with [Front Matter at a template level](/docs/pages/#page-variables), are referenced via `$page->varname`.
In the above example, variables like `$page->language` or `$page->charset` are pulled from the config, while others like `$page->preheader` have been defined with Front Matter in the templates.

Jigsaw allows you to overwrite config variables with Front Matter, from each template.
So if one of your emails needs a different charset, you can easily override the global one.

Read the Jigsaw docs [on custom Front Matter variables](http://jigsaw.tighten.co/docs/content-markdown/).

Overriding Nested Keys

        Currently, it's not possible in Jigsaw to inherit nested config keys in Front Matter.
        If you want to override a single nested key-value pair, you will need to specify all other sibling pairs, too.

## Default extends

Maizzle PHP defines an `extends` key in your config, so that any `*.[blade].md` files will automatically extend the master Layout that the framework provides.
This way, you don't need to always specify a layout when using Blade-Markdown hybrid files.

```php
// config.php

/*
|-----------------------------------------------------------------------------
| Layout                                    https://maizzle.com/docs/layouts/
|-----------------------------------------------------------------------------
|
| Define a master layout that all templates will extend by default.
|
| Maizzle comes with a default layout that sets various tags to
| use settings from your config, but you can of course create
| your own layouts and extend them at a template level,
| with front matter.
|
*/
'extends' => '_layouts.master',
```

Of course, each Template can extend its own Layout, by specifying it with Front Matter:

```
// emails/letter.blade.md

---
extends: _layouts.letter
---
```

... or with a Blade directive:

```
// emails/example.blade.php

---
title: Extending with a Blade directive, though Front Matter would work too
---

@@extends('_layouts.example.master')
```

Layouts are referenced relative to the source directory using dot notation, where each dot represents a directory separator in the file name and the .blade.php extension omitted.

repository: jasonlbeggs/docs-php

gha_license_id: MIT

repo_name: jasonlbeggs/docs-php


text:
#User Stories

##Marketing Research Team
*As a member of the marketing research team, I want to see the total number of clicks for each picture, so I get a general idea based off from the raw data.
*As a member of the marketing research team, I want to see the percentage of times that a picture was clicked when it was shown, so I get a set of normalized data.
*As a member of the marketing research team, I want to see the data presented visually, so the result is easy to follow.
*As a member of the marketing research team, I want to see top 5 items selected at the top of the data set, so that I get a quick summary of the result.

##Developer
*As a developer, I want to plan my course of action beforehand, so that I know what to do and not get lost.
*As a developer, I want to use object constructor so that I can showcase the lesson learned from the last week and write the JS in a more efficient manner.
*As a developer, I want to create an event handler that shows three random pictures which were not displayed immediately before, so that it doesn't taint the result.
*As a developer, I want to track the number of clicks and the number each picture were displayed, so I can use those numbers to calculate the percentage of times that a picture was clicked when it was shown.

##Focus Group
*As a member of the focus group, I want a consistent UI/UX, so that I don't get distracted while going through the selection.
*As a member of the focus group, I want to see the pictures side by side but in a random order, so that I don't form any biases.
*As a member of the focus group, I want a complete product with a minimal glitch, so it behaves the way I expect and not distract me.
*As a member of the focus group, I want a limited round of selections, so that I don't get bored and close the application.

repository: jeremymaya/bus-mall

gha_language: JavaScript

repo_name: jeremymaya/bus-mall


text:
**STRUCT**

# `GraphQLVariable`

```swift
public struct GraphQLVariable
```

## Methods
### `init(_:)`

```swift
public init(_ name: String)
```

repository: abdullahselek/apollo-ios

repo_name: abdullahselek/apollo-ios


text:
# Tailor news

## Abstract

This has been released to share the code presented during [Voice Tech TO #3](https://www.meetup.com/voice-tech-to/events/247849388/)
Voice Tech TO is a Toronto based event exploring Google Home, Amazon Alexa, and the wide world of voice.

Voice Tech TO #3 was workshop on the setup for building for voice.
The repository holds the code that was used as an illustration in the meetup.

> This tutorial was written on a Mac architecture, please make the necessary adjustment if the code is used on another architecture.

## Description

Tailor news is the idea of a voice skill that will fetch the top headline on a user inputted topic.

#### Requirements

A machine equiped with
* a [https://developer.amazon.com](http://developer.amazon.com) account
* an [https://console.aws.amazon.com](https://console.aws.amazon.com) account (keep the AWS secret and user ID that will be given at the end)
* [aws-cli](https://aws.amazon.com/cli/) installed
* [Node](https://docs.npmjs.com/getting-started/installing-node)
* [Yarn](https://yarnpkg.com/lang/en/docs/install/)
* [ask-cli](https://developer.amazon.com/docs/smapi/quick-start-alexa-skills-kit-command-line-interface.html)

#### Setup

###### Installation

```
# build the lambda
cd lambda/custom
yarn
```

###### Test

* command:
```
# from the lambda directory
cd lambda/custom # if not already in the directory
yarn test
```
* expected output:
```
  Handlers testing
    Launch request
      ✓ should send back a speak, listen response (9ms)
    News Inquiry Intent
      ✓ should return the summary of the top article for a category (9ms)

Test Suites: 1 passed, 1 total
Tests:       2 passed, 2 total
```

##### Deploy

> Execute the following from the repository root directory

* make a local copy of the [ask config](./docs/data/ask-config-example.json) in the current directory under `.ask/config` (create the .ask directory)
* update the skill id and lambda arn in the previously created file.
* `ask-cli deploy`

## Next steps

#### Features

A list of features to make has been shared on the Voice Tech TO slack channel ([https://goo.gl/hNZryK](https://goo.gl/hNZryK))

#### Code

the integration tests should include tests for bad / malformed data.

## License

MIT

## Readings

* [https://developer.amazon.com/alexa-skills-kit/design](https://developer.amazon.com/alexa-skills-kit/design)
* [https://github.com/alexa/alexa-cookbook](https://github.com/alexa/alexa-cookbook)


text:
# Generational Garbage Collection Process

The pictures below walks through the object allocation and aging process in the JVM.

## Step I: Object Allocation

First, any new objects are allocated to the eden space. Both survivor spaces start out empty

![Object Allocation](img/gc-process-1.PNG)

## Setp II: Filling the Eden Space

When the eden space fills up, a minor garbage collection is triggered.

![Filling the Eden Space](img/gc-process-2.PNG)

## Step III: Copying Referenced Objects

Referenced objects are moved to the firet survivor space. Unreferenced objects are deleted when the eden space is cleared.

![Copying Referenced Objects](img/gc-process-3.PNG)

## Step IV: Object Aging

At the next minor GC, the same thing happens for the eden space. Unreferenced objects are deleted and referenced objects are moved to a survivor space. However, in this case, they are moved to the second survivor space (S1). In addition, objects from the last minor GC on the first survivor space (S0) have their age incremented and get moved to S1. Once all surviving objects have been moved to S1, both S0 and eden are cleared. Notice we now have differently aged object in the survivor space.

![Object Aging](img/gc-process-4.PNG)

## Step V: Additional Aging

At the next minor GC, the same process repeats. However this time the survivor spaces switch. Referenced objects are moved to S0. Surviving objects are aged. Eden and S1 are cleared.

![Additional Aging](img/gc-process-5.PNG)

## Step VI: Promotion

This slide demonstrates promotion. After a minor GC, when aged objects reach a certain age threshold (8 in this example) they are promoted from young generation to old generation.

![Promotion](img/gc-process-6.PNG)

## Step VII: Additional Promotion

As minor GCs continue to occure objects will continue to be promoted to the old generation space.

![Additional Promotion](img/gc-process-7.PNG)

## Step VIII: GC Process Summary

So that pretty much covers the entire process with the young generation. Eventually, a major GC will be performed on the old generation which cleans up and compacts that space.

![GC Process Summary](img/gc-process-8.PNG)

# Generational Garbage Collection Process Lab

[*Generational Garbage Collection Process Lab*](../labs/lab-heap-usage.md)

repository: kylinsoong/troubleshooting

repo_name: kylinsoong/troubleshooting


text:
iPython Notebook Docker image
===================

Based on the [sti-python](https://github.com/openshift/sti-python) repo,
this repository contains the source for building the iPython
Notebook application as a reproducible Docker image using
[source-to-image](https://github.com/openshift/source-to-image).
Users can choose between RHEL and CentOS based builder images.
The resulting image can be run using [Docker](http://docker.io).

Usage
---------------------
To build the iPython Notebook application using standalone
[S2I](https://github.com/openshift/source-to-image) and then run the
resulting image with [Docker](http://docker.io) execute:

*  **For RHEL based image**
    ```
    $ s2i build https://github.com/grafuls/sti-ipython-nb.git --context-dir=2.7 openshift/ipython-27-rhel7 ipython-nb-app
    $ docker run --net=host ipython-nb-app
    ```

*  **For CentOS based image**
    ```
    $ s2i build https://github.com/grafuls/sti-ipython-nb.git --context-dir=2.7 openshift/python-27-centos7 ipython-nb-app
    $ docker run --net=host ipython-nb-app
    ```

**Notice! The images need to be built first!**

**Accessing the application:**
```
$ curl <IP_ADDRESS>:8888
```
Or via browser at [https://localhost:8888](https://localhost:8888)

Repository organization
------------------------
* **`<python-version>`**

    * **Dockerfile**

        CentOS based Dockerfile.

    * **Dockerfile.rhel7**

        RHEL based Dockerfile. In order to perform build or test actions on this
        Dockerfile you need to run the action on a properly subscribed RHEL machine.

    * **`s2i/bin/`**

        This folder contains scripts that are run by [S2I](https://github.com/openshift/source-to-image):

        *   **assemble**

            Used to install the sources into the location where the application
            will be run and prepare the application for deployment (eg. installing
            dependencies, etc.)

        *   **run**

            This script is responsible for running the application by using the
            application web server.

        *   **usage***

            This script prints the usage of this image.

    * **`contrib/`**

        This folder contains a file with commonly used modules.

Source repository layout
------------------------

You do not need to change anything in your existing Python project's repository.
However, if these files exist they will affect the behavior of the build process:

* **requirements.txt**

  List of dependencies to be installed with `pip`. The format is documented
  [here](https://pip.pypa.io/en/latest/user_guide.html#requirements-files).

* **setup.py**

  Configures various aspects of the project, including installation of
  dependencies, as documented
  [here](https://packaging.python.org/en/latest/distributing.html#setup-py).
  For most projects, it is sufficient to simply use `requirements.txt`.

repository: grafuls/sti-ipython-nb

repo_name: grafuls/sti-ipython-nb


text:
# Getting started

## Installation
```bash
go get github.com/thomaspoignant/go-feature-flag
```

## SDK Initialisation

First, you need to initialize the `ffclient` with the location of your backend file.
```go linenums="1"
err := ffclient.Init(ffclient.Config{
    PollingInterval: 3 * time.Second,
    Retriever: &ffclient.HTTPRetriever{
        URL:    "http://example.com/flag-config.yaml",
    },
})
defer ffclient.Close()
```
*This example will load a file from an HTTP endpoint and will refresh the flags every 3 seconds (if you omit the
PollingInterval, the default value is 60 seconds).*

## Evaluate your flags
Now you can evaluate your flags anywhere in your code.

```go linenums="1"
user := ffuser.NewUser("user-unique-key")
hasFlag, _ := ffclient.BoolVariation("test-flag", user, false)
if hasFlag {
    // flag "test-flag" is true for the user
} else {
    // flag "test-flag" is false for the user
}
```
You can find more examples in the [examples/](https://github.com/thomaspoignant/go-feature-flag/tree/main/examples) directory.

repository: ibakhti/go-feature-flag

branch_name: refs/heads/main

repo_name: ibakhti/go-feature-flag


text:
# express-token-api-middleware
[![npm version](https://badge.fury.io/js/express-token-api-middleware.svg)](http://badge.fury.io/js/express-token-api-middleware)
[![Build Status](https://travis-ci.org/mallocator/express-token-api-middleware.svg?branch=master)](https://travis-ci.org/mallocator/express-token-api-middleware)
[![Coverage Status](https://coveralls.io/repos/mallocator/express-token-api-middleware/badge.svg?branch=master&service=github)](https://coveralls.io/github/mallocator/express-token-api-middleware?branch=master)
[![Dependency Status](https://david-dm.org/mallocator/express-token-api-middleware.svg)](https://david-dm.org/mallocator/express-token-api-middleware)

An express middleware that allows to protect an api behind token authentication, rate limiting and endpoint permissions.

## About

This is a middleware for express that will hopefully make your life easier if you want to make your api available to 3rd party developer
by using token to authenticate. The philosophy behind this middleware is to be completely database independent and rely on token to
retrieve information about the user such as access rights and rate limitations.

Tokens are encrypted using AES256 with GCM and can be given out to users without them having access to the data within. Tokens can also
be used to store additional metadata, note though that token will increase in size if you do.

## Getting Started

First let's initialize the middleware

```Javascript
var express = require('express');
var tokens = require('express-token-api-middleware');

var app = express();
var mw = tokens({
    password: 'my super secret password',
    salt: 'something that will be at least 16 bytes long (preferably a Buffer)'
});
app.use(mw);
app.get('/', (req, res) => {
    console.log(req.user);      // => { id: 'My own user id' }
    res.end();
}):
```

We've now set up the token authentication for all requests. Sending requests to the server will all be blocked with a 401 status (No auth provided).
Now we need a token to authenticate our requests:

```Javascript
var token = mw.getToken({ id: 'My own user id' })
```

This token will hold additional data such as a rate limit or access rights (more on that in the API section). Now we need to use the token to
authenticate our server (we make use of the supertest request helper, but the code should be self explanatory):

```Javascript
request(app).get('/').set('Authorization', token).end();
// => Status 200 "OK"
```

And that's all. Now let's dive into the API for more details:

# Api

### <constructor>(config)

There are a few options that you can set to change the behavior of the middleware:

```Javascript
var mw = tokens({
    param: 'token',
    nodes: 1,
    password: <needs to be set>,
    salt: <needs to be set>,
    logger: null,
    timeout: undefined
})
```

A look at the individual options:

* param:    The token can be sent either as Authorization header or as a get parameter or even as a cookie parameter (if enabled). This option specifies the parameter name.
* nodes:    If this server is running as part of a cluster, you can specify how many nodes are in the cluster. This will be factored in when calculating wait times based on the rate limit.
* password: This is the password used to encrypt tokens. A unique password is recommended.
* salt:     The password will be salted to make it more secure. A longer salt means better randomness (recommended min length is 16 byte). You can generate on using the crypto library: ```crypto.randomBytes(16)```.
* logger:   In case you want to know what's going on you can pass in a logger function that accepts a string as first parameter (e.g. console.log)
* timeout:  Sets how long a request will be made to wait to fulfill the rate limit before it will be rejected with status 429 - Too Many Requests. Protects against overflowing request queues.
* error:    A custom error handler that will be used instead of letting the middleware respond to requests. The signature is ```error(req, res, next, status, message)```.

Note that password and salt are used to create a single key tha will encrypt all tokens. This tool does not use a key per user as this
would require much more memory or a database tie in. In most cases this should be safe enough, but if you require higher security a different
solution is probably better suited.

### getToken(config)

To authenticate your users you need to give them the tokens they can authenticate with. This where this method comes in.
There are a few standard properties that are used by the middleware, but you can add any additional properties which will
be stored in the token alongside the user configuration.

```Javascript

var token = mw.getToken({
    id: 'user id',
    path: '/path',
    rate: 100,
    exp: Date.now() + 86400000
    custom: 'whatever'
});
```

Again a few options that need explaining:

* id:   The unique user id e.g. from your database. This will be used to associate incoming requests to the same user.
* path: A regular expression or string that will be treated as regex that decides whether the user is allowed to access an endpoint on the server.
* rate: Define the minimum interval between requests that a user can make. This setting can be a number (in ms) or a string with a unit (e.g. "100ms").
* exp:  Set an expiration rate for this token. This can be a number, a Date object or a string that Date.parse understands.

Rate limitation works in such a way that incoming requests will have a minimum interval of the given value. If 2 requests come in faster than that,
the second request will be delayed until the desired rate has been reached. The rate format supports multiple units: ns, ms, s, m, h, d, w. Note
though that if you use ns (= nano seconds) the minimum wait time between 2 requests is 1 ms).

## Cluster support

Now if you're running this server in any serious environment you'll probably have a cluster with more than 1 node running. Since there's no
communication between the middleware on the nodes built in, there's 2 ways you can still get a global rate emulated:

### set nodes

*Nodes configuration*

The simplest method is to specify how many nodes are in the cluster. The middleware will use that number to multiply the wait factors for
allowed rates. There are two ways how you can set the current number of nodes:

1. _Set the configuration on the constructor_
   Remember that nodes configuration option on the constructor? That's what that is for.
2. _Set it on the middleware instance_
   Since nodes will join and leave a cluster you might want to update the number of nodes dynamically. That's where the next api call comes in:

The nodes setter on the middleware allows you to change the number of nodes that are active even after the middleware has been initialized.

```Javascript
mw.nodes = 10;
```

### notify(user, requests = 1)

*Notify requests*

The second method to use for cluster support is to notify the middleware of requests on other nodes. Unfortunately you will have to take care
of setting up the infrastructure for communicating such information between nodes. You'll also have to specify a user object with rate limit to
make use of this feature.

```Javascript
mw.notify({
    id: 'user on other node',
    rate: 100
}
```

You can specify the number of requests the user has made on other nodes and thereby make any requests to this instance wait even more. How well
this approach will work with your setup depends on how good the communication is set up between nodes.

In any case I would recommend to only use this if you know what you're doing and otherwise stick to the nodes configuration method.

## Events

The middleware also emits events in case you want to react to some of the possible error events. All events have the same signature and can
be used as such:

```Javascript
mw.on('missing', req => console.log('Missing auth token from request', req));
```

### missing(req)

Triggered whenever the middleware rejects a request because no token has been found. The request object does not include the user object as
there is nothing to decrypt.

### fail(req)

Triggered whenever the middleware was unable to decrypt a token. The request object does not include the user object as we were unable to
decrypt it.

### reject(req)

Triggered whenever a user is rejected access to a specific path. The request object includes the decrypted user object.

### expired(req)

Triggered whenever a user token has expired. The request object includes the decrypted user object.

### timeout(req)

The timeout event is triggered whenever the request queue is full and request get rejected. The request object includes the decrypted
user object.

### success(req)

Triggered when a request has successfully been queued up and has already been processed or will be processed once the rate limit queue
has caught up.

## Tests / Examples

For more examples and to check out the tests you can look at [middleware.test.js](test/middleware.test.js) in the test directory.

## Roadmap

Some ideas to maybe work on in the future:

* Rate limit based on number of calls instead of timing

repository: mallocator/express-token-api-middleware

repo_name: mallocator/express-token-api-middleware


text:
Phrase2vec
==============

This is an extension of word2vec to learn n-gram (phrase) embeddings as described in the following paper (Section 3.1):

Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018. **[Unsupervised Statistical Machine Translation](https://arxiv.org/pdf/1809.01272.pdf)**. In *Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2018)*.

Further extended to include code to train embeddings directly from a compressed (gzipped) file.

If you use this software for academic research, please cite the paper in question:
```
@inproceedings{artetxe2018emnlp,
  author    = {Artetxe, Mikel  and  Labaka, Gorka  and  Agirre, Eneko},
  title     = {Unsupervised Statistical Machine Translation},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  month     = {November},
  year      = {2018},
  address   = {Brussels, Belgium},
  publisher = {Association for Computational Linguistics}
}
```

Usage is equivalent to word2vec, with the addition of an optional parameter `--phrases <file>` to specify the set of phrases (one per line) to learn embeddings for. For best results, we recommend disabling subsampling (i.e. `--sample 0`). Here is an example call with the hyperparameters used in our experiments:
```
./word2vec -cbow 0 -hs 0 -sample 0 -size 300 -window 5 -negative 10 -iter 5 \
           -train CORPUS.TXT \
           -phrases PHRASES.TXT \
           -output OUTPUT.TXT
```

In order to train from a compressed file, simply replace `./word2vec` by `./word2vec-zlib`.

For more details on word2vec, please refer to the original README at `README-original.txt`.


text:
# OPULauncher

OPULauncher is a convenience launcher for Outpost 2, that specifies enabling the Outpost Universe unofficial modifications. It can also check online for new OPU mod updates.

Aside from the update checker, OPULauncher is the equivalent of running `Outpost2.exe /OPU`. To run the game under a debugger, you may wish to do that instead of going through OPULauncher.exe in order to sidestep child process debugging issues.

OPULauncher does not itself inject the op2ext mod loader into Outpost 2; that is handled by the OPULoaderShim project.

repository: OutpostUniverse/OPULauncher

repo_name: OutpostUniverse/OPULauncher


text:
## Documentation

[**Invoking the calculation**](https://github.com/jutunen/Paydown.js/tree/master/doc#invoking-the-calculation)

  [Method arguments](https://github.com/jutunen/Paydown.js/tree/master/doc#method-arguments)

  [Init_data object properties](https://github.com/jutunen/Paydown.js/tree/master/doc#init_data-object-properties)

  [Recurring object properties](https://github.com/jutunen/Paydown.js/tree/master/doc#recurring-object-properties)

  [Event object properties](https://github.com/jutunen/Paydown.js/tree/master/doc#event-object-properties)

  [Passing events to the calculator instance](https://github.com/jutunen/Paydown.js/tree/master/doc#passing-events-to-the-calculator-instance)

[**Calculation results**](https://github.com/jutunen/Paydown.js/tree/master/doc#calculation-results)

[**Debug logging**](https://github.com/jutunen/Paydown.js/tree/master/doc#debug-logging)

[**Interest calculation**](https://github.com/jutunen/Paydown.js/tree/master/doc#interest-calculation)

[**Negative interest calculation**](https://github.com/jutunen/Paydown.js/tree/master/doc#negative-interest-calculation)

[**"Interests only" recurring payments**](https://github.com/jutunen/Paydown.js/tree/master/doc#interests-only-recurring-payments)

[**Fee calculation**](https://github.com/jutunen/Paydown.js/tree/master/doc#fee-calculation)

### Invoking the calculation
```javascript
Paydown.calculate(init_data, events, payments, debug_log)
```
The calculate method can be considered as pure, Paydown object doesn't preserve any state after it has been called.
#### Method arguments

Argument|Direction|Type|Optional|Description
--------|---------|----|--------|-----------
init_data|input|object|no|Initial calculation values
events|input|array of objects|yes|List of changes during calculation period
payments|output|array|yes|List of individual payments
debug_log|output|array|yes|Detailed info about paid interests etc.

#### Init_data object properties

Name|Type|Optional|Description|Format / Values|Default value
----|----|--------|-----------|---------------|-------------
start_date|string|no|Calculation start date|"dd.mm.yyyy"
end_date|string|no|Calculation end date|"dd.mm.yyyy"
principal|number|no|Principal amount at the start date|
rate|number|no|Interest rate at the start date|
day_count_method|string|yes|Determines how interest accrues over time|"act/360"
"act/365"|"act/360"
round_values|boolean|yes|Sets output value rounding to 2 decimals|true
false|true
debug_logging|boolean|yes|Enables debug logging|true
false|false
initial_fee|number|yes|Loan establishment fee||0
recurring|object|yes|Defines recurring payments|See below|

#### Recurring object properties

Name|Type|Optional|Description|Format / Values|Default value
----|----|--------|-----------|---------------|-------------
amount|number|no|The amount of recurring payment||
payment_method|string|yes|Payment method for recurring payments|"equal_installment"
"equal_reduction"|"equal_installment"
first_payment_date|string|no|First recurring payment date|"dd.mm.yyyy"|
payment_day|number|no|Monthly payment day of the recurring payment, applied after the month of the first payment date|1 - 31 (integer value required)
31 equals to the last day of the month|
payment_fee|number|yes|Recurring payment fee||0
payment_period|number|yes|Recurring payment period length in months|1 - 12 (integer value required)|1

#### Event object properties

Name|Type|Optional|Description|Format / Values
----|----|--------|-----------|---------------
date|string|no|Event date|"dd.mm.yyyy"
rate|number|yes|New interest rate|
recurring_amount|number|yes|New recurring payment amount|
pay_installment|number|yes|Individual installment payment|
pay_reduction|number|yes|Individual principal reduction payment|
payment_method|string|yes|New payment method|"equal_installment"
"equal_reduction"
pay_single_fee|number|yes|Individual fee|
recurring_fee_amount|number|yes|New recurring payment fee amount|

#### Passing events to the calculator instance

Events are passed to calculator as an array of event objects. Event date based chronological order of event objects in the array is not required, calculator shall sort the events by itself. Each event object must have a unique date, do not pass in objects with same dates.

### Calculation results

Paydown.calculate method returns an object with following properties:

Name|Type|Description
----|----|-------
sum_of_interests|number|Total sum of accrued interests during calculation period
sum_of_reductions|number|Total amount of the principal reductions during calculation period
sum_of_installments|number|Total sum of installments during calculation period
remaining_principal|number|Remaining principal after calculation end date
days_calculated|number|Number of days calculated
actual_end_date|string|Actual calculation end date ( "dd.mm.yyyy" )
latest_payment_date|string|Latest payment date ( "dd.mm.yyyy" )
unpaid_interest|number|Interests accrued after latest payment date ( within calculation period )
sum_of_fees|number|Total sum of fees during calculation period

If Paydown.calculate method is provided with an array as its 3rd argument, the array contents are interpreted as follows:

    [ date, interest rate, installment, principal reduction, interest, remaining principal, fee ]

### Debug logging

Debug logging can be enabled by setting init_data object property *debug_logging* to *true*.

Debug data shall be logged to an array, if Paydown.calculate method is provided with an array as its 4th argument, otherwise debug data shall be logged through console.log.

### Interest calculation

Daily interests are calculated with following rules:

- daily interest is always calculated for the whole day with a single rate
- daily interest is never calculated for a part of a day or with multiple rates
- calculation start and end date interests are always included to the calculation completely
- payment day interest is calculated from the capital before the payment
- interest of the rate change day is calculated with the new rate

### Negative interest calculation

If payment method is equal reduction, the negative interest shall be compensated by subtracting the interest from the installment.

If payment method is equal installment, the negative interest shall be compensated by adding the interest to the reduction.

### "Interests only" recurring payments

Recurring payments can be set to cover only the interests by setting recurring object property *amount* to zero or by creating an event with *recurring_amount* set to zero.

### Fee calculation

Fee calculation is optional and completely isolated from other payment calculations. Fee calculation doesn't have any effect on installment / reduction / interest / principal calculations.


text:
# ML_GROUP020
Assignment 10

     Due Tuesday by 23:59 Points 26 Submitting a file upload Available Nov 26 at 0:00 - Dec 10 at 23:59 15 days

Groups 10,20,30,40,50,60,70,80,90,100,110,120,130,140,150
Instructions

    Follow the instructions in each question carefully.
    1. A Jupyter notebook along with output for each cell is expected.
    2. Any assignment submitted using  other python IDEs are not considered for grading.
    3. Use appropriate labels for all visualizations.
    4. Upload the output.csv file along with the notebook when required.

Question 1

Spam filtering is an example of document classification task which involves classifying an email / SMS as spam or non-spam (a.k.a. ham).

    1. Import the dataset from https://www.kaggle.com/uciml/sms-spam-collection-dataset. (1 point).
    2. Split the data into training and testing. (1 point). Use 10-fold cross validation.(1 point)
    3. Extract features using TF-IDF and display the features. ( 2 points)
    4. Model and train the classifier using GaussianNB, BernoulliNB and MultinomialNB.( 3 points)
    5. Evaluate classifiers on Test Data. ( 2 points)
    6. Plot the decision boundary, visualize training and test results of all the models (3 points)

Question 2

The following dataset is used to classify the car acceptability into classes: unacceptable, acceptable, good and very good.

    1. Import the data dataset from https://archive.ics.uci.edu/ml/machine-learning-databases/car/ (Links to an external site.) (1 points).
    2. Extract X as all columns except the last column and Y as last column. (1 points)
    3. Visualize the dataset using any two appropriate graphs. (2 points)
    4. Visualize the correlation between all the variables of dataset. (1  points)
    5. Split the data into training set and testing set. (1 points).Perform 10-fold cross validation (1 point).
    6. Train a Logistic regression model for the dataset. (1 points)
    7. Compute the accuracy and confusion matrix. (2 points)
    8. Plot the decision boundary, visualize training and test results (2 point)
    9. Predict and display the class label of a car with following attributes : buying, maint, doors, persons, lug_boot, safety as [vhigh,low,4,more,small,med]. (1 points)


text:
# LYNX ACCESS CARD SYSTEM [LACS] (FINAL YEAR PROJECT)

A school management system that allows students to print an access card (for exams). Printing of access card is dependent on certain criteria being met (i.e student's monthly balance needs to be at 0.00 in order to qualify) - developed using the MERN stack.

#### What you will need to run this code
1. Node (13.12.0)
2. NPM (6.14.4) or Yarn (1.22.4)
3. MongoDB (4.2.0)

####  How to run this code
1. Make sure MongoDB is running on your system and that you have a MongoAtlas account set up.
2. Clone this repository
3. Open command line in the cloned folder,
   - To install dependencies, run ```  npm install  ``` or ``` yarn ```
   - To run the application for development, run ```  npm run development  ``` or ``` yarn development ```
4. Open [localhost:3000](http://localhost:3000/) in the browser
----

Live web version hosted on Heroku: https://lynx-access.herokuapp.com/
Test it out using these two students credentials:

Student1 login details(fees up to date):
Email address<EMAIL_ADDRESS>Password: password

Student2 login details(fees not up to date):
Email address<EMAIL_ADDRESS>Password: password

repository: CodeMerchant/school-access-card-system

repo_name: CodeMerchant/school-access-card-system


text:
# go-rest-api

## Some Bug fix

#### Unable to determine go version with Go installed using Snap

```
"go.alternateTools": {
    "go": "/snap/go/current/bin/go"
}
```


text:
## v1.4.2
### 2020-07-13

### Download
https://github.com/vmware-tanzu/velero/releases/tag/v1.4.2

### Container Image
`velero/velero:v1.4.2`

### Documentation
https://velero.io/docs/v1.4/

### Upgrading
https://velero.io/docs/v1.4/upgrade-to-1.4/

### All Changes
  * log a warning instead of erroring if an additional item returned from a plugin can't be found in the Kubernetes API (#2595, @skriss)
  * Adjust restic default time out to 4 hours and base pod resource requests to 500m CPU/512Mi memory. (#2696, @nrb)
  * capture version of the CRD prior before invoking the remap_crd_version backup item action (#2683, @ashish-amarnath)

## v1.4.1

This tag was created in code, but has no associated docker image due to misconfigured building infrastructure. v1.4.2 fixes this.

## v1.4.0
### 2020-05-26

### Download
https://github.com/vmware-tanzu/velero/releases/tag/v1.4.0

### Container Image
`velero/velero:v1.4.0`

### Documentation
https://velero.io/docs/v1.4/

### Upgrading
https://velero.io/docs/v1.4/upgrade-to-1.4/

### Highlights

 * Added beta-level CSI support!
 * Added custom CA certificate support
 * Backup progress reporting
 * Changed backup tarball format to support all versions of a given resource

### All Changes
  * increment restic volumesnapshot count after successful pvb create (#2542, @ashish-amarnath)
  * Add details of CSI volumesnapshotcontents associated with a backup to `velero backup describe` when the `EnableCSI` feature flag is given on the velero client. (#2448, @nrb)
  * Allow users the option to retrieve all versions of a given resource (instead of just the preferred version) from the API server with the `EnableAPIGroupVersions` feature flag. (#2373, @brito-rafa)
  * Changed backup tarball format to store all versions of a given resource, updated backup tarball format to 1.1.0. (#2373, @brito-rafa)
  * allow feature flags to be passed from install CLI (#2503, @ashish-amarnath)
  * sync backups' CSI API objects into the cluster as part of the backup sync controller (#2496, @ashish-amarnath)
  * bug fix: in error location logging hook, if the item logged under the `error` key doesn't implement the `error` interface, don't return an error since this is a valid scenario (#2487, @skriss)
  * bug fix: in CRD restore plugin, don't use runtime.DefaultUnstructuredConverter.FromUnstructured(...) to avoid conversion issues when float64 fields contain int values (#2484, @skriss)
  * during backup deletion also delete CSI volumesnapshotcontents that were created as a part of the backup but the associated volumesnapshot object does not exist (#2480, @ashish-amarnath)
  * If plugins don't support the `--features` flag, don't pass it to them. Also, update the standard plugin server to ignore unknown flags. (#2479, @skriss)
  * At backup time, if a CustomResourceDefinition appears to have been created via the v1beta1 endpoint, retrieve it from the v1beta1 endpoint instead of simply changing the APIVersion. (#2478, @nrb)
  * update container base images from ubuntu:bionic to ubuntu:focal (#2471, @skriss)
  * bug fix: when a resource includes/excludes list contains unresolvable items, don't remove them from the list, so that the list doesn't inadvertently end up matching *all* resources. (#2462, @skriss)
  * Azure: add support for getting storage account key for restic directly from an environment variable (#2455, @jaygridley)
  * Support to skip VSL validation for the backup having SnapshotVolumes set to false or created with `--snapshot-volumes=false` (#2450, @mynktl)
  * report backup progress (number of items backed up so far out of an estimated total number of items) during backup in the logs and as status fields on the Backup custom resource (#2440, @skriss)
  * bug fix: populate namespace in logs for backup errors (#2438, @skriss)
  * during backup deletion also delete CSI volumesnapshots that were created as a part of the backup (#2411, @ashish-amarnath)
  * bump Kubernetes module dependencies to v0.17.4 to get fix for https://github.com/kubernetes/kubernetes/issues/86149 (#2407, @skriss)
  * bug fix: save PodVolumeBackup manifests to object storage even if the volume was empty, so that on restore, the PV is dynamically reprovisioned if applicable (#2390, @skriss)
  * Adding new restoreItemAction for PVC to update the selected-node annotation (#2377, @mynktl)
  * Added a --cacert flag to the install command to provide the CA bundle to use when verifying TLS connections to object storage (#2368, @mansam)
  * Added a `--cacert` flag to the velero client describe, download, and logs commands to allow passing a path to a certificate to use when verifying TLS connections to object storage. Also added a corresponding client config option called `cacert` which takes a path to a certificate bundle to use as a default when `--cacert` is not specified. (#2364, @mansam)
  * support setting a custom CA certificate on a BSL to use when verifying TLS connections (#2353, @mansam)
  * adding annotations on backup CRD for k8s major, minor and git versions (#2346, @brito-rafa)
  * When the EnableCSI feature flag is provided, upload CSI VolumeSnapshots and VolumeSnapshotContents to object storage as gzipped JSON. (#2323, @nrb)
  * add CSI snapshot API types into default restore priorities (#2318, @ashish-amarnath)
  * refactoring: wait for all informer caches to sync before running controllers (#2299, @skriss)
  * refactor restore code to lazily resolve resources via discovery and eliminate second restore loop for instances of restored CRDs (#2248, @skriss)
  * upgrade to go 1.14 and migrate from `dep` to go modules (#2214, @skriss)
  * clarify the wording for restore describe for namespaces included

repository: vmware-tanzu/velero

gha_language: Go

repo_name: vmware-tanzu/velero


text:
# Photo Dupe Finder
Greg Kennedy 2012

Got dupes?  I did.  And various photo-manip tools had added their own EXIF headers over the years, but left the original data unchanged.  So I wrote a tool (mainly shelling out to `jpegtran`) to find them.

**There are much better photo duplicate finders out there - including some that use visual similarity - but this worked for me.**

Identifies duplicate photos by image data:
* strips EXIF info and converts to optimize + progressive
* before performing MD5 on image data

## Requires "jpegtran" application from libjpeg project
Mac users: http://www.phpied.com/installing-jpegtran-mac-unix-linux/

license: Unlicense


text:
# Spring Batch Performance Monitoring (SBPM)
This Tool provides the means to monitor the performance of Spring Batch applications without the need to manipulate the respective code basis.

Through Listeners and AOP it evaluates the throughput of a monitored Job and writes the result to a simple database-schema (a file-based H2 by default).

It measures the performance of Job, Step, Chunk, Reader/Processor/Writer/Tasklet down to indivdual Items.

## Installation/Usage

1. Add the dependency to your POM:
```xml
<dependency>
  <groupId>de.viadee</groupId>
  <artifactId>springBatchPerformanceMonitoring</artifactId>
  <version>...</version>
</dependency>
```
2. Add the de.viadee.spring.batch.infrastructure.Configurator.class to your Spring-Application-Context (via @Import on your Configuration)
3. Run the Job
4. By default the monitoring result is written to project-folder/target/database/monitoringDB.mv.db

You can access the database with the credentials sa/sasa. It contains all perfomance data measured in snowflake style as well as number of prepared views for typical questions on all the detail levels mentioned above. You can see the [SQL schema in H2 syntax in the code](https://github.com/viadee/springBatchPerformanceMonitoring/blob/master/src/main/resources/SQL/schema-h2.sql).

## Commitments
This library will remain under an open source licence indefinately.

We follow the [semantic versioning](http://semver.org) scheme (2.0.0).

In the sense of semantic versioning, the resulting database schema is the _only public API_ provided here. We will keep the database schema as stable as possible, in order to enable users to analyse performance logs with the toolsets of their choice.

For the same reason, we do not use any database specific features and we minimize the assumptions made regarding Spring versions and the whole runtime environment.

## Cooperation
Feel free to add issues, questions, ideas or patches. We are looking forward to it.

[![Build Status](https://travis-ci.org/viadee/springBatchPerformanceMonitoring.svg?branch=master)](https://travis-ci.org/viadee/springBatchPerformanceMonitoring)

## License (BSD 3-Clause License)

Copyright (c) 2018, viadee IT-Unternehmensberatung AG
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

* Redistributions of source code must retain the above copyright notice, this
  list of conditions and the following disclaimer.

* Redistributions in binary form must reproduce the above copyright notice,
  this list of conditions and the following disclaimer in the documentation
  and/or other materials provided with the distribution.

* Neither the name of the copyright holder nor the names of its
  contributors may be used to endorse or promote products derived from
  this software without specific prior written permission.

repository: viadee/springBatchPerformanceMonitoring

gha_language: Java

repo_name: viadee/springBatchPerformanceMonitoring


text:
---
title: microshift-csi-snapshot-integration
authors:
  - copejon
reviewers:
  - dhellman
  - pmtk
  - eggfoobar
  - pacevedom
  - "jsafrane, storage team"
approvers:
  - dhellmann
api-approvers:
  - None
creation-date: 2023-05-10
last-updated: 2023-06-02
tracking-link:
  - https://issues.redhat.com/browse/USHIFT-1140
see-also: []
---

# CSI Snapshotting Integration

## Summary

MicroShift is a small form-factor, single-node OpenShift targeting IoT and Edge Computing use cases characterized
by tight resource constraints, unpredictable network connectivity, and single-tenant workloads. See
[kubernetes-for-devices-edge.md](./kubernetes-for-device-edge.md) for more detail.

This document proposes the integration of the CSI Snapshot Controller to support backup and restore scenarios for
cluster workloads.  The snapshot controller, along with the CSI external snapshot sidecar, will provide an API
driven pattern for managing stateful workload data.

## Motivation

CSI snapshot functionality was originally excluded from the CSI driver integration in MicroShift to support the
low-resource overhead goals of the project. However, user feedback has made it clear that a supportable, robust
backup/restore solution is necessary.  While it would be possible to run a workflow out-of-band to manage workload
data, this would be reinventing the wheel and contribute significantly to technical debt.  CSI Snapshots are already
integrated into OpenShift with ongoing downstream support.

### User Stories

As an edge device owner running stateful workloads on MicroShift, I want to create in-cluster snapshots of that state
and to restore workloads to that state utilizing existing Kubernetes patterns.

### Goals

* Enable an in-cluster workflow for snapshotting and restoration of cluster workload data
* Follow the [MicroShift design principles](https://github.com/openshift/microshift/blob/main/docs/design.md)

### Non-Goals

* Enable backup/restore workflows external to MicroShift
* Provide a workflow for exporting data from a MicroShift cluster
* Enabling MicroShift deployment without CSI Snapshotter.  This falls under platform composability and is out of scope
* Describe MicroShift installation and configuration processes in detail
* Support snapshots of the root disk / partition. In the context of this document, only PersistentVolumeClaims and
  PersistentVolumes can be snapshot.

## Proposal

Deploy the CSI snapshot controller and CSI plugin sidecar with MicroShift out of the box to provide users with a means of
snapshotting, cloning, and restoring workload state.

### Workflow Description

#### Deploy MicroShift with Snapshotting Support

_Prerequisites_

* The device owner has created an LVM volume group and a thin-pool on this volume group.
* The `/etc/microshift/lvmd.yaml` config includes a `deviceClass` to represent the thin-pool.  The lvmd.yaml may
contain a mix of thin and thick `deviceClasses`.

```yaml
device-classes:
- name: ssd-thin
  volume-group: myvg1
  type: thin
  thin-pool:
    name: pool0
    overprovision-ratio: 50.0
- name: default
  volume-group: rhel
  default: true
  spare-gb: 0
```

_Workflow_

1. The user installs the MicroShift RPMs and starts the MicroShift systemd service.
2. The Service-Manager initiates the `startCSISnapshotter` service.
3. The `startCSISnapshotter` service deploys the CSI VolumeSnapshot CSI Snapshot Controller and Validating Webhook, and a
   default VolumeSnapshotClass.
   - The snapshot controller is not a startup dependency of LVMS; both can be started concurrently.
4. The CSI VolumeSnapshot Controller begins listening for VolumeSnapshot API events.
5. The user observes the CSI Snapshot Controller pod reaches the **Ready** state.
   - `$ oc get pod -n kube-system csi-snapshot-controller-$SUFFIX`
6. The user observes the `topolvm-controller` and `csi-snapshot-controller` pods reach the **Ready** state.
    - `$ oc get pod -n openshift-storage topolvm-controller-$SUFFIX`
    - The `topolvm-controller` includes the `csi-external-snapshotter` sidecar container, which is why the user must
      verify its phase.

#### Create a Snapshot Dynamically from a PVC

This section describes how to create snapshot from a PVC utilizing a VolumeSnapshotClass to dynamically specify
provisioning parameters.

_Prerequisites_

* A running MicroShift cluster
* CSI Snapshot Controller is in **Ready** state
* Topolvm-Controller is in **Ready** state
* `volumeSnapshotClass` has been deployed
* `storageClass` has been deployed
* The user has deployed a Pod with an attached PVC backed by a LVM thin volume.

_Workflow_

1. The user stops the workload by deleting the consuming Pod, or if managed by a replication controller, scales the
   replicas to 0.
2. The user creates a `volumeSnapshot` object with the following key-values:
   - `.spec.volumeSnapshotClassName` set to the `volumeSnapshotClass`'s name. If not provided, falls back to default
     class.
   - `.spec.source.persistentVolumeClaimName` set to the source PVC's name.
3. The _snapshotting validation webhook_ intercepts the API `volumeSnapshot` API request.
4. If validation succeeds, the `volumeSnapshot` is persisted in etcd.
   - Else: the `volumeSnapshot` is rejected. (see [Failure Modes](#failure-modes)).
5. The _CSI external snapshotter_ sidecar, part of the _topolvm-controller_ deployment, detects the `volumeSnapshot`
    "create" event. It generates a `volumeSnapshotContent` object and triggers the `CreateSnapshot` CSI gRPC process.
6. LVMS executes the `CreateSnapshot` gRPC process and creates a snapshot of the LVM thin volume.
7. LVMS creates a `logicalVolume.topolvm.io` CR and returns the snapshot volume's metadata to the _CSI external
   snapshotter_.
8. The _CSI external-snapshotter_ updates the `volumeSnapshotContent`'s `status` field to indicate it is ready.
9. The _CSI snapshot controller_ detects the update to the `volumeSnapshotContent` status and binds the
    `volumeSnapshot` to the `volumeSnapshotContent` instance. It sets the `volumeSnapshot`'s `status.readyToUse`
    field to `true`.

From here, users will follow the [Restore](#restore) procedure to consume the snapshot volume.

#### Create a Snapshot Manually from a Pre-Provisioned Snapshot

This section describes how to create snapshot directly from a `volumeSnapshotContent` object.

_Prerequisites_

* A running MicroShift cluster
* CSI Snapshot Controller is in **Ready** state
* Topolvm-Controller is in **Ready** state
* Default `storageClass` has been deployed automatically

> NOTE: This is a static provisioning process, meaning the source `volumeSnapshotContent` object is already present.
> There is not need for dynamically provisioning the `volumeSnapshotContent` and therefore we do not need to specify
> a `volumeSnapshotClass`.

1. The user creates a `volumeSnapshot` object with the following key-values:
   - `.spec.source.volumeSnapshotContentName` set to the source `volumeSnapshotContent`'s name.
2. The _snapshotting validation webhook_ intercepts the API volume snapshot API request.
3. If validation succeeds, the `volumeSnapshot` is persisted in etcd.
   - Else: the VolumeSnapshot is rejected. (see [Failure Modes](#failure-modes)).
4. The _CSI external-snapshotter_ sidecar, part of the _topolvm-controller_ deployment, detects the `volumeSnapshot`
    event. It generates a `volumeSnapshotContent` object and triggers the `CreateSnapshot` CSI gRPC.
5. LVMS executes the `CreateSnapshot` gRPC and creates a snapshot of the LVM thin volume.
6. LVMS creates a `logicalVolume.topolvm.io` CR and returns the snapshot volume's metadata to the _CSI external
   snapshotter_.
7. The _CSI external-snapshotter_ updates the `volumeSnapshotContent`'s `status` field to indicate it is ready.
8. The _CSI snapshot controller_ detects the update to the `volumeSnapshotContent` status and binds the
    `volumeSnapshot` to the `volumeSnapshotContent` instance. It sets the `volumeSnapshot`'s `status.readyToUse`
    field to `true`.

From here, users will follow the [Restore](#restore) procedure to consume the snapshot volume.

#### Deleting a Volume Snapshot

> Backing logical volumes will be handled according to the deletion policy of the volumeSnapshotClass. Possible
> values are: "Retain", "Delete".

_Prerequisites_

* A running MicroShift cluster
* CSI Snapshot Controller is in **Ready** state
* Topolvm-Controller is in **Ready** state
* A `volumeSnapshot` bound to a `volumeSnapshotContent`

_Workflow_

> NOTE: `volumeSnapshot` and `volumeSnapshotContent` instances will have finalizers set on them to control the deletion
> sequence.

1. The user deletes the `volumeSnapshot` object. Finalizers on this object prevent immediate deletion.

_If the reclaimPolicy is **Delete**:_

2. The `CSI snapshot controller` detects the deletion event and deletes the bound `volumeSnapshotContent`
   object. Finalizers on this object prevent immediate deletion.
3. The _CSI external-snapshotter_ detects the `volumeSnapshotContent` deletion event.  It calls the `DeleteSnapshot` CSI
   gRPC.
4. LVMS executes the `DeleteSnapshot` process.  The LVM snapshot is destroyed and the process returns to the caller.
5. The _CSI external-snapshotter_ removes the finalizer on the `VolumeSnapshotContent`, allowing it to be garbage collected
6. The _CSI snapshot controller_ removes the finalizer on the `VolumeSnapshot`, allowing it to be garbage collected

_If the reclaimPolicy is **Retain**:_

2. The `CSI snapshot controller` detects the deletion event. It removes finalizers on `volumeSnapshot` and
   `volumeSnapshotContent` objects.
3. The `volumeSnapshot` is garbage collected.
4. The `volumeSnapshotContent` and backing LVM thin volume are retained.  A user can then delete the backing storage
   by deleting the `volumeSnapshotContent`.

#### Restore

Restoring is the process of creating a new lvm volume from a snapshot that is populated with data from the snapshot.
This can be achieved two ways via the `spec.dataSource` PVC field.

_Prerequisites_

* A `storageClass` is deployed automatically
* A `volumeSnapshotClass` is deployed automatically
* A `volumeSnapshot` is bound to a `volumeSnapshotContent` and `readyToUse` is `true

_Workflow_

1. The user creates a PVC with the following fields:
   - `spec.storageClassName`: must be the same storage class that provisioned the source volume
   - `spec.dataSource.name`: name of the source `volumeSnapshot`
   - `spec.dataSource.kind`: VolumeSnapshot
   - `spec.dataSource.apiGroup`: snapshot.storage.k8s.io
2. The user creates a Pod to consume the PVC. Required because LVMS only supports `WaitForFirstConsumer` provisioning.
3. The _CSI external provisioner_ sidecar, already integrated into the _topolvm-controller_, detects the PVC request,
   and makes a `CreateVolume` gRPC call to LVMS.
4. _topolvm-controller_ creates a `logicalVolume.topolvm.io` CRD with the same values of the source `logicalVolume.
   topolvm.io` instance. This ensures the new volume is created on the same node as the source.
5. _topolvm-node_ creates a new thin-volume from the volume snapshot.
6. _topolvm-controller_ returns the success status to the `CreateVolume` gRPC caller.
7. The _CSI external provisioner_ creates a `persistentVolume` to track the new thin-volume
8. The _CSI external provisioner_ binds the PVC and PV together
9. The workload attaches the new volume and starts

#### Deploying

The CSI Snapshot Controller and LVMS components are deployed by default on MicroShift. The manifests for these
components are baked into the MicroShift binary and are deployed upon startup. This follows the existing pattern
for MicroShift’s control-plane elements.

The CSI Snapshotter configuration is managed via the VolumeSnapshotClass API. This API serves a similar purpose as
StorageClasses and allows admins to specify dynamic snapshotting parameters at runtime.  MicroShift will deploy a
default VolumeSnapshotClass on startup. This instance will reference the default StorageClass that is already
deployed by MicroShift to enable snapshotting out of the box.

The MicroShift deployment model utilizes rpm-ostree layers. Following the existing deployment pattern,
VolumeSnapshotClass manifests can be packaged and deployed onto target devices.
(See [kubernetes-for-device-edge.md#workflow-description](./kubernetes-for-device-edge.md#workflow-description)).
Application deployers can predefine VolumeSnapshotClass manifests.  These can be written the manifests to
/etc/microshift/manifests. This pattern is congruent with how a device owner would install custom StorageClasses. It
is recommended that new custom StorageClasses be packaged with VolumeSnapshotClasses that reference them.

#### Upgrading

CSI components are upgraded by MicroShift-internal automation.  These scripts pull the CSI image digests and
related manifests from the OpenShift LVMS Operator Bundle.  This helps to keep MicroShift's version of
CSI compatible with the default storage provisioner (LVMS). MicroShift deploys CSI `v1`, with micro-versions being
guaranteed backwards compatible within each major version.  Therefore, micro-version upgrades will not require user
intervention to protect snapshots from deletion or orphaning.

Upgrading the CSI containers will introduce a small interrupt window to provisioning operations.  Therefore,
to mitigate any risk or orphaning resources, it is recommended that snapshotting be allowed to complete before
restarting MicroShift and initiating the upgrade.

#### Configuring

Cluster admins will use the VolumeSnapshotClass API to set dynamic configurations for snapshot creation.
Driver-specific configuration is available through the ‘parameters’ sub-field, which is a string:string map and is
defined by the particular storage provider.

#### Deploying Applications

### API Extensions

The APIs to be added are defined under the `snapshot.storage.k8s.io/v1` API group. They are:

1. VolumeSnapshot
2. VolumeSnapshotClass
3. VolumeSnapshotContent

For a detailed description of these APIs, see
[OpenShift documentation](https://docs.openshift.com/container-platform/4.13/storage/container_storage_interface/persistent-storage-csi-snapshots.html#volume-snapshot-crds)

### Risks and Mitigations

- Increased overhead.  The CSI snapshot controller and sidecar increase the on-disk footprint by a
total of 360MB.  At idle, the components use a negligible amount of CPU (>1% CPU on a 2 core machine)
and roughly 18Mb.

### Drawbacks

## Design Details

### CSI Snapshot Controller

The CSI Snapshot Controller is maintained SIG-Storage upstream. Downstream OCP releases of this container
image will be used by MicroShift.  The controller is a standalone component responsible for watching
`volumeSnapshot`, `volumeSnapshotContent`, and `volumeSnapshotClass` objects.  When a `volumeSnapshot` is created,
the controller will trigger a snapshot operation by creating a `volumeSnapshotConent` object, which will be
detected by the _CSI external-snapshotter sidecar_.

### CSI External Snapshotter

The CSI External Snapshotter is maintained SIG-Storage upstream. Downstream OCP releases of this container
image will be used by MicroShift. The container is deployed as a sidecar of the _topolvm-controller_. It
watches for `volumeSnapshotContent` and `volumeSnapshotClass` events.  When an event is detected, it
makes the appropriate CSI gRPC call to LVMS via a shared unix socket.

### CSI Validation Webhook

The CSI Snapshot Validation Webhook is maintained SIG-Storage upstream. Downstream OCP releases of this container
image will be used by MicroShift.  The webhook serves as gatekeeper to `volumeSnapshot` CREATE and UPDATE events.
For the specifics of validation, _see_
[Kubernetes KEP](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/1900-volume-snapshot-validation-webhook#validating-scenarios).

> For a deeper explanation of CSI snapshot controller and sidecar, refer to
[OpenShift documentation](https://docs.openshift.com/container-platform/4.13/storage/container_storage_interface/persistent-storage-csi-snapshots.html#persistent-storage-csi-snapshots-controller-sidecar_persistent-storage-csi-snapshots).
> Snapshot webhook validation is described in detail [here](https://github.com/kubernetes-csi/external-snapshotter#validating-webhook).

### Asset Management

MicroShift manages the deployment of embedded component assets through the [service-manager](https://github.com/openshift/microshift/blob/3e08567344fa040a10fb30e013182fa62248f403/pkg/servicemanager/type.go#L1)
interface. The logic behind this interface deploys components in an order that respects interdependencies, waits for
services to start, and intelligently stops services on interrupt.

Infrastructure controllers (service-ca, DNS, routes, CSI and CNI plugins) are managed by the
[infrastructure service](https://github.com/openshift/microshift/blob/878630eac00ab3884072c8ca23c429d5fce4bb6d/pkg/components/controllers.go).
The CSI Snapshotter will be appended to the list of controllers that the infrastructure service manages.

The manifests will be stored under the `microshift/assets/components/csi-snapshot-controller` directory.
The following files will be added:

1. `csi_controller_deployment.yaml` specifies the CSI Snapshot Controller
2. `serviceaccount.yaml` specifies the controller's service account
3. `05_operand_rbac.yaml` provides RBAC configuration to the CSI controller
4. `webhook_config.yaml`
5. `webhook_deployment.yaml`
6. `volumesnapshotclasses_snapshot.storage.k8s.io.yaml` defines the VolumeSnapshotClass CRD
7. `volumesnapshotcontents_snapshot.storage.k8s.io.yaml` defines the VolumeSnapshotContents CRD
8. `volumesnapshots_snapshot.storage.k8s.io.yaml` defines the VolumeSnapshot CRD

### Vendored Packages

None of the snapshotter logic will run from the MicroShift binary, meaning there will be no additional vendored dependencies.

### Greenboot Changes

MicroShift's greenboot scripts will be changed to include checks for the CSI Snapshot Controller and
WebhookValidation Pod.  Greenboot checks already exist to check the _topolvm-controller_, which includes the snapshot sidecar.

This will be implemented as the addition a the `kube-system` namespace to the list of checked namespaces in the
[microshift-running-checks.sh](https://github.com/openshift/microshift/blob/809c1b9182aa04a9d40ac101ed60cb0d7b6a9d09/packaging/greenboot/microshift-running-check.sh#L6).

### Deployment

CSI Snapshot controller manifests will be integrated into the CSI service-manager. The controller manifests will be
deployed as part of MicroShift's LVMS startup process.  Because LVMS does not depend on the controller to be
running prior to its deployment, it is not necessary to wait for the controller to reach a ready state before
starting LVMS.

### Test Plan

Smoke tests will be written and run as part of pre-submit tasks on MicroShift CI.  At a minimum, the tests should
verify that a VolumeSnapshot created from an existing PVC containing data:

1. results in the snapshotting of the underlying volume
2. is addressable and accessible via a consumer PVC
3. enables rollback of data on the original volume

### Graduation Criteria

#### Dev Preview -> Tech Preview

- Ability to utilize the enhancement end to end
- End user documentation, relative API stability
- Sufficient test coverage
- Gather feedback from users rather than just developers
- Write symptoms-based alerts for the component(s)

#### Tech Preview -> GA

- More testing (upgrade, downgrade, scale)
- Sufficient time for feedback
- Available by default
- Document SLOs for the component
- Conduct load testing
- User facing documentation created in [openshift-docs](https://github.com/openshift/openshift-docs/)

#### Removing a deprecated feature

### Upgrade / Downgrade Strategy

**Images**

The CSI Snapshot controller and sidecar are pieces of the Kubernetes CSI implementation.  OpenShift
maintains downstream versions of these images and tracks them as part of OCP releases.

**Manifests**

MicroShift's rebase automation specifies repos in the OpenShift github organization from which specific manifests
are obtained. For the CSI Snapshot Controller, we will specify
https://github.com/openshift/cluster-csi-snapshot-controller-operator/tree/release-$RELEASE/assets
as the remote source from which to derive the controller manifests and image references.

### Version Skew Strategy

The CSI Snapshot version is tracked by the ocp-release image.  This provides sufficient guardrails
against version skew. The images are versioned in step with the `volumeSnapshot` APIs.  Thus, we should always use
the image references from the same source as the CRD manifests to ensure compatibility.

### Operational Aspects of API Extensions

#### Failure Modes

- _Not Enough Storage Capacity:_ If the volume group capacity has been reached, snapshotting can fail because the
  storage driver refuses to create a snapshot.  This will be observable as the VolumeSnapshot being stuck in the
  **Pending** phase.

- _Valification Failure_: Webhook validation checks for malformed VolumeSnapshots and will post errors to the
  problem object as Events.

#### Support Procedures

_Troubleshooting_

To determine the cause of snapshot failures, use `oc describe` to examine VolumeSnapshot and VolumeSnapshotContent
events.

- **VolumeSnapshot:**  `$ oc describe volumesnapshot -n $NAMESPACE $NAME`
- **VolumeSnapshotContent:** `$ oc describe volumesnapshotconent $NAME`

Errors related to restoring a snapshot to a PVC will be captured in the PVC's events.

- **PersistentVolumeClaim:** `$ oc describe pvc -n $NAMESPACE $NAME`

If you need to delve deeper, the logs can be examined with the following commands:

- CSI Snapshot Controller: `$ oc logs -n kube-system csi-snapshot-controller-$SUFFIX`
- CSI External Snapshotter: `$ oc logs -n openshift-storage topolvm-controller-$SUFFIX csi-external-snapshotter`
- LVMS Controller: `$ oc logs -n openshift-storage topolvm-controller-$SUFFIX topolvm-controller`
- LVMS Node: `$ oc logs -n openshift-storage topolvm-node-$SUFFIX topolvm-node`
- LVMS LVM Daemon: `oc logs -n openshift-storage topolvm-node-$SUFFIX lvmd`

## Implementation History

## Alternatives

MicroShift design does not support CSI snapshotting.

repository: openshift/enhancements

repo_name: openshift/enhancements


text:
# jsHack

## Running

```
git clone git@github.com:Nebual/jsHack.git
cd jsHack
npm install
npm run build
```
Then host the public folder on a webserver, or open public/index.html locally.

* Its just a singleplayer game. Probably.

## Design Principles
* Encourage researching your targets
* Allow multiple routes so solve most obstacles
* Avoid 'press the decrypt button', instead 'figure out what you're looking at and what it means'

## Todo
* Terminal Commands
  * [ ] Add SSH to connect to another node
  * [ ] cp
  * [ ] mv
  * [ ] file editing (nano?)
* Saving
  * [ ] Serializing game state
  * [ ] Writing to localStorage
  * [ ] Loading from localStorage
  * [ ] Import/Exporting to save files
* IRC
  * [ ] Implement

repository: Nebual/jsHack

repo_name: Nebual/jsHack


text:
# v1.5 To v1.6 Migration

The v1.6 release of Delta includes a change on how project statistics are computed which requires to run a query against the Cassandra primary store.

## Delete the project statistics row in the projections_progress table

This command has to be run after stopping Delta 1.5 and before starting Delta 1.6

        DELETE FROM {delta_1_5}.projections_progress where projection_id = 'ProjectsCounts';

Note that as project statistics will be recomputed according to all events in the system, it may take some time for the statistics computation to reach back to the most recent events.

repository: samuel-kerrien/nexus

repo_name: samuel-kerrien/nexus


text:
## Back of the Line

Johnny Acorn is an elementary school student. In his class whenever the students line up to move to a new location, the teacher lines the students up in alphabetical order based on their last names. The teacher always leads the line. Johnny is often in the front of the line, but he is a mischievous boy. He wants to cause trouble at the back of the line.

Johnny is given the opportunity to change his last name given a word bank. He can apply a sequence of operations to the word bank before choosing one of the words for his last name.

- Reverse a word in the word bank.

- Concatenate two words in the word bank (in any order) and insert the result back into the word bank.

Help Johnny find the last name which would maximize his chances to be in the back of the line for his class. Johnny may keep `Acorn` as his last name.

Each word in the word bank will only consist of lowercase English letters but you must capitalize the first letter of the final last name.

[Wikipedia’s definition for alphabetical order](https://en.wikipedia.org/wiki/Alphabetical_order): To determine which of two strings comes first in alphabetical order, their first letters are compared. If they differ, then the string whose first letter comes earlier in the alphabet comes before the other string. If the first letters are the same, then the second letters are compared, and so on. If a position is reached where one string has no more letters to compare while the other does, then the first (shorter) string is deemed to come first in alphabetical order.

### Input
The input begins with an integer *n*,1 ≤ n ≤ 1,000, the initial size of the word bank. This is followed by *n* lines each containing a word in the word bank. Each word contains no more than 20 lowercase English letters.

### Output
Output the last name Johnny should choose.

#### Sample Input 1
```
2
apple
seed
```

#### Sample Output 1
```
Seedelppa
```

#### Sample Input 2
```
3
aa
ab
cca
```

#### Sample Output 2
```
Ccabaaa
```

#### Sample Input 3
```
2
aba
aaa
```

#### Sample Output 3
```
Acorn
```


text:
# PiCar (base) from SunFounder©
*can be use for PiCar-V or PiCar-S*

This project is intended to provide a CAD base to :
- Have a digital version;
- Make improvements;
- Export for simulators;
- Export for 3D printer;

# Changelog

## Version 2

Improved Support engine :
- Engine position fix ( 1 -> 3 points)
- Add Encoded wheel for more position accurate

## Version 1

Improved steering :
- Bearing assembled with clamping

## Version 0

Version without modification provided by SunFounder

# Structure repository

```
|- CAD : CAD files (freecad)
|- doc : Documentation
|- export : generated file for 3th app
| |- STL : for 3D printer / simulator
| |- DAE : for Simulator(ROS - Gasebo/Ignition)
```


text:
# Table of Contents

1.  [cellxgene proxy setup](#org8535249)
    1.  [etc/cellxgene_templates](#org70b949b)
        1.  [group_to_port.map defines group to port location](#org2593de5)
        2.  [mkcellxgene_config.pl](#org9d9afed)
        3.  [apache2 template](#orga8e4ab3)
        4.  [systemd.service template:](#org2413369)
    2.  [var/www/](#org90a423a)
    3.  [etc/apache2 - apache2 configs](#org352083b)
    4.  [cellxgene && cellxgene-gateway](#org0b8bc08)
    5.  [Basic usage](#orgbaa67c6)
    6.  [links](#orga301c60)

# cellxgene proxy setup

This setup provides example files and short description for setting up apache2 reverse proxy with
authentication based on mod_auth_form for cellxgene_gateway processes listening on the current or remote hosts.

## etc/cellxgene_templates

### group_to_port.map defines group to port location

### mkcellxgene_config.pl

-   script which reads etc/cellxgene/group_to_port.map file and templates and generates systemd, apache2 configs
    and updates html page for apache's mod_auth_form module
-   deletetion is not implemented
-   runs in root context, since it will write systemd and apache configs

### apache2 template

-   Ldap auth part is optional, this setup is using Openldap and groups with memberUid attribute.
    The <RequireAny/RequireAll> stanza allows users from different groups access only their cellxgene locations.
    The group sysop and defined users can access any resource.
-   See etc/apache2/sites-available/public.conf for the simplest config without authentication.

### systemd.service template:

-   needs properly defined user and a number of ENV variables.
-   `<group>` and `<port>` will be updated by the mkcellxgene_config.pl script
-   One might need to call  `systemctl enable cellxgene_<group>` to make services start on boot

## var/www/

-   html and css files for mod_auth_form
-   contains the main index.html updated by `mkcellxgene_config.pl`

## etc/apache2 - apache2 configs

-   list of enabled modules (in addition to the default ones) ( you might have different preferences!)
    `ssl,rewrite,headers,ldap,session,authnz_ldap,request,session_cookie,session_crypto,proxy,proxy_http,proxy_html`
-   sites-available/cellxgene.conf - the main config for http and https virtualhosts, also handles the logout action
    includes cellxgene/\* dir with group configs generated by `mkcellxgene_config.pl`
-   public.conf for the public section

## cellxgene && cellxgene-gateway

-   it is assumed that cellxgene and cellxgene-gateway are installed in the cellxgeneUser home dir

## Basic usage

1.  Create the data directory for the new group and set appropirate permissions.
    (see the CELLXGENE_DATA var in the systemd template)
2.  Update /etc/cellxgene_templates/group_to_port.map
3.  Call `/etc/cellxgene_templates/mkcellxgene_config.pl <group>` to setup systemd, apache2 config
    and /var/www/index.html
4.  Start the new systemd service: `service cellxgene_<group> start`
5.  Reload apache2 config: service apache2 reload
6.  Navigate to the url and test the application
7.  Debug errors with `journalctl -xe` and (or) apache2 error log files

## Links

-   cellxgene: <https://github.com/chanzuckerberg/cellxgene>
-   cellxgene-gateway: <https://github.com/Novartis/cellxgene-gateway>
-   mod_auth_form: <https://httpd.apache.org/docs/current/mod/mod_auth_form.html>


text:
# Covid Heroes
A memorial to those that have lost their lives fighting the COVID-19 pandemic.

---

### Data Resources
- [Covid Heroes Twitter feed](https://twitter.com/HeroesCovid)
- [Medscape Memorial](https://www.medscape.com/viewarticle/927976)
- [Memorial Page](https://nursingnotes.co.uk/covid-19-memorial/) from [Nursing Notes](https://twitter.com/NursingNotesUK/)
- [List of Italain medical professional casualties](https://portale.fnomceo.it/elenco-dei-medici-caduti-nel-corso-dellepidemia-di-covid-19/)
- [NYSNA Nurses Memorial](https://www.nysna.org/memoriam-fallen-nysna-nurses)
- https://twitter.com/Covid19Docs
- https://www.facebook.com/groups/317464005892297/

### Alternate Data Types
- [Transit Workers Union Memorial](http://www.twulocal100.org/memoriam)

### Former Data Resources
- ~~[Spreadsheet](https://docs.google.com/spreadsheets/d/1jlqsf3MVYyLBFewxO3MHkgGsXZOzrteo18t9ZljUIMg/edit#gid=1746956167) by [Jacques Fortier](https://twitter.com/jacquesgt)~~
- ~~[Spreadsheet](https://docs.google.com/spreadsheets/d/1pFdoZqjnDRaSzJi0JJJ3f5zdb87Q5tL3zc4nGx1nejI/edit#gid=1744604459) by [C. Michael Gibson MD](https://twitter.com/CMichaelGibson)~~
- ~~[Expanded bios from Design To Combat COVID-19](https://docs.google.com/document/d/13OGjxdv1MKCvRGWg8sHRvaU0LNoMNl2RNlmC6YoXRUw/edit)~~

---

## Running code locally
After cloning the repo, navigate to the correct folder and install Jekyll, and any necessary dependencies.

### Installation
Jekyll is a Ruby program so you need to install Ruby on your machine to begin with. Head over to the [install guide](https://jekyllrb.com/docs/installation/) and follow the instructions for your operating system.

With Ruby setup you can install Jekyll by running the following in your terminal:
```
gem install jekyll bundler
```

To create a new `Gemfile` to list your project’s dependencies run:
```
bundle init
```

Now edit the `Gemfile` and add jekyll as a dependency:
```
gem "jekyll"
```

Finally run `bundle` to install jekyll for your project.

You can now prefix all jekyll commands listed in this tutorial with `bundle exec` to make sure you use the jekyll version defined in your `Gemfile`.

The Airtable importer requires Node.js. Install Node using NVM.

### Build
Jekyll is a static site generator so we need Jekyll to build the site before we can view it. There are two commands you can run in the root of your site to build it:

- `jekyll build` - Builds the site and outputs a static site to a directory called `_site`.
- `jekyll serve` - Does the same thing except it rebuilds any time you make a change and runs a local web server at `http://localhost:4000`.

When you’re developing a site you’ll use jekyll serve as it updates with any changes you make.

Run `jekyll serve` and go to [http://localhost:4000](http://localhost:4000) in your browser.

### Notes
- The build will fail unless the `/_data/active.json` file already exists and has at least one record in it. The Airtable import script can overwrite the file, but cannot create it.
- At current, the site appears to need the build process to run twice before new profiles are added to the `active.json` file and then be built into individual pages.

repository: brandenbyers/covid-heroes

repo_name: brandenbyers/covid-heroes


text:
# Efficiensea2 Final Conference VTS/SRS-presentation
This repository contains a presentation that will be held at the Final Conference of EfficienSea2 in Copenhagen on the 5:th and 6:th of April 2018.

[Go to presentation](https://richardnilssonsma.github.io/)

Sign up for the Final Conference of EfficienSea2 here: https://efficiensea2.org/final-conference/

The presenation is made with the help of https://github.com/hakimel/reveal.js

Videos recorded with: https://linux.die.net/man/1/byzanz-record

Videos edited with: https://zulko.github.io/moviepy/index.html

Diagram drawn with: https://www.draw.io/


text:
# Wave-Generator
Analog signal generator, with USART control feature, intended for STM32F4 attached to custom circuit board. This was a final project for a microprocessors class taken during my last semester in college.

See docs/readme.pdf and docs/maintainer.pdf for information on the individual source files and program as a whole. Some files mentioned in docs/readme.pdf have been omitted, renamed and/or relocated, but most of the source code files are present and unchanged.

repository: tirish/Wave-Generator

repo_name: tirish/Wave-Generator


text:
# Mini-messenger frontend part

This project based on React framework.

Used technologies:
 - React
 - Material UI
 - Centrifuge (for websocket connections)
 - Webpack

To start:

```
npm run start
```

repository: Kyzyl-ool/mm-frontend

repo_name: Kyzyl-ool/mm-frontend


text:
# ApiKeyInfoProvider

## Registration and Implementation

In Rosmarin, the `ApiKeyInfoProvider` is the central class where the check takes place whether a request contains a
valid API key. The class responsible for this should be registered with the method `registerApiKeyInfoProvider` in
the `RestApplication`.

```typescript
const app: RestApplication = new RestApplication()

app.registerApiKeyInfoProvider(MyApiKeyInfoProvider)
```

The class `MyApiKeyInfoProvider` must implement the interface
[`IApiKeyInfoProvider`](src/api/api-key/api-key-info-provider.ts) and be annotated with the `@ApiKeyInfoProvider`
decorator.

```typescript
import { IApiKeyInfoProvider } from "./api-key-info-provider";

@ApiKeyInfoProvider()
class MyApiKeyInfoProvider implements IApiKeyInfoProvider {
  public async get(apiKeyHeader: ApiKeyHeader): Promise<ApiKeyInfo> {
    // access the header with the name 'x-api-key'
    const isValid: boolean = apiKeyHeader.getApiKey('x-api-key') === 's3cr3t'

    return new ApiKeyInfo(isValid)
  }
}
```

## Usage

The `ApiKeyInfoProvider` is automatically injected into each state. However, by default, no API key verification is performed.
You must activate it in the `configureState` method via the `activateApiKeyCheck` method.

```typescript
import { AbstractGetState } from "./abstract-get-state";

class MyState extends AbstractGetState {

  public configureState(): void {
    this.activateApiKeyCheck()
  }
}
```

If you have activated the check and the client has sent an invalid API key, it will automatically respond with a `401`
status code.

repository: matthyk/rosmarin

repo_name: matthyk/rosmarin


text:
Spree::TaxCloud
=======================

Spree::TaxCloud is a US sales tax extension for Spree using the Tax Cloud service.

Based on the work of Chris Mar and Drew Tempelmeyer.

TaxCloud Configuration
-----

Create an account with Tax Cloud...

[https://taxcloud.net](https://taxcloud.net)

...and get an `api_login_id` and `api_key`.

Spree Configuration
------------------------

Add the extension to your Gemfile:

```ruby
gem 'spree_tax_cloud', github: 'spree-contrib/spree_tax_cloud', branch: 'master'
```

And add it to your bundle:

```
bundle install
```

Run below to install migrations:

```
bundle exec rails g spree_tax_cloud:install
```

In the Admin section of Spree, go to Configurations, then select TaxCloud Settings.

Enter your `api_login_id` and `api_key`, and optionally your USPS login.
You can also configure the default Product TIC and Shipping TIC for TaxCloud to use, although it is recommended to leave the defaults as is: `00000` for product default and `11010` for shipping default.

All Products will default to the default product TIC specified here unless they are given an explicit value.
Specific product-level TICs may be specified per-product in the Products section of the Spree admin backend. If you are uncertain about the correct TIC for a product (whether it be clothing, books, etc.), taxability code information may be obtained from [Tax Cloud](https://taxcloud.net/tic/default.aspx).

To complete your Spree::TaxCloud configuration, you will need to create a TaxRate to apply rates obtained from Tax Cloud to your Spree LineItems and Shipments.
Under Configuration select Tax Rates, and click Create a New Tax Rate. Recommended defaults are as follows:

- Name: `Sales Tax` (This label will be visible to users during the checkout process)
- Zone: `USA` (Note that TaxCloud is only designed for United States sales tax)
- Rate: `0.0` (Note that the actual rates will be applied by the calculator)
- Tax Category: `Taxable`
- Included in Price: `False` (US taxes are 'additional' rather than 'included')
- Show Rate in Label: `False` (We will not display the static rate, which is left at `0%`)
- Calculator: `Tax Cloud`

Notes
------------------------

Spree::TaxCloud is designed to function in a single TaxCategory.
It is expected that all Products and all ShippingMethods will be in the same TaxCategory as the one configured for the TaxRate using the Tax Cloud calculator above (in this example, `Taxable`).

Spree::TaxCloud is designed to perform all US-related tax calculation itself, and as such does not use Spree configuration like TaxCategories to specify whether goods are Taxable, Tax-Exempt, Clothing, Food, etc.
Spree::TaxCloud does not use the Spree configuration `tax_address` (which specifies whether the shipping or billing address should be used to compute tax), instead _always_ using the shipping address if possible, and only falling back to the billing address if the shipping address is `nil`.
(Asking Spree::TaxCloud to compute orders whose shipping _and_ billing addresses are `nil` will result in an exception.)

TODO
----

Some work on the Spree:TaxCloud extension is ongoing. Namely:

- [x] Fill out a more complete set of feature specs using the test case scenarios provided by Tax Cloud.

- [ ] Address Validation: Support an address validation step via the Tax Cloud gem (which will fix the one non-passing spec at the moment).

- [ ] Split Shipments: Scope Tax Cloud transactions to Shipments rather than Orders, to account for the unusual cases where sales tax depends on the origin address as well as, or instead of, the destination address.

- [ ] Item Returns: Create feature specs and make the appropriate API calls to properly process sales tax on item returns.

- [ ] Promotions: Spree::TaxCloud is not (yet) fully compatible with some types of Spree promotions. For instance in cases such as "$10 off all orders over $100," it is not explicit how such a discount will affect the costs of individual items. In these cases, Spree::TaxCloud will fall back to charging sales tax on the full (undiscounted) item price.

Discussion and pull requests addressing this functionality are welcomed.

COPYRIGHT
---------

[Copyright]( http://jet.mit-license.org/ ) by Jerrold R Thompson