text:
# PettingZoo
A collection of sharded system recipes for [ZooKeeper][] written in Python.

[ZooKeeper]: https://github.com/apache/zookeeper

### Why Petting Zoo?
We need to be able to do stream processing of observations of student interactions with course material. This involves multiple models that have interdependent parameters. This requires:

* Sharding along different axes dependent upon the models
* Subscriptions between models for parameters
* Dynamic reconfiguration of the environment to deal with current load

How does this help us scale?

* Makes discovery of dependencies simple
* Adds to reliability of system by quickly removing dead resources
* Makes dynamic reconfiguration simple as additional resources become available

### Current State

* Relies on [zc.zk][] (will eventually be merged with the unified [Kazoo][] project)
* Documented, doc strings
* Tests (mock ZooKeeper)
* All recipes implemented in a Java version as well

[zc.zk]: https://github.com/python-zk/zc.zk
[Kazoo]: https://github.com/python-zk/kazoo

### Recipes

* [Distributed Discovery](#distributed-discovery)
* [Distributed Bag](#distributed-bag)
* [Leader Queue](#leader-queue)
* [Role Match](#role-match)

* * *

## Distributed Discovery
Allows services in a dynamic, distributed environment to be able to be quickly alerted of service address changes.

* Most service discovery recipes only contain host:port, Distributed Discovery can share arbitrary data as well (using yaml)
* Can handle load balancing through random selection of config
* Handles rebalancing on pool change

### Why Distributed Discovery?

* Makes discovery of dependencies simple
* Adds to reliability of system by quickly removing dead resources
* Makes dynamic reconfiguration simple as additional resources become available

### DD Example: Write

	from pettingzoo.discovery import write_distributed_config
	from pettingzoo.utils import connect_to_zk
	conn = connect_to_zk('localhost:2181')
	config = {
		'header': {
			'service_class': 'kestrel', 'metadata': {
				'protocol': 'memcached', 'version': 1.0
			}
		},
		'host': 'localhost',
		'port': 22133
	}
	write_distributed_config(conn, 'kestrel', 'platform', config)

### DD Example: Read

	from pettingzoo.discovery import DistributedMultiDiscovery
	from pettingzoo.utils import connect_to_zk
	conn = connect_to_zk('localhost:2181')
	dmd = DistributedMultiDiscovery(conn)
	conf = None
	def update_configs(path, updated_configs):
		conf.update(updated_configs)
	conf = dmd.load_config('kestrel', 'platform', callback=update_configs)

* * *

## Distributed Bag
A recipe for a distributed bag (dbag) that allows processes to share a collection.  Any participant can post or remove data, alerting all others.

* Used as a part of [Role Match](#role-match)
* Useful for any case where processes need to share configuration determined at runtime

![dbag](http://i.imgur.com/CJgip.png)

### Why Distributed Bag?

* Can quickly alert processes as to who is subscribing to them
* Reduces load by quickly yanking dead subscriptions
* Provides event based subscriptions, making implementation simpler

### Dbag Details

* Sequential items contain the actual data
* Can be ephemeral
* Clients set delete watch on discrete items
* Token is set to id of highest item
* Clients set a child watch on the "Tokens" node
* Can determine exact adds and deletes with a constant number of messages per delta

![Distributed Bag diagram](http://imgur.com/iqcTT.png)

### Dbag Example

	import yaml
	from pettingzoo.dbag import DistributedBag
	from pettingzoo.utils import connect_to_zk

	...
	conn = connect_to_zk('localhost:2181')
	dbag = DistributedBag(conn, '/some/data')
	docs = {}
	def acb(cb_dbag, cb_id):
		docs[cb_id] = cb_dbag.get(cb_id)
	def rcb(cb_dbag, cb_id):
		docs.remove(cb.id)

	dbag.add_listeners(add_callback=acb, remove_callback=rcb)
	add_id = dbag.add(yaml.load(document), ephemeral=True)
	docs[add_id] = document

* * *

## Leader Queue

Recipe is similar to Leader Election, but makes it easy to monitor your spare capacity.

* Used in [Role Match](#role-match)
* As services are ready to do work, they create an ephemeral, sequential node in the queue.
* Any member always knows if either they are in the queue or at the front
* Watch lets leader know when it is elected

### Why Leader Queue?

* Gives a convenient method of assigning work
* Makes monitoring current excess capacity easy

### LQ Details

* Candidates register with sequential, ephemeral nodes
* Candidate sets delete watch on predecessor
* Candidate is elected when it is the smallest node
* When elected, candidate takes over its new role
* When ready, candidate removes itself from the queue
* Only one candidate needs to call get_children upon any node exiting

![Leader Queue diagram](http://imgur.com/sikWd.png)

### LQ Example

	from pettingzoo.leader_queue import LeaderQueue, Candidate
	from pettingzoo.utils import connect_to_zk

	class SomeCandidate(Candidate):
		def on_elected(self):
			<do something sexy>

	conn = connect_to_zk('localhost:2181')
	leaderq = LeaderQueue(conn)
	leaderq.add_candidate(SomeCandidate())

* * *

## Role Match

Allows systems to expose needed, long lived jobs, and for services to take over those jobs until all are filled.

* Dbag used to expose jobs
* Leader queue used to hold applicants
* Records which jobs are presently held with ephemeral node
* Lets a new process take over if a worker dies
* We use it for sharding/segmentation to dynamically adjust the shards as needed due to load

### Why Role Match?

* Core of our ability to dynamically adjust shards
* Lets the controlling process adjust problem spaces and have those tasks become automatically filled
* Monitoring is easy to identify who is working on what, when

### RM Details

* Leader monitors for open jobs
* Job holder creates an ephemeral assignment
* Assignment id matches job id, indicating that it is claimed

![Role Match diagram](http://imgur.com/CmsHs.png)

* * *

## Future: Distributed Config

* Allows service config to be recorded and changed with a yaml config
* Every process that connects creates a child node of the appropriate service
* Any change in a child node's config overrides the overall service config for that process
* Any change of the parent or child fires a watch to let the process know that it's config has changed

repository: urjitbhatia/pettingzoo-python

repo_name: urjitbhatia/pettingzoo-python


text:
# DRAWR (Drawing Really Awful Works of aRt)
Meet DRAWR, the really bad artist. DRAWR learns to draw by looking at images. Sadly, while he may have a good memory, he doesn't really understand what actually makes an image art... DRAWR records the colors of neighboring pixels and uses a Markov algorithm to generate new images.

Some things that DRAWR can and can't do:
* DRAWR is able to generate images.
* DRAWR is able to learn how to draw from images (jpg, png).
* DRAWR is NOT able to actually understand what is sees in images (i.e. it cannot understand what a head is).

As a warning, DRAWR IS VERY SLOW. One of the issues with DRAWR is that there are about 4 billion pixel combinations (about 16.5 million combinations without alpha); add on the generation method and it becomes really slow. DRAWR will take a long time if you are generating a very large image or the source image has a large number of unique pixels (colors).

## The Future of DRAWR
I'd really like to expand the scope of DRAWR. Currently, the program uses a simple Markov algorithm to generat images.

First, I'd like to expand the depth of the markov chain. At a depth of 1, the algorithm only looks at the pixel to the left and above when generating a new point. At a depth of n, the algorithm would look left and up in lines of length n pixels.

Second, I want to add an optional web interface for DRAWR. This will provide an easy to use GUI that will also be able to actually render the images generated. Not to mention, it will make it easier to host on a remote server (Note: DRAWR's web interface will likely not be very secure or complex. It is not really intended to be used as a functioning service. If you desire to have DRAWR publically accessible, it will be reccomended to have another service handle routing traffic to DRAWR).

Third, I would like to add an option to have DRAWR generate images based on square chunks rather than lines. This would be really slow, but it would likely make DRAWR generate much better looking images.

Next, I'd like to add the ability for DRAWR to pull images a url. First itterations would likely require the url to link directly to the image, but later versions would probably be able to search a page for images.

Finally, I'd like to add a mode to let DRAWR look at source images more analytically. Namely, DRAWR would find groups of similarly colored pixels and store the geometry along with color data. The geometries would be categorized as "similar looking" based on size and shape. Perhaps, DRAWR would ask for the user to define tags for the geometries? DRAWR would then decide on a starting geometry and use said geometry to generate more geometries in the image. Once DRAWR has finished with geometries, it would fill the geometries and decide upon colorization for the rest of the image using the combined data of the geometries. This would hopefully allow for images that actually look like something...

## Compiling
DRAWR is written in Go (so you need a go compiler). DRAWR does not have any external dependencies.

A simple `go build boot.go` should spit out an executable.

## Using DRAWR
DRAWR has three major components. Put together, they form the DRAWR executable. For the most part, you'll only need to really care for the generator compenent.

If desired, you can use these components without the DRAWR front end.

### Generators
Generators are the compenents that actually generate the image. Generators are designed to support multiple generation modes. Different generators are able to utilize different types of containers. This is done to allow for more flexible generators (i.e. for different types of data) as well as user defined generators.

Currently, DRAWR has one generator implemented:
* SimpleGenerator

#### SimpleGenerator
This generator is designed to handle only the data above and left. It has 6 modes:
* 0 -- SimpleGeneratorModeStandard: Adds all data for the above and left pixel without doing any scaling.
* 1 -- SimpleGeneratorModeExclusive: When possible, it will use only pixels that exist in data for both the above and left pixels. If not possible, it defaults to left pixel data then above pixel data. Great for patterns.
* 2 -- SimpleGeneratorModeMultiplicative: Makes values which appear in data for both the upper and left pixels have a multiplicative bonus (about 100x bonus).
* 3 -- SimpleGeneratorModeSuperMultiplicative: Makes values which appear in data for both the upper and left pixels have a multiplicative bonus (about 2000x bonus).
* 4 -- SimpleGeneratorModeInverseMultiplicative: Makes values which appear in data for both the upper and left pixels have an inverse multiplicative bonus (about 1/5x bonus). Usually more chaotic.
* 5 -- SimpleGeneratorModeInverseSuperMultiplicative: Makes values which appear in data for both the upper and left pixels have an inverse multiplicative bonus (about 1/250x bonus). Usually more chaotic.

## License
DRAWR is licensed under an MIT license.

license: MIT

repository: joemahmah/DRAWR

branch_name: refs/heads/master

extension: md

license_type: permissive

repo_name: joemahmah/DRAWR


text:
#How to develop his own plugin

__This feature is only possible with Hadoop Unit v3.x.__

There are 2 classes which have to be implemented:

* a class that implements `Bootstrap` (from module `hadoop-unit-commons`) or `BootstrapHadoop` (from module `hadoop-unit-commons-hadoop`)
* a class that implements `ComponentMetadata` (from module `hadoop-unit-commons`)

Moreover, as _Service Provider Interface_ is used, the file `META-INF/services/fr.jetoile.hadoopunit.component.Bootstrap` has to be present with the reference to the class which implements the interface `Bootstrap` or `BootstrapHadoop`.

#Interface Bootstrap

This interface define 6 methods:

* `start()` which is called when the plugin is started
* `stop()` which is called when the plugin is stopped
* `getProperties()` which is called when the banner is printed
* `loadConfig(Map<String, String> configs)` which is called to get overrided properties by the maven's plugin
* `getMetadata()` which return the plugin's metadata as dependencies and so on
* `getName()` which is a default method which lookup into the implementation of ComponentMetadata to get the plugin name

![elastic_class](https://blog.jetoile.fr/hadoop-unit/images/elastic_class.png)

#Interface BootstrapHadoop

This interface extends the interface `Bootstrap` but declare the method `getConfiguration()` which return an instance of `org.apache.hadoop.conf.Configuration`.

![bootstrap_class](https://blog.jetoile.fr/hadoop-unit/images/bootstrap_class.png)

#Interface ComponentMetadata

This interface defines 2 methods:

* `getName()` which gives the plugin's name
* `getDependencies()` which lists the plugins the current plugin is dependent

![componentMetadata_class](https://blog.jetoile.fr/hadoop-unit/images/componentMetadata_class.png)

#Add a plugin to Hadoop Unit Standalone

To add a plugin to Hadoop Unit in the standalone mode, add the following information into the file `hadoop-unit-default.properties`:

* `<pluginName>.artifactId` which must have the format `<groupId>:<artifactId>:<version>`
* `<pluginName>.mainClass` which must indicate the implementation of `Bootstrap` or `BootstrapHadoop`
* `<pluginName>.metadataClass` which must indicate the implementation of `ComponentMetadata`

Note : the plugin's jar has to be in your repository manager (or on maven central) as it will be searched and downloaded.

Sample of `hadoop-unit-default.properties`:

```bash
hdfs.artifactId=fr.jetoile.hadoop:hadoop-unit-hdfs:3.1
hdfs.mainClass=fr.jetoile.hadoopunit.component.HdfsBootstrap
hdfs.metadataClass=fr.jetoile.hadoopunit.component.HdfsMetadata
```

Add your plugin's name into the file `hadoop.properties`.

Sample of `hadoop.properties`:

```bash
<pluginName>=true
```

Note that your plugin name has to be in lowercase and that his name declared into the ComponentMetadata's class has to be in uppercase.

#Add a plugin to Hadoop Unit's maven plugin

To add a plugin to Hadoop Unit's maven plugin, add your plugin into the `configuration` element as shown below:

```xml
<build>
    <plugins>
        <plugin>
            <artifactId>hadoop-unit-maven-plugin</artifactId>
            <groupId>fr.jetoile.hadoop</groupId>
            <version>${hadoop-unit.version}</version>
            <executions>
                <execution>
                    <id>start</id>
                    <goals>
                        <goal>embedded-start</goal>
                    </goals>
                    <phase>pre-integration-test</phase>
                </execution>
                <execution>
                    <id>embedded-stop</id>
                    <goals>
                        <goal>embedded-stop</goal>
                    </goals>
                    <phase>post-integration-test</phase>
                </execution>
            </executions>
            <configuration>
                <components>
                    <componentArtifact implementation="fr.jetoile.hadoopunit.ComponentArtifact">
                        <componentName>HDFS</componentName>
                        <artifactId>hadoop-unit-hdfs</artifactId>
                        <groupId>fr.jetoile.hadoop</groupId>
                        <version>${hadoop-unit.version}</version>
                        <mainClass>fr.jetoile.hadoopunit.component.HdfsBootstrap</mainClass>
                    </componentArtifact>
                </components>
            </configuration>
        </plugin>
    </plugins>
</build>
```

Note : the plugin's jar has to be in your repository manager (or on maven central) as it will be searched and downloaded.

#HDFS Sample

This diagram shows the hdfs' plugin class diagram:

![hdfs_class](https://blog.jetoile.fr/hadoop-unit/images/hdfs_class.png)

This diagram shows a class diagram with multiple plugins:

![multiple_class](https://blog.jetoile.fr/hadoop-unit/images/multiple_class.png)

repository: CleverCloud/hadoop-unit

repo_name: CleverCloud/hadoop-unit


text:
# Contributing to `react-native-screens`

### Welcome

Thank you for considering contributing to `react-native-screens`. It's people like you that make open source projects thrive! We love to receive contributions from our community and there are many ways for *you* to be a part of this.

_tl;dr **Please** do not ignore templates for issues and pull requests and everything will be fine_

### Found a bug?

If you’ve encountered a bug, don't hesitate to submit [an issue](https://github.com/software-mansion/react-native-screens/issues). Just check if someone didn't report it lately.

When filing an issue, make sure to provide:
- a short description of the problem
- an operating system that you are currently working on
- what happened
- what you expected to happen
- **snippet of code or Snack that reproduces the bug**. Check out [this guide](https://stackoverflow.com/help/minimal-reproducible-example)
- version of `react-native` and `react-native-screens`

We've provided a template on GitHub that simplifies the process of filing an issue.

Following these few steps will show great respect for the time of the developers managing and developing this open-source project.

We inform you that unrespectful issues will be closed.

### Got a question or an idea for a feature?

We use GitHub issues exclusively for tracking bugs. For questions and feature requests check out [Discussions](https://github.com/software-mansion/react-native-screens/discussions).

### Pull requests

Pull requests are more than welcome 🚀

We've created a template to help you with the process of submitting a pull request to this project. Remember to change or add documentation as needed and provide a code snippet (preferably added to `TestsExample/` project) that the developers could test your change on. In the PR template we've added a checklist for all the places where the documentation should be updated.

If it's your first pull request [this guide](https://github.com/MarcDiethelm/contributing/blob/master/README.md) might be helpful for you.

### Naming convensions

For commits and pull request names we follow a [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) specification.

For branch names we use `@username/description-separated-by-dashes` convention.

repository: software-mansion/react-native-screens

branch_name: refs/heads/main

gha_language: TypeScript

gha_license_id: MIT

repo_name: software-mansion/react-native-screens


text:
# Subscribe to Android broadcasts

[![](https://jitpack.io/v/I60R/BroadcastRx.svg)](https://jitpack.io/#I60R/BroadcastRx)

--------
--------

```java
        BroadcastRx.init(anyContext);

        BroadcastRx
                .local("my.app.ACTION")
                .mergeWith(BroadcastRx.global(Intent.ACTION_CONFIGURATION_CHANGED))

                    ...

                .subscribe(broadcast -> {
                    Log.d("received", broadcast.action);
                    Log.d("hasExtra", String.valueOf(broadcast.extras != null));

                        ...

                });
```

--------
--------

```xml
        <receiver android:name="i60r.broadcastrx.BroadcastRx">
            <intent-filter>
                <action android:name="android.intent.action.ACTION_POWER_CONNECTED" />
            </intent-filter>
        </receiver>
```

```java
        BroadcastRx
                .local(Intent.ACTION_POWER_CONNECTED)
                .subscribe(broadcast -> {

                    ...

                });
```

--------
--------

```java
        BroadcastRx
                .hook(upstream -> upstream
                        .doOnNext(broadcast -> Log.d("( ( ( (", broadcast.action)));
```


text:
# Incognitus Feature Flag (Vue 2 & 3)

![Continuous Integration](https://github.com/Incognitus-Io/client-vue/workflows/Continuous%20Integration/badge.svg)
[![codecov](https://codecov.io/gh/Incognitus-Io/client-vue/branch/master/graph/badge.svg?token=HJ4XoCv8oZ)](https://codecov.io/gh/Incognitus-Io/client-vue)

## Usage

Install this as your projects's dependency

### Vue 2

```
npm i @incognitus/client-vue
# or
yarn add @incognitus/client-vue
```

### Vue 3

```
npm i @incognitus/client-vue@next
# or
yarn add @incognitus/client-vue@next
```

## Installing the plugin

Before you're able to use the service you'll need to install the plugin with your tenant and application IDs.

### main.ts - vue 2

```typescript
import { incognitus } from '@incognitus/client-vue';

Vue.use(incognitus, {
  tenantId: '{your tenant key}',
  applicationId: '{your app id}',
});
```

### main.ts - vue 3

```typescript
import { incognitus } from '@incognitus/client-vue';

createApp(App)
  .use(incognitus, {
    tenantId: '{your tenant key}',
    applicationId: '{your app id}',
  })
  .mount('#app');
```

| Key           | Description                               |
| ------------- | ----------------------------------------- |
| tenantId      | Your tenant id                            |
| applicationId | The id of the application and environment |

## Checking features

### Component

You can use the included component to check feature flags. This is transparent and will not add an extra depth in the
DOM. Using slots, you can configure different aspects like the loading state, or if the flag is enabled/disabled.

```html
<template>
  <feature-flag flag="feature name">
    <template #loading> Checking features </template>
    <template #enabled> Something behind the flag </template>
    <template #disabled> The old implementation </template>
  </feature-flag>
</template>
```

#### Props

| Prop   | Description                           |
| ------ | ------------------------------------- |
| flag   | The name of the feature flag          |
| hidden | Hide content when the flag is enabled |

#### Slots

| Slot     | Description                              |
| -------- | ---------------------------------------- |
| loading  | Displayed when fetching the feature flag |
| enabled  | Rendered when the flag is enabled        |
| disabled | Rendered when the flag is disabled       |

### Composition API hook

If you would like to interact with the service directly, then it can be accessed through the `useIncognitus` hook.
This returns `isReady` which returns `true` when the service is initiatlized and ready to be used, and `service`
which returns the raw service

#### Service methods

| Method                          | Description                                                 |
| ------------------------------- | ----------------------------------------------------------- |
| service.isEnabled(featureName)  | Checks if the flag is enabled                               |
| service.isDisabled(featureName) | Check if the flag is disabled                               |
| service.getFeature(featureName) | Fetches the feature from the server and returns it's status |
| service.getAllFeatures()        | Fetches all features and stores them in the cache           |

### Service

You can also use the service directly by importing `IncognitusService.instance`, however this is not recommended.

## Caching

Currently all known feature flags are cached when the app initializes. New features that are not found
in the cache are retrieved on-demand. The cache stays in place until the app is reloaded or by calling the `getAllFeatures()` method on the service.

repository: Incognitus-Io/client-vue

repo_name: Incognitus-Io/client-vue


text:
---
layout: post
title:      "Tweet Sentiment Analysis"
date:       2020-10-06 17:52:14 -0400
permalink:  mod_3_project
---

Repository Link: https://github.com/singsang2/dsc-mod-4-project-v2-1-onl01-dtsc-ft-070620/

    *Major vocabularies used in positive sentiments and negative sentiments.

## Business Model

Consumers' voices are getting much attention by big companies not only because their opnions matter, but also their reach of their voice is expanding due to growth of social media platforms such as Facebook, Instagram and Twitter. Thus, it is imperative to monitor and analyze how consumers are responding to one's product not only for customer service purpose, but also to analyze their trends.

In this project, we will deal with a portion of what we have mentioned above. We will focus on identifying negative sentiments of Google, Apple and Android products from Twitter data.

### Goals
The goal of this project is to produce a model that can efficiently

[1] identify tweets with negative sentiments towards a product (ex. Apple, Google, or Android products), and
-  By identifying negative tweets, a company can provide relatively quick feedback to the customers which will improve brand's image and credibility.

[2] correctly categorize tweets into negative, neutral and positive sentiments.
- By cumulating various sentiments about a company's various products, a company could read and predict people's trend.

[Extra] We will also train a model that can identify which product a tweet is mentioning.
- A company/brand usually has more than just one product (ex. Apple), so by correctly identifying which product tweets are mentioning, it will be more beneficial to the company be aware of their products review by people.

## Data

 [1]. The main dataset comes from CrowdFlower via data.world. Data Link

 [2]. Amazon and Best Buy electronics review dataset. Data Link

 [3]. Amazon electronics review dataset: Data Link

# Models

## Binary Model

The following methods were used to prepare our binary classification text data:

    [1] NLTK - TF-IDF vectorizer
    [2] spaCy - TF-IDF vectorizer
    [3] spaCy - word2vec word embedding.

Then either `MultinomialNB` or `LogisticRegression` were used to train the model.

## Multiclass Model

The following methods were used to prepare our multiclass classification text data:

    [1] spaCy - TF-IDF vectorizer
    [2] TextBlob - sentiment analyzer
    [3] spaCy - word2vec word embedding.
    [4] Added additional datasets from Amazon and Best Buy

Then either `MultinomialNB` or `LogisticRegression` were used to train the model.

# Model Evaluation

## Binary Model

The following table shows the top 5 model that performed for binary classification.

Logistic Regression model using word embedding did the best in terms for negative recalls; however it does have a longer average time to fit and predict the result signifcantly.

The best model was using `MulticlassNB` with spaCy's word embedding.

- Overall Test Accuracy: 81%

- Overall Test Recall (weighted): 81%

- Negative Sentiment Recall: 74%

## Multiclass Model

The best model was using MulticlassNB with spaCy's word embedding.

- Overall Test Accuracy: 60%

- Overall Test Recall (weighted): 60%

- Negative Sentiment Recall: you 71%

# Conclusion

Our final models can

    [1] Binary Sentiment
        * Most 81% accuracy with 76% negative sentiment recall
    [2] Multiclass Sentiment
        * 60% accuracy with 71% negative sentiment recall
    [EXTRA] Product Predictor
        * 91% of accuracy

## Recommendations
We would recommend the following:

[1] **Use binary sentiments (positive and negative)**
   - Multiclass not only complicates the model itself, but also it dilutes negative and positive sentiments together, therefore, we recommend using binary sentiment classification model

[2] **For the best negative recall model:** *logistic regression word-embedding model*
   - this is the best model to flag negative sentiments but it does take some time to train; however, as long as the train is well-done, implementing the model should not cause significant calculation cost. (Further study should be done)

[3] **For the fastest computing model:** *Naive Bayes' TF-IDF model with SMOTE*
   - this model also did a great job in identifying negative sentiments, but it uses SMOTE which is not recommended in NLP analysis.

# Future Direction

[1] Further data acquisition:
Due to imbalance in data, our model suffered. Acquiring similar datasets with more balanced classes would increase our model's performance and reliability.

[2] Real time Twitter analysis
Have this model refined and put in a production so that it can monitor tweets real time and analyze and negative flags about any particular product.

[3] Research various costs between models
Different models have different strengths and weaknesses, and we believe researching different costs including computing and labor costs using different models would allow us to choose a better model

[4] Try different NLP techniques
There are so many different NLP techniques and they are growing due to works of so many different research entities. It would be very valuable to try different approach including skip-gram and even deep learning NLP for this project.

extension: markdown


text:
# flask app for s3 upload

This is a simple python-flask app to upload a file to AWS S3.

# How to Install

1. Please fork or clone this repository
> https://github.com/JustinDPerkins/flask.git

2. Create a Dockerfile
```bash
FROM python:3.6

# set a directory for the app
WORKDIR /usr/src/app

# copy all the files to the container
COPY . .

ENV S3_BUCKET_NAME=<your bucket name>
ENV S3_ACCESS_KEY=<your access key>
ENV S3_SECRET_ACCESS_KEY=<your secret key>

# install dependencies
RUN pip install --no-cache-dir -r requirements.txt

# tell the port number the container should expose
EXPOSE 5000

# run the command
CMD ["python", "./wsgi.py"]
```

3. Deploy

repository: JustinDPerkins/flask

repo_name: JustinDPerkins/flask


text:
---
title: CI/CD Setup Guide
sidebar_label: Introduction
---

Or _"The steps for reaching Pact Nirvana"_

This is a technical guide for developers and testers who want to use Pact to implement consumer driven contract testing.

By the end of the guide, you will understand how to create a release pipeline that allows you to independently deploy any application with the confidence that it will work correctly with the other applications in its environment - without having to run a suite of end to end tests.

Although many guides assume a greenfields project with no existing code, this document is intended to also be useful for retrofitting Pact into an existing project.

Before you read this document, you should:

* have a basic understanding of the concepts of both [consumer driven contracts](https://martinfowler.com/articles/consumerDrivenContracts.html) and Pact,
* have read the [Pact Broker Overview](pact_broker/overview.md)
* have read the section on [versioning in the Pact Broker](getting_started/versioning_in_the_pact_broker.md)

### How to use this document

Each integration is different. Each organisation has different history and culture, and each team may have different processes for development, testing, and deployment. Each of these differences affect the best choices for Pact workflow.

However, there are many similarities in the steps necessary on the journey to a full-featured and effective Pact setup \(_"Pact Nirvana"_\). This document describes those steps.

Feel free to pick and choose the steps that apply best to your team. You may implement only the first few steps described below, and just use Pact as a precursor to your standard integration tests; or you may throw away your integration tests altogether and reach "Pact Nirvana".

As Pact has been implemented in many different languages, this document will outline the theory for each step. You will need to consult the documentation of your chosen language to learn the relevant syntax for each task. See the [implementation guides](/implementation_guides/cli) section for more information.

### What are the steps for reaching Pact Nirvana?

1. Talk
2. Spike \(Optional\)
3. Write and verify a real Pact
4. Automate the contract and verification results exchange
5. Allow contracts to change without breaking your builds
6. Use tags to ensure your provider is compatible with your production consumer
7. Use The Pact “Matrix” as a gateway to deployments
8. Prevent missing verifications


text:
```text
SPDX-License-Identifier: Apache-2.0
Copyright (c) 2020 Intel Corporation
```
<!-- omit in toc -->
# Using ACC100 eASIC in Smart Edge Open: Resource Allocation, and Configuration
- [Overview](#overview)
- [Intel vRAN Dedicated Accelerator ACC100 FlexRAN Host Interface Overview](#intel-vran-dedicated-accelerator-acc100-flexran-host-interface-overview)
- [Intel vRAN Dedicated Accelerator ACC100 Orchestration and Deployment with Kubernetes\* for FlexRAN](#intel-vran-dedicated-accelerator-acc100-orchestration-and-deployment-with-kubernetes-for-flexran)
- [Using the Intel vRAN Dedicated Accelerator ACC100 on Smart Edge Open](#using-the-intel-vran-dedicated-accelerator-acc100-on-smartedge-open)
  - [ACC100 (FEC) Ansible Installation for Smart Edge Open Network Edge](#acc100-fec-ansible-installation-for-smartedge-open-network-edge)
    - [Converged Edge Experience Kits](#converged-edge-experience-kits)
  - [FEC VF configuration for Smart Edge Open Network Edge](#fec-vf-configuration-for-smartedge-open-network-edge)
  - [Requesting Resources and Running Pods for Smart Edge Open Network Edge](#requesting-resources-and-running-pods-for-smartedge-open-network-edge)
  - [Verifying Application POD Access and Usage of FPGA on Smart Edge Open Network Edge](#verifying-application-pod-access-and-usage-of-fpga-on-smartedge-open-network-edge)
- [Reference](#reference)

## Overview

Intel® vRAN Dedicated Accelerator ACC100 plays a key role in accelerating 4G and 5G Virtualized Radio Access Networks (vRAN) workloads, which in turn increases the overall compute capacity of a commercial, off-the-shelf platform.

Intel® vRAN Dedicated Accelerator ACC100 provides the following features:

- LDPC FEC processing for 3GPP 5G:
  - LDPC encoder/decoder
  - Code block CRC generation/checking
  - Rate matching/de-matching
  - HARQ buffer management
- Turbo FEC processing for 3GPP 4G:
  - Turbo encoder/decoder
  - Code block CRC generation/checking
  - Rate matching/de-matching
- Scalable to required system configuration
- Hardware DMA support
- Performance monitoring
- Load balancing supported by the hardware queue manager (QMGR)
- Interface through the DPDK BBDev library and APIs

Intel® vRAN Dedicated Accelerator ACC100 benefits include:
- Reduced platform power, E2E latency and Intel® CPU core count requirements as well as increase in cell capacity than existing programmable accelerator
- Accelerates both 4G and 5G data concurrently
- Lowers development cost using commercial off the shelf (COTS) servers
- Accommodates space-constrained implementations via a low-profile PCIe* card form factor.
- Enables a variety of flexible FlexRAN deployments from small cell to macro to Massive
MIMO networks
- Supports extended temperature for the most challenging of RAN deployment scenarios

For more information, see product brief in [Intel® vRAN Dedicated Accelerator ACC100](https://builders.intel.com/docs/networkbuilders/intel-vran-dedicated-accelerator-acc100-product-brief.pdf).

This document explains how the ACC100 resource can be used on the Open Network Edge Services Software (Smart Edge Open) platform for accelerating network functions and edge application workloads. We use the Intel® vRAN Dedicated Accelerator ACC100 to accelerate the LTE/5G Forward Error Correction (FEC) in the 5G or 4G L1 base station network function such as FlexRAN.

FlexRAN is a reference layer 1 pipeline of 4G eNb and 5G gNb on Intel® architecture. The FlexRAN reference pipeline consists of an L1 pipeline, optimized L1 processing modules, BBU pooling framework, cloud and cloud-native deployment support, and accelerator support for hardware offload. Intel® vRAN Dedicated Accelerator ACC100 card is used by FlexRAN to offload FEC (Forward Error Correction) for 4G and 5G.

## Intel vRAN Dedicated Accelerator ACC100 FlexRAN Host Interface Overview
Intel® vRAN Dedicated Accelerator ACC100 card used in the FlexRAN solution exposes the following physical functions to the CPU host:
- One FEC interface that can be used of 4G or 5G FEC acceleration
  - The LTE FEC IP components have turbo encoder/turbo decoder and rate matching/de-matching
  - The 5GNR FEC IP components have low-density parity-check (LDPC) Encoder / LDPC Decoder, rate matching/de-matching, and UL HARQ combining

![Intel® vRAN Dedicated Accelerator ACC100 support](acc100-images/acc100-diagram.png)

## Intel vRAN Dedicated Accelerator ACC100 Orchestration and Deployment with Kubernetes\* for FlexRAN
FlexRAN is a low-latency network function that implements the FEC. FlexRAN uses the FEC resources from the ACC100 using POD resource allocation and the Kubernetes\* device plugin framework. Kubernetes* provides a device plugin framework that is used to advertise system hardware resources to the Kubelet. Instead of customizing the code for Kubernetes* (K8s) itself, vendors can implement a device plugin that can be deployed either manually or as a DaemonSet. The targeted devices include GPUs, high-performance NICs, FPGAs, InfiniBand\* adapters, and other similar computing resources that may require vendor-specific initialization and setup.

![Intel® vRAN Dedicated Accelerator ACC100 Orchestration and deployment with Smart Edge Open Network Edge for FlexRAN](acc100-images/acc100-k8s.png)

_Figure - Intel® vRAN Dedicated Accelerator ACC100 Orchestration and deployment with Smart Edge Open Network Edge for FlexRAN_

## Using the Intel vRAN Dedicated Accelerator ACC100 on Smart Edge Open
Further sections provide instructions on how to use the ACC100 eASIC features: configuration and accessing from an application on the Smart Edge Open Network Edge.

When the Intel® vRAN Dedicated Accelerator ACC100 is available on the Edge Node platform it exposes the Single Root I/O Virtualization (SRIOV) Virtual Function (VF) devices which can be used to accelerate the FEC in the vRAN workload. To take advantage of this functionality for a cloud-native deployment, the PF (Physical Function) of the device must be bound to the DPDK IGB_UIO userspace driver to create several VFs (Virtual Functions). Once the VFs are created, they must also be bound to a DPDK userspace driver to allocate them to specific K8s pods running the vRAN workload.

The full pipeline of preparing the device for workload deployment and deploying the workload can be divided into the following stages:

- Enabling SRIOV, binding devices to appropriate drivers, and the creation of VFs: delivered as part of the Edge Nodes Ansible automation.
- Queue configuration of FPGAs PF/VFs with an aid of [pf-bb-config](https://github.com/intel/pf-bb-config) utility: Docker\* image creation delivered as part of the Edge Nodes Ansible automation. The images being pushed to a local Harbor registry, sample pod (job) deployment via Helm charts.
- Enabling orchestration and allocation of the devices (VFs) to non-root pods requesting the devices: leveraging the support of "accelerator" SRIOV VFs from K8s SRIOV Device Plugin. K8s plugin deployment is delivered as part of the Edge Controller's Ansible automation.
- Simple sample BBDEV application to validate the pipeline (i.e., SRIOV creation - Queue configuration - Device orchestration - Pod deployment): Script delivery and instructions to build Docker image for sample application delivered as part of Edge Apps package.

### ACC100 (FEC) Ansible Installation for Smart Edge Open Network Edge
To run the Smart Edge Open package with ACC100 (FEC) functionality, the feature needs to be enabled on both Edge Controller and Edge Node. It can be deployed via the ["flexran" flavor of Smart Edge Open](https://github.com/smart-edge-open/ido-converged-edge-experience-kits/tree/master/flavors/flexran).

#### Converged Edge Experience Kits
To enable ACC100 support from CEEK, SRIOV must be enabled in Smart Edge Open:
```yaml
# flavors/flexran/all.yml
kubernetes_cnis:
- <main CNI>
- sriov
```

Also, enable the following options in `flavors/flexran/all.yml`:
The following device config is the default config for the Intel® vRAN Dedicated Accelerator ACC100.
```yaml
#  flavors/flexran/all.yml

acc100_sriov_userspace_enable: true

acc100_userspace_vf:
  enabled: true
  vendor_id: "8086"
  vf_device_id: "0d5d"
  pf_device_id: "0d5c"
  vf_number: "2"
  vf_driver: "vfio-pci"
```

Run setup script `deploy.py` with `flexran` flavor defined in `inventory.yml` for specific cluster.

> **NOTE**: for more details about deployment and defining inventory please refer to [CEEK](../../getting-started/converged-edge-experience-kits.md#converged-edge-experience-kit-explained) getting started page.

After a successful deployment, the following pods will be available in the cluster:
```shell
kubectl get pods -A

NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-646546699f-wl6rn  1/1     Running   0          3d19h
kube-system   calico-node-hrtn4                         1/1     Running   0          3d19h
kube-system   coredns-6955765f44-mrc82                  1/1     Running   0          3d19h
kube-system   coredns-6955765f44-wlvhc                  1/1     Running   0          3d19h
kube-system   etcd-silpixa00394960                      1/1     Running   0          3d19h
kube-system   kube-apiserver-silpixa00394960            1/1     Running   0          3d19h
kube-system   kube-controller-manager-silpixa00394960   1/1     Running   0          3d19h
kube-system   kube-multus-ds-amd64-2zdqt                1/1     Running   0          3d18h
kube-system   kube-multus-ds-amd64-db8fd                1/1     Running   0          3d19h
kube-system   kube-proxy-dd259                          1/1     Running   0          3d19h
kube-system   kube-proxy-sgn9g                          1/1     Running   0          3d18h
kube-system   kube-scheduler-silpixa00394960            1/1     Running   0          3d19h
kube-system   kube-sriov-cni-ds-amd64-k9wnd             1/1     Running   0          3d18h
kube-system   kube-sriov-cni-ds-amd64-pclct             1/1     Running   0          3d19h
kube-system   kube-sriov-device-plugin-amd64-fhbv8      1/1     Running   0          3d18h
kube-system   kube-sriov-device-plugin-amd64-lmx9k      1/1     Running   0          3d19h
openness      eaa-78b89b4757-xzh84                      1/1     Running   0          3d18h
openness      edgedns-dll9x                             1/1     Running   0          3d18h
openness      interfaceservice-grjlb                    1/1     Running   0          3d18h
openness      nfd-master-dd4ch                          1/1     Running   0          3d19h
openness      nfd-worker-c24wn                          1/1     Running   0          3d18h
openness      syslog-master-9x8hc                       1/1     Running   0          3d19h
openness      syslog-ng-br92z                           1/1     Running   0          3d18h
```

### FEC VF configuration for Smart Edge Open Network Edge
To configure the VFs with the necessary number of queues for the vRAN workload, the BBDEV configuration utility is going to run as a job within a privileged container. The configuration utility is available to run as a Helm chart available from `/opt/openness/helm-charts/bb_config`.

Sample configMap, which can be configured by changing values, if other than typical config is required, with a profile for the queue configuration is provided as part of Helm chart template `/opt/openness/helm-charts/bb_config/templates/acc100-config.yaml` populated with values from `/opt/openness/helm-charts/bb_config/values.yaml`. Helm chart installation requires a provision of hostname for the target node during job deployment. Additionally, the default values in Helm chart will deploy FPGA config, a flag needs to be provided to invoke ACC100 config.

Install the Helm chart by providing configmap and BBDEV config utility job with the following command from `/opt/openness/helm-charts/` on Edge Controller (this job needs to be re-run on each node reboot):

```shell
helm install --set nodeName=<node_name> --set device=ACC100 intel-acc100-cfg bb_config
```

Verify if the job has completed and that the state of the pod created for this job is “Completed”. Check the logs of the pod to see a complete successful configuration.
```
kubectl get pods
kubectl logs intel-acc100-cfg-<node_name>-xxxxx
```
Expected: `ACC100 PF [0000:af:00.0] configuration complete!`

To redeploy the job on another node, use the following command:

```
helm upgrade --set nodeName=<another_node_name> intel-acc100-cfg bb_config
```

To uninstall the job, run:
```
helm uninstall intel-acc100-cfg
```

### Requesting Resources and Running Pods for Smart Edge Open Network Edge
As part of the Smart Edge Open Ansible automation, a K8s SRIOV device plugin to orchestrate the ACC100 VFs (bound to the userspace driver) is running. This enables the scheduling of pods requesting this device. To check the number of devices available on the Edge Node from Edge Controller, run:

```shell
kubectl get node <node_name> -o json | jq '.status.allocatable'

"intel.com/intel_fec_acc100": "2"
```

To request the device as a resource in the pod, add the request for the resource into the pod specification file by specifying its name and the amount of resources required. If the resource is not available or the amount of resources requested is greater than the number of resources available, the pod status will be “Pending” until the resource is available.
**NOTE**: The name of the resource must match the name specified in the configMap for the K8s devices plugin [configMap.yml](https://github.com/smart-edge-open/converged-edge-experience-kits/blob/master/roles/kubernetes/cni/sriov/controlplane/templates/configMap.yml.j2).

A sample pod requesting the ACC100 (FEC) VF may look like this:

```
apiVersion: v1
kind: Pod
metadata:
  name: test
  labels:
    env: test
spec:
  containers:
  - name: test
    image: centos:latest
    command: [ "/bin/bash", "-c", "--" ]
    args: [ "while true; do sleep 300000; done;" ]
    resources:
      requests:
        intel.com/intel_fec_acc100: '1'
      limits:
        intel.com/intel_fec_acc100: '1'
```

To test the resource allocation to the pod, save the above code snippet to the `sample.yaml` file and create the pod.
```
kubectl create -f sample.yaml
```
Once the pod is in the 'Running' state, check that the device was allocated to the pod (a uioX device and an environmental variable with a device PCI address should be available):
```
kubectl exec -it test -- ls /dev
kubectl exec -it test -- printenv | grep FEC
```
To check the number of devices currently allocated to pods, run (and search for 'Allocated Resources'):

```
kubectl describe node <node_host_name>
```

### Verifying Application POD Access and Usage of FPGA on Smart Edge Open Network Edge
To verify the functionality of all the features are working together (SRIOV binding - K8s device plugin - BBDEV config) and functionality of the ACC100 (FEC) VF inside a non-root pod, build a Docker image and run a simple validation application for the device.

The automation of the Docker image build is available from the Edge Apps package. The image must be built on the same node that it is meant to be deployed or a server with the same configuration as the node that will run the workload. This is due to the Kernel dependencies of DPDK during the application build.

Navigate to:

```
edgeapps/fpga-sample-app
```

Build the image:

`./build-image.sh`

From the Edge Controller, deploy the application pod. The pod specification is located at `/fpga`:

```
kubectl create -f fpga-sample-app.yaml
```

Execute into the application pod and run the sample app:
```shell
# enter the pod
kubectl exec -it pod-bbdev-sample-app -- /bin/bash

# run test application
./test-bbdev.py --testapp-path ./testbbdev -e="-w ${PCIDEVICE_INTEL_COM_INTEL_FEC_ACC100}" -i -n 1 -b 1 -l 1 -c validation -v ./test_vectors/ldpc_dec_v7813.data

# sample output
Executing: ./dpdk-test-bbdev -w0000:b0:00.0 -- -n 1 -l 1 -c validation -i -v ldpc_dec_v7813.data -b 1
EAL: Detected 96 lcore(s)
EAL: Detected 2 NUMA nodes
Option -w, --pci-whitelist is deprecated, use -a, --allow option instead
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL:   using IOMMU type 1 (Type 1)
EAL: Probe PCI driver: intel_acc100_vf (8086:d5d) device: 0000:b0:00.0 (socket 1)
EAL: No legacy callbacks, legacy socket not created

===========================================================
Starting Test Suite : BBdev Validation Tests
Test vector file = ldpc_dec_v7813.data
Device 0 queue 16 setup failed
Allocated all queues (id=16) at prio0 on dev0
Device 0 queue 32 setup failed
Allocated all queues (id=32) at prio1 on dev0
Device 0 queue 48 setup failed
Allocated all queues (id=48) at prio2 on dev0
Device 0 queue 64 setup failed
Allocated all queues (id=64) at prio3 on dev0
Device 0 queue 64 setup failed
All queues on dev 0 allocated: 64
+ ------------------------------------------------------- +
== test: validation
dev:0000:b0:00.0, burst size: 1, num ops: 1, op type: RTE_BBDEV_OP_LDPC_DEC
Operation latency:
        avg: 31202 cycles, 13.5661 us
        min: 31202 cycles, 13.5661 us
        max: 31202 cycles, 13.5661 us
TestCase [ 0] : validation_tc passed
 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +
 + Test Suite Summary : BBdev Validation Tests
 + Tests Total :        1
 + Tests Skipped :      0
 + Tests Passed :       1
 + Tests Failed :       0
 + Tests Lasted :       413.594 ms
 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +
```
The output of the application should indicate a total of ‘1’ tests and ‘1’ test passing; this concludes the validation of the FEC VF working correctly inside a K8s pod.

## Reference
- [Intel® vRAN Dedicated Accelerator ACC100](https://networkbuilders.intel.com/solutionslibrary/intel-vran-dedicated-accelerator-acc100-product-brief)


text:
# WCS1800

A WCS1800 is an analog Hall effect sensor, that delivers a measure by means of voltage.

![wcs 1800 specifications](https://user-images.githubusercontent.com/36057481/37062708-41545b76-2165-11e8-9dcb-40e44ae11cf3.JPG)

## Attributes

* **analogPin:** Arduino Analog Pin where the WCS1800 is connected.
* **sensitivity:** This value store the sensitivity expressed in mV/A of the WCS1800.
* **zeroRead:** Stores the value read by Arduino when the current is 0A, set between 0 and 1024.
* **zero:** This value is the Vout when the current is 0A, (Vcc/2).
* **freq:** Line power frequency, normally 60Hz.

## Methods

* **WCS1800:** Constructor class, takes the value of the analogPin, sensitivity, zero and frequency as parameters.
* **Calibrate:** Sets the value of zeroRead, calibrates the sensor read when current is 0A.
* **getVout:** Gets the voltage from 0 to 5 v, where 0 v translates 0 A and 5 v translates 35 A
* **getDC:** Gets the DC current perceived thanks to the hall effect inside the WCS1800.
* **getAC:** Gets the AC current approximation.


text:
# PSXLSReader

[![CI Status](https://img.shields.io/travis/DeveloperPans/PSXLSReader.svg?style=flat)](https://travis-ci.org/DeveloperPans/PSXLSReader)
[![Version](https://img.shields.io/cocoapods/v/PSXLSReader.svg?style=flat)](https://cocoapods.org/pods/PSXLSReader)
[![License](https://img.shields.io/cocoapods/l/PSXLSReader.svg?style=flat)](https://cocoapods.org/pods/PSXLSReader)
[![Platform](https://img.shields.io/cocoapods/p/PSXLSReader.svg?style=flat)](https://cocoapods.org/pods/PSXLSReader)

PSXLSReader is a branch of [DHlibxls](https://github.com/dhoerl/DHlibxls). DHlibxls seem to been deprecated since three years ago and no one maintained. so PSXLSReader came here.

## Different with DHXLSReader
* Swift Compatiable
* CocoaPods supported
* Modern Objective-C Api

## Example

## Installation

PSXLSReader is available through [CocoaPods](https://cocoapods.org). To install
it, simply add the following line to your Podfile:

```ruby
pod 'PSXLSReader'
```

## License

PSXLSReader is available under the MIT license. See the LICENSE file for more info.

repository: MrPans/PSXLSReader

repo_name: MrPans/PSXLSReader


text:
# Usage

All examples are detailed in the iPython notebooks in the repository under `examples/`.
This document should give a conclusive guide to new users.

## Feature Importances

The user can calculate local and global feature importances with proven algorithms from a unified interface.
Prerequisites to use are a fitted sklearn classifier and the training data.
In some cases, also the training labels are required.
As an example dataset, TIC comes with the breast cancer dataset from sklearn.

```python
>>> test_data = load_test_data()
>>> class_names = test_data['target_names']
>>> feature_names = test_data['feature_names']
>>> training_data = test_data['dataset']['X_train']
>>> instance = test_data['dataset']['X_test'].sample(1).values[0]
>>> clf = pickle.load(open('.classifiers/logistic_regression.clf', 'rb'))
```

Then, call the `tic` method to retrieve the explanation, feature importances and a figure.
In this case, the explanation is based on [LIME](https://github.com/marcotcr/lime).

```python
>>> from tic.interpret import lime
>>> lime.explain_local(
...     clf=clf,
...     instance=instance,
...     training_data=training_data,
...     feature_names=feature_names,
...     class_names=class_names,
...     explanation_kwargs={'num_features': 5}
... )
{'explanation': <lime.explanation.Explanation at 0x7fad7f620978>,
 'feature_importances': [('mean perimeter', -0.22634431587962064),
  ('mean area', 0.22087904592306984),
  ('worst area', -0.16743486284244824),
  ('area error', -0.13807183263161452),
  ('mean radius', 0.08172324856930932)],
 'figure': <Figure size 432x288 with 1 Axes>}
```

As another example, global surrogate models can be trained and explained for any classifier.
Notice that not all interpretability models support both local and global explanations.

```python
>>> clf_mlp = pickle.load(
...     open('.classifiers/multi_layer_perceptron.clf', 'rb')
... )
>>> from tic.interpret import surrogate
>>> surrogate.explain_global(
...     clf=clf_mlp,
...     num_features=5,
...     X_train=X_train,
...     surrogate_type='tree',
... )
{'surrogate': DecisionTreeClassifier(...),
 'feature_importances': [('worst radius', 0.7830391127502633),
  ('area error', 0.07139301639577515),
  ('symmetry error', 0.05386874344979909),
  ('worst concave points', 0.053428792945424415),
  ('mean compactness', 0.03827033445873795)]}
```

repository: tommartensen/tic

gha_language: Python

repo_name: tommartensen/tic


text:
# Wizard Salon
**Johnny Mayer**

## Hair Salon C# Project

A C# simulated Hair Salon.  This is an MVC app that allows a user to add Stylists, and add Clients to specific Stylists.

###Specifications
* User can create a Stylist
	* Input: "Jane Doe"
	* Output: "Jane Doe"

* User can create a Client
	* Input: "Super Client"
	* Output: "Super Client"

* Client is assigned to Stylist upon creation
	* Input: "Super Client" assigned to "Jane Doe"
	* Output: "Client: Super Client, Stylist: Jane Doe"

## Setup/Installation/Requirements

* .NET Framework and MAMP required
* Clone the repository via terminal ```$git clone https://github.com/johnnymayer/HairSalon```
* Launch MAMP and Start Servers
* Create the database in MySql with the following command:

```
CREATE TABLE `clients` (
  `name` varchar(255) DEFAULT NULL,
  `id` int(11) NOT NULL,
  `stylist_id` int(11) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

CREATE TABLE `stylists` (
  `name` varchar(255) DEFAULT NULL,
  `id` int(11) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
```
* From the terminal, navigate to ~/HairSalon.Solution/HairSalon
* Run ```$dotnet restore```
* Launch the program with ```dotnet run```
* Navigate to [http://localhost:5000](http://localhost:5000) in your browser, and engage with the program.

##Built With
* HTML
* Razor
* Bootstrap & Custom CSS
* C#
* .NET Core 1.1
* MySql
* Mamp
* Atom

####License
MIT License applies.

**Copyright 2018 Johnny Mayer**
[Johnny Mayer GitHub](https://github.com/johnnymayer)

repository: johnnymayer/HairSalon

repo_name: johnnymayer/HairSalon


text:
# Speaker notes for CDATX Advanced Training.

[http://tech.paulcz.net/cdatx-advanced-docker](http://tech.paulcz.net/cdatx-advanced-docker)

## 1 - Welcome

## 2 - Who am I?

## 3 - BBC

## 4 - Agenda

## 5 - Intro - Optimizing dockerfiles

## 6 - Intro

Docker images are “supposed” to be small and fast. However unless you’re precompiling GO binaries and dropping them in the busybox image they can get quite large and complicated. Without a well constructed Dockerfile to improve build cache hits your docker builds can become unnecessarily slow.

Dockerfile’s are regularly [and incorrectly] treated like bash scripts and therefore are often written out as a series of commands which you would curl | sudo bash from a website to install. This usually makes for an inefficient and slow Dockerfile

## 7 - Example dockerfile

Make sure everybody understands what all is going on.   Ask someone to talk through it.

## 8 - Order Matters

When you’re building a new Dockerfile for an application there can be a lot of trial and error in determining what packages are needed and what commands need to run. Optimizing your Dockerfile ensures that the build cache will hit more often and each build between changes will be faster.

## 9 - Order Matters

The general rule of thumb is to sort your commands by frequency of change, the time it takes to run the command and how sharable it is with other images.

## 10 - Order Matters

Sny ADD ( or other commands that invalidate cache ) commands should go as far down the bottom as possible as this is where you’re likely to make lots of changes that will invalidate the cache of subsequent commands.

Commands like WORKDIR, CMD, ENV should go towards the bottom while a RUN apt-get -y update should go towards the top as it takes longer to run and can be shared with all of your images.

## 11 - Choose base images wisely

There’s a lot of base images to choose from from the bare OS images like ubuntu:trusty to application specific ones for python:2 or java:7. Common sense might tell you to use ruby:2 to run an ruby based app and python:3 to run a python app. However now you have two base images with little in common that you need to download and build. Instead if you use ubuntu:trusty for both then you only need to download the base image once.

## 12 - Use Layers to your advantage

Each command in a Dockerfile is an extra layer. You can very quickly end up with an image that’s 30+ layers. This is not necessarily a problem, but by joining RUN commands together, and using a single EXPOSE line to list all of your open ports you can reduce the number of layers.

By grouping RUN commands together intelligently you can share more layers between containers. Of course if you have a common set of packages across multiple containers then you should look at creating a seperate base image containing these that all of your images are built from.

For each layer that you can share across multiple images you can save a ton of disk space.

## 13 - Cheat

If you’ve built an image and discover when you run it that there’s a package missing add it to the bottom of your Dockerfile rather than in the RUN apt-get command at the top. This means you can rebuild the image faster. Once your image is correct and working you can reorganize your Dockerfile to clean such changes up before commiting it to source control.

## 14 - Volume containers

If you use Volume containers, don’t bother trying to save space by using a small image, Use the image of the application you’ll be serving data to. If you do that and docker commit the data volume you not only have your data commited to the container, but the actual application as well which is very useful for debugging.

## 15 - pop quiz slide

## 16 - example dockerfile

Ask class to re-order this Dockerfile.

## 17 - Agenda Docker vs Config Management

## 18 - I don't need...

talk to fallacy of this statement.

* manage non docker
* greenfield vs brownfield
* Skeuomorph

## 19

## 20 - Persistent Data

Docker is not [yet] good place for data to live

* docker containers are ephemeral/immutable by design
* AUFS didn't write when it say it did
* docker commit to ensure data is written to disk

* bind mounting from host came about.  Gabe @ Deis
    * until docker supports user name spaces this data owned by root

* volume mounts from other containers
    * only useful when on same host
    * all same issues as data being in container
        * except it persists after container stops.

## 21 - Brownfield / infrastructure

* Persistent data and other infrastructure services have to be managed by something.
* If not docker, then VMs or baremetal
* now we need orchestration / CM
    - Configure and run applications - mysql, haproxy
    - Configure and deploy services - RDS, ELB

## 22 - docker cookbook

* really strong cookbook that installs docker
* has a bunch of LWRPs to allow you to perform most ( if not all) docker commands from within chef.
* Most people I know that are running non-trivial systems that include docker are using this. -  See Drama Fever

## 23 - docker cookbook example

Example registry recipe

* pull the docker registry image
* run a docker registry container
    - if backed by object store, run on every node
* pull a list of images from the public docker registry
* push them to the private registry

## 24 - docker cookbook example

Example 'trusted build' recipe

* clones a git repo containing a Dockerfile
* when that changes, it notifies docker build to build an image
* could easily extend this to push the resultant image to registry - public or private.

## 25 - anything you can do i can do better

Almost anything that can be done with docker can be done via the docker cookbook.

Puppet and Ansible both have a similar story, but I haven't used them.

## 26 - CM in the container

You probably shouldn't.   But if you need to, or it makes sense to from where you are ... then ignore the haters and do it.

* Chef for Containers was a thing
* Helped provision inside the container and provided a lightweight init system.
* recently ( like last week ) it was depreciated as a standalone thing and is being wrapped into chef-provisioning.  I don't know how/if the functionanility will change.

* Bind mount chef-client into the container
* Have chef client locally or in a volume container
* bind mount it in to run chef-client in a base linux container
* commit the resulting container and push it to registry.

## 27 - CM to build containers

* if you already subscribe to the immutable infrastructure and burn AMIs or simply burn AMIs to reduce CM converge time you're probably already using a tool like packer.
* packer supports docker and you can probably, with very little tweaking tell it to build packer instead.

## 28 - ChefDK in a container

ChefDK is still fairly young and support for it is really focussed around OSX.  There's some Linux support, but it's for outdated, even no longer supported versions of Ubuntu ...  If you have anything else ... GOOD LUCK TO YOU SIR.

One way to work around this is to run chefdk in a ubuntu 12.04 container and bind mount in your current cookbook directory.   now you can hit it with `docker exec -ti chefdk rake style spec`.   Some things might not work easily like virtualbox support for testkitchen,  but most things do work.

## 29 - agenda screen

## 30 - Logging and Metrics

In the early days of docker both logging and metrics gathering was very difficult.  there was very little support in docker and no real tools in the ecosystem for what did exist.

Thankfully this has changed and Docker has some good ( and getting better ) support for both logging and metrics.

## 31 - Logging

The first thing you want to be really cognizant about is that you should do your very best to never actually log to a file in a container in the first place.

Otherwise A long living container will start to need log management, rotation, space management, etc to make sure that you keep it under control

## 32 - Logspout

Logspout was created by Jeff Lindsay who has had his hand in just about every awesome thing in the Docker ecosystem since before docker was even an actual thing.

* watches the docker APIs log stream
* runs in a container - bind mount in the docker socket
* forwards logs to one or more syslog servers defined at runtime

## 33 - /dev/stdout

If you app insists on logging to a file you can set it to  use /dev/stdout and /dev/stderr which do exactly what you might guess, any text written to them with through the magic of linux show up in the process' stdout/stderr.

## 34 - but even then I can't

I discovered once that mysql insists on logging errors to a file with extension of .err if you didn't specify an extension.  I had to symlink /dev/stderr to /var/log/mysql.err.

You can also run syslog in the container itself as a second process and have your applications log to syslog instead.   haproxy has a hard time logging to anything that's not syslog,  this can help with that.

There's an issue filed against logspout for creating a lightweight syslog server that such containers can use as a syslog target and thus route through logspout the same as other containers.

worse case scenario you can always mount a volume from the host or a volume container and log to there.   It's not ideal but it moves responsibility for managing log files to outside the container running the app.

## 35 - logging improvements coming soon

1.6 should have improved logging,  will support logging drivers which should allow for native transport via syslog and other methods.

## 36 - Metrics

Until very recently ( 1.5 ) docker didn't have any native support for collecting metrics.   However because docker is a wrapper around LXC we have access to all sorts of metrics if we know where to you.

LXC exposes metrics in `/proc` and `/sys` and other weird places that you probably don't know to look.   Example memory stats for a given container would go to `/sys/fs/cgroup/memory/lxc/[container_id]/`.

As I said,  you really have to know LXC well to know where all the stats end up.

## 37 - Metrics - CAdvisor

Google had already solved the metrics problem internally as they are one of the early pioneers of LXC and contributed a lot of the kernel code for cgroups etc.

As you can see from the docker command here its pretty intrusive, needing a lot of volumes mounted into it,  thankfully most can be mounted read only.

## 38 - Metrics - Cadvisor

Cadvisor collects metrics for any running containers ( and not just docker containers ) and provides access to them via API or GUI and can push them to a couple of different metrics collection engines, currently InfluxDB and Prometheus,   I would expect to see support for graphite at some point as well.

It collects per container metrics for the usual suspects of interesting machine properties such as CPU, Memory, Network, Disk.

## 39 - agenda slide

## 40 - multiprocess containers

So this is a highly charged topic.  You here a lot of noise telling you to only ever run a single process in a container, and if you run more than that you're doing it wrong.  I would go as far as to say the rhetoric for this argument gets quite poisonous.

Docker themselves have said countless times that they see room for both single process containers and multi-process containers.   I am firmly on this side of the bench.

I will agree that the fewer processes the better, and if you happen to be greenfielding you should really be paying attention to the whole 12factor mindset,  but most of us are in situations where this is not entirely possible.

There's a few ways to achieve this, and they each have varying levels of merit and issues which make them attrative for different reasons.

## 41 - backgrounded processes

The easiest way to run multiple containers is to have a short shell script that you run inside the container.  This can be added in your dockerfile and set to ENTRYPOINT or CMD.

It simply runs the first command with an & and then the second command in the foreground like so.

## 42 - problems with backgrounded processes

There's a few problems with doing this, so I wouldn't recommend it for anything other than a quick dev or test container.  If its not going to be long lived, then this is by far the easiest way to achieve multiple processes.

Problems:

* Any of the processes started could die ( including the process that is the script itself ) and the container may continue to live on in a busted state.
* Zombie processes,  discussed later
* We're trying to emulate (badly) what is already a solved problem, and that solved problem is init systems like runit or systemd.
* At the very least when doing this, run your final command with `exec`.  This will give the primary pid to that process.

## 43 - Init systems - Runit

* Runit is a fairly lightweight init system with very easy to write services.

* With runit installed and run as your container's entrypoint/cmd it will execute any scripts in `/etc/service/<name>/run`.

* Usually this consists of a shell script written to run a process in the foreground and maybe set a user to run as, but it can be any executable.

* runit restarts processes on exit/crash

* runit services can be controlled by the standard service commands in linux,  and if you want to restart a process in the container you can issue a service restart command via docker exec.

* runit handles zombie processes appropriately.

## 44 - Zombies!

* unix processes are ordered in a tree with PID 1 being the top of the tree.  Each process has a parent and can have children.

* when a process dies it turns in defunct or zombie process.  Unix expects that the parent process will explicitly wait for the child process to end in order to collect its exit status.

* most applications wait for their children processes to exit correctly,  but running things from random shell scripts can play havoc with this.  If the parent process is killed, all of its children become orphans.

* the kernel deals with this problem by giving orphaned processes to PID1 / the init system and expects it to deal with them appropriately.

* Bash is actually capable of reaping child processes, but it does not handle signals properly nor does it pass those signals down to its children.   thus bash cannot be used /reliably/ as a lightweight init system.

* This /is/ a problem that can be solved with bash, and I had started working on solving it myself when I found the blog post about how phusion images handle things.

* They wrote a very lightweight init system called my_init which in turn loads runit, or a list of provided apps.   I'm not 100% sure if they solve something that needs to be solved when just running runit, or if they just use my_init first for consistency.

## 45 - Agenda slide

## 46 - Container Centric OSes

* CoreOS - the oldest most mature Container centric OS of the bunch.
* Atomic - Redhat's minimal container OS
* Snappy - Ubuntu Core - Ubuntu's
* RancherOS - recent addition to the lineup from the folks at rancherlabs.

## 47 - CoreOS

CoreOS is a free linux operating system.  It is provided under an Apache2.0 license and the source is freely available on github.

They do offer a support model that gives access to some premium features as 'CoreOS Managed Linux' which offers some tooling around managing fleets of CoreOS servers and update management.

CoreOS also has an enterprise registry which is priced quite reasonably and gives you an amazing behind the firewall docker registry.

At its heart CoreOS is a fork of Google's ChromeOS and is focussed at being a very lightweight server OS.

It uses FastPatch for updates, which is a active/passive root partition scheme, the OS is updated as an entire unit on the passive partition and then the partitions are simply switched and rebooted to update.

CoreOS philosphy is to patch frequently and to focus on writing applications / architectures that can handle frequent rebooting on individual nodes.  They have a series of update options that use ETCD to help enforce rules about how many systems can be patching/rebooting at a given time.

Updates are available to all users,  paying customers get access to CoreUpdate which is a hosted dashboard to help manage updates across a fleet of systems.

There is no package manager,  instead a given binary is either part of the OS itself or is completely the responsibility of the user.  It is generally expected that anything not part of the OS itself is going to run inside a container.

CoreOS comes with some tooling to help support native clustering of your hosts.  more on this in a minute.

Rocket is a new containerization technology that CoreOS has recently announced. At this time I no real experience with it beyond reading some documentation and internet opinions.

## 48 - CoreOS - Clustering

CoreOS comes with a set of tools designed to help you with clustering CoreOS hosts and applications running on them.

### etcd

etcd is a distributed, consistent key value store  written in Go for shared configuration and service discovery using the Raft consensus algorithm with a focus on being:

* Simple: curl'able user facing API (HTTP+JSON)
* Secure: optional SSL client cert authentication
* Fast: benchmarked 1000s of writes/s per instance
* Reliable: properly distributed using Raft

### fleet

fleet ties together systemd and etcd into a distributed init system. Think of it as an extension of systemd that operates at the cluster level instead of the machine level.

A fleet object looks exactly like a systemd service definition with some additional fields to handle scheduling hints etc.

### flannel

flannel (originally rudder) is an overlay network that gives a subnet to each machine for use with Kubernetes.  It can be used to give each host's docker0 interface a set of real IPs and manages routing between the hosts.   It is not turned on by default in coreos.

## 49 - agenda

## 50 - app configuration

Unless your application is a mythical 12factor application that is configured by environment variables you'll need to figure out how to configure your application in the container.

## 51 - in dockerfile

If your application config is very static,  you can put the config file in the same directory as your Dockerfile and simply use ADD in the Dockerfile to put the config file in the correct location,  You might want to use this for say an apache container that hosts only static content that is always found in /var/www/html

## 52 - via volume mount

You can keep your config files on your hosts ( maybe placed there by configuration management ) and then bind mount them into the container.

This is not a bad way to go if you're treating docker as a packaging format and you don't mind exposing parts of the hosts filesystem to the container.

The usual security implications of doing a bind mount apply.

## 53 - start script

If you have a config file with very few changes,  you can create a simple templated pattern of keywords in the config file like xxxPORTxxx and add it in during the docker build process.

You also have a startup script that you add in that runs an inline sed to change those keywords into the values of environment variables before running your actual command with an `exec` line at the bottom of the script.

## 54 - using confd

This is my personal favorite way of doing handling config files in containers.

Confd is is a lightweight templating engine that takes values from a key value store such as etcd or consul.  It also supports environment variables.

## 55 - using confd

As shown here you can download a binary distribution of confd from github and mark it executable in your Dockerfile.  This shows grabbing etcd as well, this isn't essential for confd to work,  but it is cheap.

## 56 - using confd

This is an example startup script that you run in your docker container.  You can see it takes an ENV variable called `backend` and if that doesn't exist defaults to `env`.

It then attempts to write out any templates you have stored in `/etc/confd`.  I have this in a loop so that if the key-value store isn't read yet, it will just keep looping here.   Very useful if you need to wait for another service to start first and register itself in etcd, but not really necessary for environment variables.

Once confd has written the templates out it will run apache in foreground via the final `exec` command.

## 56 - using confd

Assuming you have templated the apache port and apache root you would call the apache container like this to set those values.

## 57 - using confd

We'll see more confd magic as we go through the demo shortly.

## 58 - agenda

## 59 - DEMO

### factorish/example

So we're going to do a demo and talk through some of these concepts.

Let's take a look at this example Docker repo - https://github.com/factorish/factorish/tree/master/example

* talk through dockerfile,  pointing out confd and etcd, runit.

* talk through the contents of bin/
    - boot
    - healthcheck
    - my_init

* show confd and templates directory and talk about how confd uses those dirs to write templates out.  use `confd/example.conf.toml` and `confd/example.conf` to illustrate.

### factorish

* Vagrantfile
* factorish.rb
* factorish.yml
* user-data.erb
* registry and image caching
* profiles.d scripts, how they're generated etc.
* build / fetch scripts

### factorish/elk

talk through how it works,  show fleet units, etc.

repository: paulczar/cdatx-advanced-docker

branch_name: HEAD

repo_name: paulczar/cdatx-advanced-docker


text:
# member list

This repo is for SITW members.

## add your data by the data format

EXAMPLE:

```
{
  "name": "chilijung",
  "email"<EMAIL_ADDRESS>  "github-id": "chilijung",
  "twitter-id": "chilijung",
  "facebook-id": "lirong.ji",
  "department": "IIM",
  "class": 103,
  "skill": "website development, mostly coding nodejs, javscript.... . Also learned PHP, Perl...",
  "description": "I like web, and think web is a powerful tool to change the word"
}

```

in `member.json`

## Contribute

### Set up enviroment:
`$ npm install`
`$ node app.js`

### Open browser
[http://localhost:8080](http://localhost:8080)

## Author

[@chilijung](https://github.com/chilijung)

## Contributers

[@garylai1990](https://github.com/garylai1990)
Feel free to add by your own!

# License

© 2013, Lijung Chi. Released under the [MIT License](http://opensource.org/licenses/mit-license.php).

repository: bendio/member-list

repo_name: bendio/member-list


text:
# sallow
[![arXiv](https://img.shields.io/badge/arXiv-2006.07050-b31b1b.svg)](https://arxiv.org/abs/2006.07050)
[![DOI](https://zenodo.org/badge/268122113.svg)](https://zenodo.org/badge/latestdoi/268122113)

A heuristic algorithm for finding [treedepth decompositions](https://en.wikipedia.org/wiki/Tree-depth).

This is a submission for the [PACE 2020](https://pacechallenge.org/2020/td/) challenge.
The algorithm relies on:
* a variety of greedy elimination algorithms, and
* Divide & Conquer along cuts obtained using a from-scratch reimplementation of Ben Strasser's [2016 FlowCutter](https://github.com/ben-strasser/flow-cutter-pace16),
which is excellently described in the paper *[Graph Bisection with Pareto Optimization](https://doi.org/10.1145/3173045)*.

The short paper *[Sallow – a heuristic algorithm for treedepth decompositions](https://arxiv.org/abs/2006.07050)* gives a more detailed overview. Cite as, e.g.:

```
@misc{Wrochna20sallow,
  author        = {Marcin Wrochna},
  title         = {Sallow -- a heuristic algorithm for treedepth decompositions},
  year          = {2020},
  archivePrefix = {arXiv},
  eprint        = {2006.07050}
}
```

Sallows (such as [Salix caprea](https://en.wikipedia.org/wiki/Salix_caprea)) are willows, a kind of shrub/small tree.

![A sallow tree](https://upload.wikimedia.org/wikipedia/commons/thumb/c/c2/20170421Salix_caprea1.jpg/640px-20170421Salix_caprea1.jpg)

(Photo by [AnRo0002](https://commons.wikimedia.org/wiki/File:20170421Salix_caprea1.jpg))

## Git
This repo uses submodules:
* clone with `git clone --recurse-submodules git@github.com:marcinwrochna/sallow.git`
* if you want, update all submodules with `git submodule update --remote --rebase` (this fetches to `master` and tries to rebase your existing changes within each submodule)

Currently the only dependency used is Microsoft's [implementation](https://github.com/microsoft/GSL) of the [C++ Guidelines Support Library](http://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#S-gsl).

## Building
Standard CMake:
* `mkdir build && cd build` (or wherever you want the build files to be, instead of `build/`)
* `cmake --config Release ..` (or whatever the path to the root CMakeLists.txt is, instead of `..`)
* `make`
* Or just open the directory in [MSVS Code](https://code.visualstudio.com/) with the [CMake extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode.cmake-tools) installed, and allow it to configure itself.
* CMake ≥ 3.13 and g++ ≥ 7 or clang ≥ 4 (that is, supporting C++17) are required.
* Change `-march=ivybridge` to `native` or remove, if needed, in `CMakeLists.txt`.

## Running
* `sallow input.gr`
* Interrupt (ctrl+c or SIGINT) to stop and print the best decomposition we have.
* If it hangs (e.g. you ran it in debug mode on extremely large graphs), you need to `pkill -9 sallow`.
* Input and output format as specified [here](https://pacechallenge.org/2020/td/) (DIMACS-like).
* See there also for test instances.

## Acknowledgements
The author is very grateful to the [PACE 2020](https://pacechallenge.org/2020) organizers at the University of Warsaw and the [OPTIL.io team](https://www.optil.io/optilion/about) at the Poznań University of Technology for making this challenge possible.
This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 714532, PI: Stanislav Živný).
<img alt="ERC logo" src="publications/logo-erc.jpg?raw=true" width="70" align="right"/>

repository: marcinwrochna/sallow

repo_name: marcinwrochna/sallow


text:
# JuceDebuggerHelper

Natvis file for Visual Studio to display JUCE types in the debugger in an easier to read format. Based on http://github.com/jcredland/juce-toys, updated and extended for the latest JUCE version.

To install copy natvis file to ```C:\Program Files\Microsoft Visual Studio\2022\Professional\Common7\Packages\Debugger\Visualizers```

repository: reFX/JuceDebuggerHelper

repo_name: reFX/JuceDebuggerHelper


text:
Return the Nth Even Number

Example(Input --> Output)

1 --> 0 (the first even number is 0)
3 --> 4 (the 3rd even number is 4 (0, 2, 4))
100 --> 198
1298734 --> 2597466
The input will not be 0.

repository: BrianLusina/KotlinGround

gha_language: Kotlin

repo_name: BrianLusina/KotlinGround


text:
# [How Discord Supercharges Network Disks for Extreme Low Latency](https://discord.com/blog/how-discord-supercharges-network-disks-for-extreme-low-latency)

## Scaling Beyond Our Hardware

* The biggest impact on Discord's (NoSQL) database performance is disk latency (how long it takes to read or write data from physical hardware)
* At a certain threshold, the database will need to wait for an outstanding disk operation to complete before it will issue another
  * Disks may take a millisecond or two to complete an operation, which causes a queue of disk operations that have yet to execute
* GCP provides Persistent Disk storage, which are not attached directly to a server, but connected nearby (probably same building) via network
  * Latency of the disk operations were on the order of a couple milliseconds vs. half a millisecond for directly-attached disks
* Local SSD / NVMe concerns - hardware issue with the disk or host, then the disks and their data are gone forever
  * Persistent Disks create point-in-time snapshots of an entire disk

## Creating the Super-Disk

* Write-through cache where Local SSDs are the cache and the Persistent Disks are the storage layer
* Local SSDs suffer from bad sectors, which can be repaired from the storage layer, but weren't able find a solution that had the capability / capacity
* `md` allows Linux to create software RAID arrays, turning multiple disks into one "array" / virtual disk
  * `md` offers a feature called "write-mostly", where individual devices / drives are excluded from normal read balancing and will only be read from when there is no other option
  * Discord requires at least 1TB or more of storage per database instance, while the Local SSDs are 375GB
  * Need to construct a bunch of smaller disks into one larger disk
  * `md` has RAID configurations which splits data across all disks
  * For example, RAID0 splits raw data across all disks and if one is lost, the entire array fails and all data is lost
  * Configuraiton is, RAID0 on top of the Local SSDs and a RAID1 between the Persistent Disk and RAID0 array

url: https://raw.githubusercontent.com/jaebradley/notes/b9cb0fac012e61205d12ed8d139bbc494319e392/Software Engineering/Data/Storage/Databases/How Discord Supercharges Network Disks for Extreme Low Latency.md

repository: jaebradley/notes

repo_name: jaebradley/notes


text:
---
# Display name
title: Nikki Theofanopoulou
authors:
- nikki-theofanopoulou
# Is this the primary user of the site?
superuser: false

# Role/position/tagline
role:  PhD Student

# Organizations/Affiliations to show in About widget
organizations:
- name: King's College London
  url: https://www.kcl.ac.uk

# Short bio (displayed in user profile at end of posts)
bio: Nikki Theofanopoulou is a PhD student in the Department of Informatics. Her research focuses on understanding how novel technology-enabled systems could be embedded in the everyday lives of children and young people, as well as in clinical settings to promote mental health.

# Interests to show in About widget
interests:
- Human-Computer Interaction
- Interaction Design
- Mental Health Promotion

# Social/Academic Networking
# For available icons, see: https://sourcethemes.com/academic/docs/page-builder/#icons
#   For an email link, use "fas" icon pack, "envelope" icon, and a link in the
#   form<EMAIL_ADDRESS>or "/#contact" for contact widget.
social:
- icon: envelope
  icon_pack: fas
  link: '/#contact'

# Link to a PDF of your resume/CV.
# To use: copy your resume to `static/media/resume.pdf`, enable `ai` icons in `params.toml`,
# and uncomment the lines below.
# - icon: cv
#   icon_pack: ai
#   link: media/resume.pdf

# Email for Contact widget or Gravatar
email: ""

# Highlight the author in author lists? (true/false)
highlight_name: false

# Organizational groups that you belong to (for People widget)
#   Remove this if you are not using the People widget.
user_groups: ["PhD Students"]
---

Nikki Theofanopoulou is a PhD student in the Department of Informatics. Her research focuses on understanding how novel technology-enabled systems could be embedded in the everyday lives of children and young people, as well as in clinical settings to promote mental health. Nikki has published papers furthering this research agenda in human-computer interaction, psychology and mental health venues, and her accolades also include receiving a Best Paper Award at the Computer-Supported Cooperative Work and Social Computing conference (2018). She also holds an MSc in Developmental Psychology from UCL and a BSc in Psychology from the University of Athens (Greece).

repository: tdjneate/hcc-kings

repo_name: tdjneate/hcc-kings


text:
# Progress Bars [![Build Status](https://travis-ci.org/cferdinandi/progress-bars.svg)](https://travis-ci.org/cferdinandi/progress-bars)
Simple CSS progress bars. [View the demo](http://cferdinandi.github.io/progress-bars/).

## Getting Started

Compiled and production-ready code can be found in the `dist` directory. The `src` directory contains development code.

### 1. Include Progress Bars on your site.

```html
<link rel="stylesheet" href="css/progress-bars.css">
```

### 2. Add the markup to your HTML.

Use percentages to set the widths of each bar. Add optional `.progress-bar-secondary`, `.progress-bar-success`, `.progress-bar-danger`, or `.progress-bar-warning` classes for additional colors, and to convey meaning.

```html
<div class="progress">
	<div class="progress-bar" style="width: 90%;">
		Basic
	</div>
</div>

<div class="progress">
	<div class="progress-bar progress-bar-secondary" style="width: 30%;"></div>
	Secondary
</div>

<div class="progress">
	<div class="progress-bar progress-bar-success" style="width: 80%;">
		Success
	</div>
</div>

<div class="progress">
	<div class="progress-bar progress-bar-danger" style="width: 60%;">
		Danger
	</div>
</div>

<div class="progress">
	<div class="progress-bar progress-bar-warning" style="width: 70%;">
		Warning
	</div>
</div>
```

And that's it, you're done. Nice work!

## Installing with Package Managers

You can install Progress Bars with your favorite package manager.

* **NPM:** `npm install cferdinandi/progress-bars`
* **Bower:** `bower install https://github.com/cferdinandi/progress-bars.git`
* **Component:** `component install cferdinandi/progress-bars`

## Working with the Source Files

If you would prefer, you can work with the development code in the `src` directory using the included [Gulp build system](http://gulpjs.com/). This compiles, lints, and minifies code, and runs unit tests. It's the same build system that's used by [Kraken](http://cferdinandi.github.io/kraken/), so it includes some unnecessary tasks and Sass variables but can be dropped right in to the boilerplate without any configuration.

### Dependencies
Make sure these are installed first.

* [Node.js](http://nodejs.org)
* [Gulp](http://gulpjs.com) `sudo npm install -g gulp`

### Quick Start

1. In bash/terminal/command line, `cd` into your project directory.
2. Run `npm install` to install required files.
3. When it's done installing, run one of the task runners to get going:
	* `gulp` manually compiles files.
	* `gulp watch` automatically compiles files when changes are made and applies changes using [LiveReload](http://livereload.com/).

## Browser Compatibility

Progress Bars works in all modern browsers, and IE 6 and above.

## How to Contribute

In lieu of a formal style guide, take care to maintain the existing coding style. Please apply fixes to both the development and production code. Don't forget to update the version number, and when applicable, the documentation.

## License

The code is available under the [MIT License](LICENSE.md).

repository: aussieweb/progress-bars

repo_name: aussieweb/progress-bars