text:
---
author: benjchristensen
comments: false
date: 2013-05-01 04:05:36+00:00
layout: post
slug: optimizing-the-netflix-api
title: Optimizing the Netflix API
---

Originally written for and posted on the [Netflix Tech Blog](http://techblog.netflix.com/2013/01/optimizing-netflix-api.html):

About a year ago the Netflix API team began redesigning the API to improve performance and enable UI engineering teams within Netflix to optimize client applications for specific devices.  Philosophies of the redesign were introduced in a previous post about [embracing the differences](http://techblog.netflix.com/2012/07/embracing-differences-inside-netflix.html) between the different clients and devices.

This post is part one of a series on the architecture of our redesigned API.

# Goals

We had multiple goals in creating this system, as follows:

###
Reduce Chattiness

One of the key drivers in pursuing the [redesign](http://techblog.netflix.com/2011/02/redesigning-netflix-api.html) in the first place was to reduce the chatty nature of our client/server communication, which could be hindering the overall performance of our device implementations.

Due to the generic and granular nature of the original REST-based Netflix API, each call returns only a portion of functionality for a given user experience, requiring client applications to make multiple calls that need to be assembled in order to render a single user experience.  This interaction model is illustrated in the following diagram:

![](/images/90083-request-multi_1252.png)

To reduce the chattiness inherent in the REST API, the discrete requests in the diagram above should be collapsed into a single request optimized for a given client.  The benefit is that the device then pays the price of WAN latency once and leverages the low latency and more powerful hardware server-side. As a side effect, this also eliminates redundancies that occur for every incoming request.

![](/images/7d4c6-request-single_1252.png)

A single optimized request such as this must embrace server-side parallelism to at least the same level as previously achieved through multiple network requests from the client.  Because the server-side parallelized requests are running in the same network, each one should be more performant than if it was executed from the device.  This must be achieved without each engineer implementing an endpoint needing to become an expert in low-level threading, synchronization, thread-safety, concurrent data structures, non-blocking IO and other such concerns.

###
Distribute API Development

A single team should not become a bottleneck nor need to have expertise on every client application to create optimized endpoints.  Rapid innovation through fast, decoupled development cycles across a wide variety of device types and distributed ownership and expertise across teams should be enabled.  Each client application team should be capable of implementing and operating their own endpoints and the corresponding requests/responses.

###
Mitigate Deployment Risks

The Netflix API is a Java application running on hundreds of servers processing 2+ billion incoming requests a day for millions of customers around the world.  The system must mitigate risks inherent in enabling rapid and frequent deployment by multiple teams with minimal coordination.

###
Support Multiple Languages

Engineers implementing endpoints come from a wide variety of backgrounds with expertise including Javascript, Objective-C, Java, C, C#, Ruby, Python and others.  The system should be able to support multiple languages at the same time.

###
Distribute Operations

Each client team will now manage the deployment lifecycle of their own web service endpoints.  Operational tools for monitoring, debugging, testing, canarying and rolling out code must be exposed to a distributed set of teams so teams can operate independently.

# Architecture

To achieve the goals above our architecture distilled into a few key points:

  * dynamic polyglot runtime

  * fully asynchronous service layer

  * functional reactive programming model

The following diagram and subsequent annotations explain the architecture:

![](/images/72a7d-architecture-overview_1252.png)

###
[1] Dynamic Endpoints

All new web service endpoints are now dynamically defined at runtime. New endpoints can be developed, tested, canaried and deployed by each client team without coordination (unless they depend on new functionality from the underlying API Service Layer shown at item 5 in which case they would need to wait until after those changes are deployed before pushing their endpoint).

###
[2] Endpoint Code Repository and Management

Endpoint code is published to a Cassandra multi-region cluster (globally replicated) via a RESTful Endpoint Management API used by client teams to manage their endpoints.

###
[3] Dynamic Polyglot JVM Language Runtime

Any JVM language can be supported so each team can use the language best suited to them.

The Groovy JVM language was chosen as our first supported language. The existence of first-class functions (closures), list/dictionary syntax, performance and debuggability were all aspects of our decision.  Moreover, Groovy provides syntax comfortable to a wide range of developers, which helps to reduce the learning curve for the first language on the platform.

###
[4 & 5] Asynchronous Java API + Functional Reactive Programming Model

Embracing concurrency was a key requirement to achieve performance gains but abstracting away thread-safety and parallel execution implementation details from the client developers was equally important in reducing complexity and speeding up their rate of innovation.  Making the Java API fully asynchronous was the first step as it allows the underlying method implementations to control whether something is executed concurrently or not without the client code changing.  We chose a functional reactive approach to handling composition and conditional flows of asynchronous callbacks. Our implementation is modeled after [Rx Observables](https://rx.codeplex.com/).

###
[6] Hystrix Fault Tolerance

As we have described in a [previous post](http://techblog.netflix.com/2012/02/fault-tolerance-in-high-volume.html), all service calls to backend systems are made via the Hystrix fault tolerance layer (which was [recently open sourced](http://techblog.netflix.com/2012/11/hystrix.html), along with its [dashboard](http://techblog.netflix.com/2012/12/hystrix-dashboard-and-turbine.html)) that isolates the dynamic endpoints and the API Service Layer from the inevitable failures that occur while executing billions of network calls each day from the API to backend systems.

The Hystrix layer is inherently mutlti-threaded due to its use of threads for isolating dependencies and thus is leveraged for concurrent execution of blocking calls to backend systems. These asynchronous requests are then composed together via the functional reactive framework.

###
[7] Backend Services and Dependencies

The API Service Layer abstracts away all backend services and dependencies behind facades.  As a result, endpoint code accesses “functionality” rather than a “system”.  This allows us to change underlying implementations and architecture with no or limited impact on the code that depends on the API.  For example, if a backend system is split into 2 different services, or 3 are combined into one, or a remote network call is optimized into an in-memory cache, none of these changes should affect endpoint code and thus the API Service Layer ensures that object models and other such tight-couplings are abstracted and not allowed to “leak” into the endpoint code.

##
Summary

The new Netflix API architecture is a significant departure from our previous generic RESTful API.

Dynamic JVM languages combined with an asynchronous Java API and the functional reactive programming model have proven to be a powerful combination to enable safe and efficient development of highly concurrent code.

The end result is a fault-tolerant, performant platform that puts control in the hands of those who know their target applications the best.

Following posts will provide further implementation and operational details about this new architecture.

extension: markdown


text:
# Ansible

This is a tool builder to use for [Ansible][ansible] projects on Linux.

## Getting Started

To build the images in the current project:

```
gcloud builds submit . --config=cloudbuild.yaml
```

[ansible]: https://www.ansible.com

repository: bluk/gcp-cloud-builders

branch_name: refs/heads/master

extension: md

license_type: permissive

repo_name: bluk/gcp-cloud-builders


text:
# Contributing

## Contents

1. [When does a component go in the design system?](#when-does-a-component-go-in-the-design-system)
1. [Maintaining Assets](#maintaining-assets)
1. [Git Strategy](#git-strategy)
1. [Code Standards](#code-standards)
1. [Code Style](#code-style)
1. [Release Process](#release-process)
1. [Rules](#rules)
1. [Build process](#build-process)
1. [ES Modules](#es-modules)

---

## When does a component go in the design system?

[Does It Belong in the System?](https://medium.com/eightshapes-llc/i-made-this-does-it-go-in-the-system-3b67b9894531)

## Icons

### Designing

Icons are exported from Figma and stored in the adjacent [`svg`](packages/assets/raw/svg) directory.

Each icon **must**:

- have it's own `32×32` artboard.
  - in other words, a `viewBox` that is **at least `32`** on one axis:
    - good: `viewBox="0 0 32 18"`, `viewBox="0 0 32 32"`.
    - bad: `viewbox="0 0 20 34"`.
- contain only one exportable layer, which has:
  1. a single path or group.
  1. a kebab-case layer name prefixed with `icon-`
     - this is essential for the [build step](#building).
  1. the export format defined as "SVG".
- have its color fill set to `#DE1E7E` (remember "DELETE") if you want the color to be modified.

Any changes to this Sketch file should be committed **as well as** the Sketch-generated SVGs. To export all SVGs in Sketch, select `File > Export` and in the next step choose the [`svg`](packages/assets/raw/svg) directory as the output location.

### Building

For performance benefits (such as code-splitting) we opted for creating individual React components for **each** icon, rather than one single component.

We make use of [SVGR](https://www.smooth-code.com/open-source/svgr/), to transform the above SVG icons in [`assets`](packages/assets/raw/svg) to React components in [`moon-icons`](packages/icons/README.mdx).

To update/add icons run: `yarn assets build`

- Runs SVGR based on the [config options](packages/icons/src/config/svgr.js).
- Builds React `.tsx` files for each component with the [template](packages/icons/src/config/template.js).

## Git Strategy

### Commits

Commits follow the [Angular Commit Message Format](https://github.com/angular/angular.js/blob/master/DEVELOPERS.md#-git-commit-guidelines). When committing changes, make use of the [Commitizen](https://github.com/commitizen/cz-cli) CLI to generate consistent commit messages:

```
yarn commit
```

### Branches

#### Base Branch

As a developer, you will you be branching and merging from `develop`, our base branch.

Consider `origin/master` to always represent the latest code deployed to production.

#### Supporting Branches

Use supporting branches for all new **features** and **bug fixes**. Unlike the base branch, these branches have a limited life-time and should be removed after merging.

The different types of branches should be named as follows:

- **Feature**: `feature/feature-name`
- **Bug fix**: `fix/fix-name`

#### Workflow

1. Create the branch locally and then push to GitHub if it does not exist yet.

   A branch should always be 'publicly' available, and should never exist in just one developer's local repo.

1. Make a pull request.
1. Add the `in progress` tag until ready for review.
1. Periodically, keep changes up to date with develop via `git rebase develop`.
1. Grab a review and remove the `in progress` tag.
1. One approved, it's down to you to merge the branch into `develop`. Code can **only** be contributed to `develop` via using pull requests.
1. Delete your branch.

#### Rebasing develop branch to your branch example

1. `git checkout develop`
1. `git pull`
1. `git checkout your-branch`
1. `git rebase develop`

#### Feature Flags

Long-lived feature branches present problems when you need to build code on top of unfinished work. Merge unfinished features into the `develop` branch (following the steps [outline above](#workflow)) so others can build off their work, but keep them hidden from your users and testers behind feature flags. Enable the flag in development to use the feature without the changes affecting anyone else. Once the feature is finished, you can remove the flags or use them to roll out to selected users and testers.

## Code Standards

### Accessibility

All features should attempt to conform to as many items on The A11Y Project's [Web Accessibility Checklist](https://a11yproject.com/checklist) as possible. If a checkbox can't be completed, the justification should be documented for future reference.

As part of our TypeScript linting process, we make use of [`react-a11y` rules](https://github.com/reactjs/react-a11y) (surfaced via [`tslint-microsoft-contrib`](https://github.com/Microsoft/tslint-microsoft-contrib)) to catch any common issues. `react-a11y-role-has-required-aria-props` is currently disabled for [incorrect results](https://github.com/Microsoft/tslint-microsoft-contrib/issues/409).

### Mobile First

Mobile devices are the most commonly used methods of browsing the web.
When it comes to designing and developing your component, always build with mobile in mind first.

Need to modify the style at specific breakpoints? Scale your changes **upwards**; build for mobile by default and _then_ add media queries for changes on larger sizes.

Want to use CSS Grid? Set your component to `display: block;` by default to stack items on mobile and other unsupported devices, then implement your grid at larger breakpoints.

## Code Style

Code formatting is handled **automatically** via [Prettier](https://prettier.io/) on pre-commit. However, you could install an [IDE extension](https://prettier.io/docs/en/editors.html) or run manually via `yarn format <prettier-args>`.

Linting will catch any further non-formatting issues:

- [TSLint](https://palantir.github.io/tslint/)
  - **Config**: [tslint.json](.config/tslint/tslint.json)
  - **Note**: As TSLint does not yet support TypeScript project `references` [(palantir/tslint#4137)](https://github.com/palantir/tslint/issues/4137), TSLint must be executed via `lerna run` to treat each package correctly.

### Component Structure

#### File Structure

Separate concerns on a component basis, rather than the more traditional "style vs. function".

With [Styled Components](https://www.styled-components.com/) we can take advantage of keeping style, markup and logic tightly bound together as a single `.tsx` file. A component file should be defined in the following order:

1. Imports
1. External (e.g. `node_modules`)
1. Internal (i.e. a different package from the design system)
1. Component partials
1. Styles
1. Settings (e.g. `const`)
1. Helper functions
1. Markup / logic
1. Exports

#### Documentation

Each component/consumable feature **must** have a README that follows the appropriate package's pre-defined `.templates/`:

- [`packages/components`](packages/components/.templates/README.mdx)

This should be written in [`.mdx`](https://mdxjs.com/) syntax, which will be included automatically in the [Style Guide](./README.md#style-guide).

#### Directory Structure

For a basic component, the directory structure would look something like this:

```
+————————————+
│   Button   │
+————————————+

components
├── button
│   ├── README.mdx
│   └── index.js
└── …
```

For larger components, consider breaking down into smaller partials:

```
+——ComponentName——+
│                 │
│      Title      │
│                 │
+—————————————————+
│                 │
│     Caption     │
│                 │
+—————————————————+

components
├── button
├── component-name
│   ├── caption.tsx
│   ├── index.ts
│   ├── README.mdx
│   ├── settings.ts
│   ├── title.tsx
│   └── utils.ts
└── …
```

**`settings`**

Common constants/raw values that can be re-used across the different component partials.

**`utils`**

Styles that are unique to the `component` but need to be shared across the partials.

These styles should be **flat**/**stateless** and not bound by any form of functionality, making use of the `CSSObject` type from [Styled Components](https://www.styled-components.com/) for compatibility.

#### Naming

- **Components** should be defined in **PascalCase**.

  For example: `<ComponentName/>`

  - Component partials should be prefixed by the component name:

    For example: `<ComponentNameTitle/>`

- **Flat re-usable styles** should be defined in **camelCase** to differentiate.

  - For example: `<div css={componentNameSizing}/>`

## Release Process

Thanks to [Lerna and Commitizen](https://github.com/lerna/lerna/tree/master/commands/version#--conventional-commits), we can generate version bumps and CHANGELOGs for each package and release automatically.

1. Assuming that all changes to be released have followed the [Git Workflow outlined above](#workflow), raise a pull request from [`develop` into `master`](https://github.com/coingaming/moon-design/compare/master...develop).
2. Ensure the pull request is up to date with `master`.
3. Obtain at least 1 approval.
4. Click the "Merge Pull Request" button to trigger an automatic release, monitoring progress in [CircleCI](https://circleci.com/gh/coingaming/moon-design).

   - If successful, a new [tag](https://github.com/coingaming/moon-design/releases) will be visible, along with new package versions on NPM.

5. Communicate the release to the team!

## ES Modules

---

### Rules

1. All your files should be named after the actual component. So the component `GroupIconCurrency` is inside the file `GroupIconCurrency.tsx`.
1. All your components should export a single default React component. This is only so we can enable tree shaking.
1. All first level subfolders inside `/src/` are considered public. People will use `import { } from @heathmont/moon-package-name/folder` to import files from those folders directly.
1. You are required to auto-generate an index file to for each subfolder which uses named exports to re-export all exports inside the folder + sub folders. This is required for tree shaking.
1. You are required to auto-generate an index file to for your package which uses named exports to re-export all exports. This is required to allow lerna to import the project proeprly.

### Build process

All packages are built in 2 separate formats, `commonjs` and `es`.

All code belongs in `/src` folder. When running yarn build, it will automatically generat a `lib` folder based on the code
inside `src`.

First we transpile `commonjs` format modules into the `lib` folder directly.
Second we transpile `es` format modules into the `lib/es` folder.
Third we generated typescript definitions into the `lib` folder directly.

### Release process

**Before** we release, we will:

1. Copy all files from `lib` folder into the root folder.
1. Update the `files`, `main`, `module` and `typings` fields in package.json

**After** we release, we will:

1. Remove all the files we copied over from `lib`.
1. Undo the changes to `files`, `main`, `module` and `typings` fields in package.json

license: MIT

gha_license_id: MIT


text:
---
layout: post
title: "New Release: Emoji Names 2.0"
post-id: emoji-names-2
thumbnail: thumbnail.png
photoswipe: true
tags:
  - Apps

screenshots:
  - src: "X screenshot 1.jpg"
    maxres: 1125 × 2436
  - src: "X screenshot 2.jpg"
    maxres: 1125 × 2436
  - src: "X screenshot 3.jpg"
    maxres: 1125 × 2436

---

**Emoji Names** is an iOS app I first wrote back in 2015. It shows you the Unicode name of any given Emoji, with immersive full-screen animations. This is the first big update for Emoji Names, bringing several new features that make the app feel a lot more useful.

{% include photoswipe.html images=page.screenshots max-height=450 %}

New Features

 - **Share Emoji** - The biggest new feature in Emoji Names is that you can share customizable images of each Emoij! You can set a custom background color, choose to show or hide the emoji's name, and toggle the emoji style between Apple's images and Twitter's vectors. This feature is the spiritual successor to [Emojicon](http://github.com/calda/Emojicon), one of my older apps that I recently removed from the App Store.

 - **Paste Emoji** - I got the suggestion for this feature from [Nate](http://natethompson.io/) and I thought it was a really cool idea — you can now copy a message in some other app, like iMessage, and then paste it into Emoji Names. The app will detect any emoji in the message and show you their name! This is pretty useful when somebody sends you an emoji you don't recognize.

 - **Twitter Emoji** - Emoji Names also supports an entirely new Emoji style, [Twemoji](https://github.com/twitter/twemoji) (Twitter's open source Emoji art pack)! This was born more out of necessity than desire, though. Apple has cracked down on the use of their Emoji art in App Store screenshots. It's pretty difficult to make marketing materials for an Emoji app without showing any Emoji, so I decided to add Twemoji support into the app. They come as .svg vectors, meaning they're significantly higher-res than Apple's system Emoji font.

Art for the sake of Art

Emoji Names is a nice little app, but it doesn't really get that many users. It doesn't do very well in App Store search results, both because "Emoji" is a pretty crowded search team and because it just isn't the kind of app people are searching for. I've been experimenting with a $0.99 price tag and using paid Search Ads to increase the audience, but I haven't gotten great results there (a cost-per-install of $5.50 isn't economically viable for a $0.99 app).

I didn't make Emoji Names with the expectation that it would be especially popular. I think it's perfectly reasonable to make an app purely for the joy of creating something "good". All of my first apps were built for fun, and I like to think it shows making apps can still be just as much my "hobby" as it can be my full-time job.

You can check out Emoji Names on the [App Store](https://itunes.apple.com/us/app/emoji-names/id1060405457?mt=8).

{% include appstore.html link='https://itunes.apple.com/us/app/emoji-names/id1060405457?mt=8' %}


text:
# BlockHashLoc

The purpose of BlockHashLoc is to enable the recovery of files after total loss of File System structures, or without even knowing what FS was used in the first place.

The way it can recover a given file is by keeping a (small) parallel BHL file with a list of crypto-hashes of all the blocks (of selectable size) that compose it. So it's then possible to read blocks from a disk image/volume, calculate their hashes, compare them with the saved ones and rebuild the original file.

With adequately sized blocks (512 bytes, 4KB, etc. depending on the media and File System), this let one recover a file regardless of the FS used, or the FS integrity, or the fragmentation level.

This project is related to [SeqBox](https://github.com/MarcoPon/SeqBox). The main differences are:

- SeqBox create a stand-alone file container with the above listed recovery characteristics.

- BHL realize the same effect with a (small) parallel file, that can be stored separately (in other media, or in the cloud), or along the original as a SeqBox file (so that it can be recovered too, as the first step), so it can be used to add a degree of recoverability to existing files.

**N.B.**

The tools are still in beta and surely not speed optimized, but they are already functional and the BHL file format is considered final.

## Demo tour

BlockHashLoc is composed of two separate tools:
 - BHLMake: create BHL files with block-hashes and metadata
 - BHLReco: recover files searching for the block's hashes contained in a set of BHL files

There are in some case many parameters but the default are sensible so it's generally pretty simple.

Here's a practical example. Let's see how 2 photos can be recovered from a fragmented floppy disk that have lost its FAT (and any other system section). The 2 JPEGs weight about 450KB and 680KB:

![Manu01](http://i.imgur.com/QKxgT5r.jpg) ![Manu02](http://i.imgur.com/jfQLlx1.jpg)

We start by creating the BHL files, and then proceed to test them to make sure they are all right:

```
c:\t>bhlmake *.jpg
creating file 'Manu01.jpg.bhl'...
  BHL file size: 29582 - blocks: 913 - ratio: 6.3%
creating file 'Manu02.jpg.bhl'...
  BHL file size: 43936 - blocks: 1363 - ratio: 6.3%

c:\t>bhlreco -t -bhl *.bhl
reading BHL file 'Manu01.jpg.bhl'...
reading BHL file 'Manu02.jpg.bhl'...
BHL file(s) OK!

```

Now we put both the JPEGs in a floppy disk image that have gone trough various cycles of files updating and deleting. At this point the BHL files could be kept somewhere else (another disk, some online storage, etc.), or put in the same disk image after being encoded in one or more [SeqBox](https://github.com/MarcoPon/SeqBox) recoverable container(s) - because, obviously, there's no use in making BHL files if they can be lost too.
As a result the data is laid out like this:

![Disk Layout](http://i.imgur.com/3MUOAjk.png)

The photos are in green, and the two SBX files in blue.
Then with an hex editor we zap the first system sectors and the FAT (in red), making the disk image unreadable!
Time for recovery!

We start with the free (GPLV v2+) [PhotoRec](http://www.cgsecurity.org/wiki/PhotoRec), which is the go-to tool for these kind of jobs. Parameters are set to "Paranoid : YES (Brute force enabled)" & "Keep corrupted files : Yes", to search the entire data area.
As the files are fragmented, we know we can't expect miracles. The starting sector of the photos will be surely found, but as soon as the first contiguous fragment end, it's anyone guess.

![PhotoRec results](http://i.imgur.com/y9phKLX.png)

As expected, something has been recovered. But the 2 files sizes are off (32K and 340KB). The very first parts of the photos are OK, but then they degrade quickly as other random blocks of data where mixed in. We have all seen JPEGs ending up like this:

![Manu01](http://i.imgur.com/bCtYJpW.jpg) ![Manu02](http://i.imgur.com/EmOid42.jpg)

Other popular recovery tools lead to the same results. It's not anyone fault: it's just not possible to know how the various fragment are concatenated, without an index or some kind of list (there are approaches based on file type validators that can in at least some cases differentiate between spurious and *valid* blocks, but that's beside the point).

But having the BHL files at hand, it's a different story. Each of the blocks referenced in the BHL files can't be fragmented, and they all can be located anywhere in the disk just by calculating the hash of every blocks until all matching ones are found.

So, the first thing we need is to obtain the BHL files, either by getting them from some alternate storage, or recovering the [SeqBox](https://github.com/MarcoPon/SeqBox) containers from the same disk image and extracting them.

Then we can run BHLReco and begin the scanning process:

```
c:\t>bhlreco disk.IMA -bhl *.bhl
creating ':memory:' database...
reading BHL file 'Manu01.jpg.bhl'...
updating db...
reading BHL file 'Manu02.jpg.bhl'...
updating db...
scan step: 512
scanning file 'disk.IMA'...
  90.4% - tot: 2274 - found: 2274 - 40.65MB/s
scan completed.
creating file 'Manu01.jpg'...
hash match!
creating file 'Manu02.jpg'...
hash match!

files restored: 2 - with errors: 0 - files missing: 0
```

All files have been recovered, with no errors!
Time for a quick visual check:

![Manu01](http://i.imgur.com/qEB9wBQ.jpg) ![Manu02](http://i.imgur.com/s6spyFq.jpg)

N.B. Here's a [7-Zip archive](http://mark0.net/download/bhldemo-diskimage.7z) with the disk image and the 2 BHL files used in the demo (1.2MB).

## Tech spec

Byte order: Big Endian

Hash: SHA-256

### BHL file structure

| section    | desc                                 | note      |
| ---------- | ------------------------------------ | --------- |
| Header     | Signature & version                  |           |
| Metadata   | Misc info                            |           |
| Hash       | Blocks hash list & final hash        |           |
| Last block | zlib compressed last block remainder | if needed |

### Header

| pos | to pos | size | desc                              |
|---- | ---    | ---- | --------------------------------- |
|  0  |     12 |  13  | Signature = 'BlockHashLoc' + 0x1a |
| 13  |     13 |   1  | Version byte                      |
| 14  |     17 |   4  | Block size                        |
| 18  |     25 |   8  | File size                         |

### Metadata

| pos | to pos | size | desc                  |
|---- | ------ | ---- | --------------------- |
| 26  |     29 |   4  | Metadata section size |
| 30  |    var |  var | Encoded metadata list |

### Hash

| pos | to pos | size | desc                              |
|---- | ------ | ---- | --------------------------------- |
| var |    var |   32 | 1st block hash                    |
| ... |    ... |   32 | ...                               |
| var |    var |   32 | Last block hash                   |
| var |    var |   32 | Hash of all previous block hashes |

### Versions:

Currently the only version is 1.

### Metadata encoding

| Bytes | Field |
| ----- | ----- |
|    3  | ID    |
|    1  | Len   |
|    n  | Data  |

#### IDs

| ID | Desc |
| --- | --- |
| FNM | filename (utf-8)                           |
| FDT | date & time (8 bytes, seconds since epoch) |

(others IDs may be added...)

## Links

 - [BlockHashLoc home page](http://mark0.net/)
 - [BlockHashLoc GitHub repository](https://github.com/MarcoPon/BlockHashLoc)

## Credits

The idea of collecting & scanning for block hashes was something I had considered while developing [SeqBox](https://github.com/MarcoPon/SeqBox), then settling on using a stand alone file container instead of the original file plus a parallel one.

Then the concept resurfaced during a nice discussion on Slashdot with user JoeyRoxx, and after some considerations I decided to put some work on that too, seeing how the two approaches could both be useful (in different situations) and even complement each other nicely.

## Contacts

If you need more info, want to get in touch, or donate: [Marco Pontello](http://mark0.net/contacts-e.html)

**Bitcoin**: 1Mark1tF6QGj112F5d3fQALGf41YfzXEK3

![Qr-Code](http://mark0.net/images/qrcode.png)

repository: shakenetwork/BlockHashLoc

repo_name: shakenetwork/BlockHashLoc


text:
Fork of [Monte Python](https://github.com/brinckmann/montepython_public) with a logflat prior.

```python
data.parameters['sigma_dmeff'] = [1e-28, 1.e-30, 1e-18, 1e-20,	1, 'cosmo', 'logflat']
```

----

# Monte Python, a Monte Carlo Markov Chain code (with Class!)

* Main developer: Thejs Brinckmann<EMAIL_ADDRESS>* Aithor: Benjamin Audren<EMAIL_ADDRESS>* License: MIT

The code is under the MIT license. As an additional clause, when using the code
in a scientific publication you are also required to cite the v3.0 release paper
``MontePython 3: boosted MCMC sampler and other features`` and the original release
paper ``Conservative Constraints on Early Cosmology`` (see the tail of this document
for the bibtex entries).

# Bibtex entry

When using *Monte Python* in a publication, please acknowledge the code by citing
the following papers. If you used *Class*, *MultiNest*, *PolyChord* or *Cosmo Hammer*,
you should also cite the original works.

Please also cite the relevant papers for each likelihood used: as of v3.0 we have a
list of references for all likelihoods in the first of the papers below. In the
future the list will be maintained on the official `Monte Python website
<https://brinckmann.github.io/montepython_public/>`_. Otherwise, this information can
often be found in the .data file of the likelihood folder.

In order to encourage people to both develop and share likelihoods with the community,
to the benefit of all users, we optionally encourage users to cite the paper in which
the *Monte Python* likelihood was first used, in addition to the papers in which data
and/or likelihoods were published.

```

    @article{Brinckmann:2018cvx,
          author         = "Brinckmann, Thejs and Lesgourgues, Julien",
          title          = "{MontePython 3: boosted MCMC sampler and other features}",
          year           = "2018",
          eprint         = "1804.07261",
          archivePrefix  = "arXiv",
          primaryClass   = "astro-ph.CO",
          SLACcitation   = "%%CITATION = ARXIV:1804.07261;%%"
    }
    @article{Audren:2012wb,
          author         = "Audren, Benjamin and Lesgourgues, Julien and Benabed,
                            Karim and Prunet, Simon",
          title          = "{Conservative Constraints on Early Cosmology: an
                            illustration of the Monte Python cosmological parameter
                            inference code}",
          journal        = "JCAP",
          volume         = "1302",
          pages          = "001",
          doi            = "10.1088/1475-7516/2013/02/001",
          year           = "2013",
          eprint         = "1210.7183",
          archivePrefix  = "arXiv",
          primaryClass   = "astro-ph.CO",
          reportNumber   = "CERN-PH-TH-2012-290, LAPTH-048-12",
          SLACcitation   = "%%CITATION = ARXIV:1210.7183;%%",
    }
```


text:
# #047 Hash Define or Const?

Comparing the use of `#define` preprocessor macros and `const` statements.

## Notes

Anyone who has some basic C education, or who has had to pickup and use some existing code, will have encountered preprocessor macros: `#define ...`.

So much so, there's a tendency to use them indiscriminately.

The article [How and Why to Avoid Preprocessor Macros](https://luckyresistor.me/knowledge/avoid-preprocessor-macros/) provides some sage advice.

I want to avoid making the mistake of replacing one set of "rules" for another. Instead, I think it is useful to think in terms of alternatives
i.e. when considering a preprocessor macro, first consider whether an alternative may be a better choice:

* `const` for defining constants
* `#pragma once` for header guards

## Preprocessor Macros

Preprocessor macros are commonly used to define mnemonics in C code.

But they can lead to much trouble, especially when one forgets they are really performing test substitution.

Here's a classic error:

```
#define A 1
#define B A + 1
#define C 3
#define X B * C
```

What is `X`? If you read those statements as expressions, the answer of 6 seems obvious, but that's wrong.

`X` is a macro that is expanded to `1 + 1 * 3` and so yields an expression that evaluates to 4.

This can be fixed with liberal deployment of parentheses:
```
#define A (1)
#define B (A + 1)
#define C (3)
#define X (B * C)
```

`X` now expands to `(((1) + 1) * (3))` which evaluates to 6 as is probably intended.

Parentheses are almost always a good idea if you must put numbers and expressions in macros. Better safe than sorry.

## Using Constants

Constants provide type safety and also behave as statements (which may better match their actual usage).

```
const int A = 1;
const int B = A + 1;
const int C = 3;
const int X = B * C;
```

If `X` is really the only value that matters for the program, and `A`, `B`, and `C` are really just inline documentation of how `X` is derived,
then one might finesse the implementation to use a mix of macros and constants:

```
#define A (1)
#define B (A + 1)
#define C (3)
const int X = B * C;
```

## Demo Program

The [demo.c](./demo.c) program demonstrates using `const` as an alternative for `#define` macros:

```
r$ make && ./demo
gcc -g -Wall -O3    demo.c   -o demo
The power of parentheses...
X  = B * C  = A + 1 * 3   = 1 + 1 * 3   = 4 (wrong!)
X2 = B2 * C = (A + 1) * 3 = (1 + 1) * 3 = 6 (correct)
Using const overcomes the issues and with extra type safety...
X3 = B3 * C3 = (A3 + 1) * 3 = (1 + 1) * 3 = 6 (correct)
```

## Credits and References
* [How and Why to Avoid Preprocessor Macros](https://luckyresistor.me/knowledge/avoid-preprocessor-macros/)
* [GCC Manual - 3 MAcros](https://gcc.gnu.org/onlinedocs/cpp/Macros.html)

repository: tardate/LittleCodingKata

branch_name: refs/heads/main

gha_language: JavaScript

repo_name: tardate/LittleCodingKata


text:
# wax-slots

# installation instructions

  [installation instructions](docs/installation.md)

# config

  add a config file with the following information:

```js
{
  "cookieSecret": "<hex string>",
  "waxIdSeed": "<hex string>",
  "centralWalletSeed": "<hex string>",
  "burnAccount": "crptomonkeys"
}
```

## todo

check out minting here:

    https://www.npmjs.com/package/atomicassets

    https://gist.github.com/Vyryn/ea4ae93818c87a57ef63f4056a21de01#file-create_claimlink-py-L51

    https://github.com/EOSIO/eosjs#signature-provider

    https://wax.bloks.io/account/atomictoolsx?tab=ABI&account=atomictoolsx

## add user so you aren't running as root

  adduser waxslots;

  su - waxslots;

# run the following command to start:

  npm start;

## to run in background, use the command:

  npm run screenstart;

## to stop, use the command:

  npm stop;

### to stop and restart, use the command:

  npm run screenrestart;

### todo

repository: DoctorNasa/banano-wax-slots

repo_name: DoctorNasa/banano-wax-slots


text:
# Database-Replication
Replicate MySQL database to Microsoft SQL Server and vice-versa

## Setup
1. Setup the App.config with the respective connection strings to run it locally
2. On MySQL, to enable the query statements going to a MySQL table, commands are: SET GLOBAL log_output = "TABLE"; SET GLOBAL general_log = 'ON';

## Improvements
Can be improved in terms of abstract out logging, constants, add unit tests, comments, etc. Also search for string "Improvements" in the code to find few comments from me.

repository: nGenieDeveloper/Database-Replication

repo_name: nGenieDeveloper/Database-Replication


text:
# Contributing
👍🎉 First off, thanks for taking the time to contribute! 🎉👍

When contributing to this repository, please first discuss the change you wish to make via issue, email, or any other method with the owners of this repository before making a change.

Please note we have a [code of conduct](https://github.com/PythonFreeCourse/calendar/blob/master/CODE_OF_CONDUCT.md), please follow it in all your interactions with the project.

## Pull Requests
Create feature branches.

One pull request per feature - If you want to do more than one thing, send multiple pull requests.

Send coherent history - Make sure each individual commit in your pull request is meaningful. If you had to make multiple intermediate commits while developing, please [squash them](https://www.git-scm.com/book/en/v2/Git-Tools-Rewriting-History#Changing-Multiple-Commit-Messages) before submitting.

## Guide
1. Fork the project.
2. Create a new branch (`git checkout -b 'branch name'`).
3. Commit your changes (`git commit -m 'Add something new'`).
4. Push to the branch (`git push -u origin 'branch name'`).
5. Open a Pull Request.

## Style Guide
Follow the [commit messages specification](https://www.conventionalcommits.org/en/v1.0.0/).

Happy coding!


text:
![zerolib logo](https://raw.githubusercontent.com/01010111/zerolib/storage/zerolib.png)

# ZEROLIB

![build status](https://api.travis-ci.org/01010111/zerolib.png)

Zerolib is a framework agnostic library of extensions, utilities, and other helpful classes for making games!

**notice: this library is in an alpha state, and may introduce breaking changes with updates**

## Installing
With Haxe installed, use this command to install:
```
haxelib install zerolib
```
Then, add the following to your game's `hxml`:
```
-lib zerolib
```
Then you'll be set!

## Usage
This library is split into two distinct sections, Extensions, and Utilities.

Check the [API](http://01010111.com/zerolib/) for specific usage instructions!

### .extensions
Extensions are a neat language feature of Haxe. They can add functionality to preexisting classes.

Learn more about using Extensions [here](https://github.com/01010111/zerolib/wiki/Extensions)!

### .utilities
A handful of utilities like vectors, a tweening library, a A* haxe implementation, and more!

**[Browse the API](http://01010111.com/zerolib/)**

gha_language: Haxe


text:
# novaugust.net

a super simple responsive landing page.

## Using this code

Obviously, you'll want to rebrand all the ld+json data, your links, etc.

**Please remove the track code from the bottom of the page!**

Someone used my code for a porn site once.
It took me a while to figure out why I was suddenly getting all this traffic from Turkey and Russia -
not to mention to such explicit paths.

So yeah. Respect your visitors and don't send their data to me.

### Features

1. super simple html5 layout
2. flexbox is real darn nifty
3. font awesome
4. ld+json in the header for mo' betta' search engine-ing
5. in-page css to minimize requests

## in the wild

It's always rewarding to see folks using my code. Hit me up if you do!

- [matt enlow](https://novaugust.net) (that's me)
- [adam merry](http://www.adammerry.com/)

## license

MIT


text:
### [CVE-2020-16849](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-16849)
![](https://img.shields.io/static/v1?label=Product&message=n%2Fa&color=blue)
![](https://img.shields.io/static/v1?label=Version&message=n%2Fa&color=blue)
![](https://img.shields.io/static/v1?label=Vulnerability&message=n%2Fa&color=brighgreen)

### Description

An issue was discovered on Canon MF237w 06.07 devices. An "Improper Handling of Length Parameter Inconsistency" issue in the IPv4/ICMPv4 component, when handling a packet sent by an unauthenticated network attacker, may expose Sensitive Information.

### POC

#### Reference
- https://blog.scadafence.com/vulnerability-report-cve-2020-16849

#### Github
No PoCs found on GitHub currently.

repository: trickest/cve

gha_language: HTML

repo_name: trickest/cve


text:
<!--
_[Please make sure you have read the submission guidelines before posting an PR](https://github.com/nrwl/nx/blob/master/CONTRIBUTING.md#submit-pr)_

# Community Plugin Submission

Thanks for submitting your Nx Plugin to our community plugins list. Make sure to follow these steps to ensure that your PR is approved in a timely manner.

## Steps to Submit Your Plugin
- Use the following commit message template: `chore(core): nx plugin submission [PLUGIN_NAME]`
- Update the `community/approved-plugins.json` file with a new entry for your plugin that includes `name`, `url`, `description`:

Example:

```json
// community/approved-plugins.json

[{
    "name": "@community/plugin",
    "url": "https://github.com/community/plugin",
    "description": "This plugin provides the following capabilities."
}]
```

Once merged, your will plugin will be available when running the `nx list` command, and will also be available in the Plugin browser on [nx.dev](https://nx.dev)
-->

# Community Plugin Submission

## {replace with plugin's name}

<!--
Describe what your plugin is and what is its goal or issues it addresses.  If you don't provide a description, we will not merge your PR.
Is it focused on a technology, tooling or behaviour? Does the plugin provide generators, executors or graph support?
Do you know who is already using the plugin? Mention who is the author of the plugin.
-->

repository: jsaguet/nx

repo_name: jsaguet/nx


text:
# class: SnapshotAssertions
* since: v1.20
* langs: js

:::tip
For visual regression testing, use [`method: PageAssertions.toHaveScreenshot#1`] and [`method: LocatorAssertions.toHaveScreenshot#1`] instead.
:::

Playwright provides methods for comparing text values with expected values stored in snapshot files.

```js
expect(text).toMatchSnapshot('snapshot.txt');
```

## method: SnapshotAssertions.toMatchSnapshot#1
* since: v1.22

Ensures that the passed [string] matches the expected snapshot stored in the test snapshots directory.

**Usage**

```js
// Basic usage.
expect(await page.title()).toMatchSnapshot('page-title.txt');

// Bring some structure to your snapshot files by passing file path segments.
expect(await page.title()).toMatchSnapshot(['page-title', 'step1.txt']);
expect(await page.title()).toMatchSnapshot(['page-title', 'step2.txt']);
```

Note that matching snapshots only work with Playwright test runner.

### param: SnapshotAssertions.toMatchSnapshot#1.name
* since: v1.22
- `name` <[string]|[Array]<[string]>>

Snapshot name.

## method: SnapshotAssertions.toMatchSnapshot#2
* since: v1.22

Ensures that passed value, either a [string] or a [Buffer], matches the expected snapshot stored in the test snapshots directory.

**Usage**

```js
// Basic usage.
expect(await page.title()).toMatchSnapshot();

// Configure the snapshot name.
expect(await page.title()).toMatchSnapshot({
  name: 'page-title.txt',
});
```

Note that matching snapshots only work with Playwright test runner.

### option: SnapshotAssertions.toMatchSnapshot#2.name
* since: v1.22
- `name` <[string]|[Array]<[string]>>

Snapshot name. If not passed, the test name and ordinals are used when called multiple times.

## method: SnapshotAssertions.toMatchSnapshot#3
* deprecated: To avoid flakiness, use [`method: PageAssertions.toHaveScreenshot#1`] instead.
* since: v1.22

Ensures that the passed [Buffer] matches the expected snapshot stored in the test snapshots directory.

**Usage**

```js
// Basic usage.
expect(await page.screenshot()).toMatchSnapshot('landing-page.png');

// Pass options to customize the snapshot comparison and have a generated name.
expect(await page.screenshot()).toMatchSnapshot('landing-page.png', {
  maxDiffPixels: 27, // allow no more than 27 different pixels.
});

// Configure image matching threshold.
expect(await page.screenshot()).toMatchSnapshot('landing-page.png', { threshold: 0.3 });

// Bring some structure to your snapshot files by passing file path segments.
expect(await page.screenshot()).toMatchSnapshot(['landing', 'step2.png']);
expect(await page.screenshot()).toMatchSnapshot(['landing', 'step3.png']);
```

Learn more about [visual comparisons](../test-snapshots.md).

Note that matching snapshots only work with Playwright test runner.

### param: SnapshotAssertions.toMatchSnapshot#3.name
* since: v1.22
- `name` <[string]|[Array]<[string]>>

Snapshot name.

### option: SnapshotAssertions.toMatchSnapshot#3.maxDiffPixels = %%-assertions-max-diff-pixels-%%
* since: v1.22

### option: SnapshotAssertions.toMatchSnapshot#3.maxDiffPixelRatio = %%-assertions-max-diff-pixel-ratio-%%
* since: v1.22

### option: SnapshotAssertions.toMatchSnapshot#3.threshold = %%-assertions-threshold-%%
* since: v1.22

## method: SnapshotAssertions.toMatchSnapshot#4
* deprecated: To avoid flakiness, use [`method: PageAssertions.toHaveScreenshot#1`] instead.
* since: v1.22

Ensures that the passed [Buffer] matches the expected snapshot stored in the test snapshots directory.

**Usage**

```js
// Basic usage and the file name is derived from the test name.
expect(await page.screenshot()).toMatchSnapshot();

// Pass options to customize the snapshot comparison and have a generated name.
expect(await page.screenshot()).toMatchSnapshot({
  maxDiffPixels: 27, // allow no more than 27 different pixels.
});

// Configure image matching threshold and snapshot name.
expect(await page.screenshot()).toMatchSnapshot({
  name: 'landing-page.png',
  threshold: 0.3,
});
```

Learn more about [visual comparisons](../test-snapshots.md).

Note that matching snapshots only work with Playwright test runner.

### option: SnapshotAssertions.toMatchSnapshot#4.maxDiffPixels = %%-assertions-max-diff-pixels-%%
* since: v1.22

### option: SnapshotAssertions.toMatchSnapshot#4.maxDiffPixelRatio = %%-assertions-max-diff-pixel-ratio-%%
* since: v1.22

### option: SnapshotAssertions.toMatchSnapshot#4.name
* since: v1.22
- `name` <[string]|[Array]<[string]>>

Snapshot name. If not passed, the test name and ordinals are used when called multiple times.

### option: SnapshotAssertions.toMatchSnapshot#4.threshold = %%-assertions-threshold-%%
* since: v1.22

repository: microsoft/playwright

gha_language: TypeScript

repo_name: microsoft/playwright


text:
Starter Data Platform

    An awesome project that will help taking the first step on your journey toward data-driven decisions.

# About

![full](https://user-images.githubusercontent.com/5653787/120114190-43a72680-c17e-11eb-897c-6323db88e8e5.png)

This repository provides a starting point for implementing a modern open source data platform. Data has become an essential part of organizations and their decision making process. Having an central repository with all the data, enables analysts to provide meaningful analytics for the rest of the organization.

This project will be used to explain and show how an data platform can be established with only open-source products.

Need info or have feedback? Do not hesitate to  [create an issue](https://github.com/Fredehagelund92/starter_data_platform/issues/new)

# Table of contents
- [About](#about)
- [Table of contents](#table-of-contents)
- [Components](#components)
  - [Storage](#storage)
  - [Extract](#extract)
  - [Transformation](#transformation)
  - [Orchestration](#orchestration)
  - [Visualization](#visualization)
- [Requirements](#requirements)
- [Getting Started](#getting-started)
  - [Setup Materialize](#setup-materialize)
  - [Setup Airbyte](#setup-airbyte)
  - [Setup dbt project](#setup-dbt-project)
  - [Setup Prefect](#setup-prefect)
    - [Clone project](#clone-project)
      - [Folder structure](#folder-structure)
    - [Install prefect](#install-prefect)
      - [Deploy dags](#deploy-dags)
  - [Development](#development)

# Components

## Storage

## Extract
[Airbyte](https://github.com/airbytehq/airbyte)  is a new open-source application that is creating the new standard for data integration. The CDK makes it easy to integrate your data from various datasources. There are other tools like Meltano, Kiba etc. however airbytes community is growing rapidly and its so easy to get started. The data integration is heavily influenced by singer.

When starting a new data project i always start by integrating following data:

* Holidays (Used to create a calendar/date dimension. This makes it way easier when supporting multiple countries)
* Addresses (Often source systems does not provide geospatial and detailed address information.)
* Exchange rates (Again if supporting multiple countries it might be needed to have metrics converted)

This enables me to quickly get started and generally it is not a problem, since storage is cheap.

## Transformation
[dbt](https://github.com/fishtown-analytics/dbt) introduces new standards to the data engineering and analyst space. For years people have been using GUI tools like Talend, SSIS, Pentaho, Alteryx etc. If you are familiar with one of these tools it can be rather quick to develop jobs. Personally i always end up banging my head against the wall due to following problems:

* Components are not very flexible and workarounds are often needed.
* Version control can be a mess due to project files.
* CI/CD for your data pipelines is limited or time consuming.

With dbt engineers and analysts are able to do their transformations using select statements and jinja macros. These statements are called "models" and can be build using the dbt CLI. dbt will then compile the models and execute it based on the desired behaviouclick 8.0.1r:

* Materialize a full table - full refresh
* Materialize a view - overwrite
* Append to existing table - incremental
* Ephemeral - data will not be stored in your database, but can instead be used as a common table expression in other models

The dbt framework also provide functionalities like:

* Documentation of models, datasources etx.
* Data lineage graph of relationships between models
* Schema and data tests using assertation (unique, not_null, accepted_values etc.)

## Orchestration
[Prefect](https://github.com/PrefectHQ/prefect)

## Visualization
[Redash](https://github.com/apache/airflow) is one of the two big Open-source applications for visualization. The other one is Metabase, which also have a large following. Whether you pick one or another depends on the organization and its competencies.

Redash reports are build directly on top SQL queries and have a lot of different types of visualizations. Metabase is more point and click type of reporting, which makes it easier for self-service setup.

# Requirements

![servers](https://user-images.githubusercontent.com/5653787/120114202-502b7f00-c17e-11eb-943f-7fa15663bd60.png)

The minimum requirement for servers are as follows:

* Materialization server
* Dagster server
* Airbyte server

The specific technical requirements heavily depends depends on various factors like data volume, number of analysts etc. If you want more details please check out the specific documentation of the applications.

For CI/CD i prefer a local [Github Actions](https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners) runner. For most cases it should be sufficient to install it on the Prefect server.

# Getting Started

## Setup Materialize
Start by installing Materialize on the first machine using [these instructions](https://materialize.com/docs/install/).

1. Login to your instance

    ```
    psql -U materialize -h localhost -p 6875 materialize
    ```

2. Create databases

    ```
    create database raw;
    create database production;
    create database development;
    ```

## Setup Airbyte
Start by installing Airbyte on the first machine using [these instructions](https://docs.airbyte.io/quickstart/deploy-airbyte).

1. Login to your instance

    ```
    psql -U materialize -h localhost -p 6875 materialize
    ```

## Setup dbt project

## Setup Prefect

### Clone project

#### Folder structure

### Install prefect

#### Deploy dags

## Development


text:
# Changelog for the DevOps OpenHack

## Changelog since v1.0.0

- Use of a container to provision the infrastructure used during the OpenHack
- Helm charts refactored to enable the use of helm package for the API. [@mathieu-benoit](https://github.com/mathieu-benoit)
- Helm charts refactored to include B/G support using labels/selectors.
- Replaced monitoring infrastructure with polling scripts to be run by the attendees (powershell and bash).
- Emphasise the use of YAML for pipelines in the challenge resources.

## v1.0.0

- Initial release as of December 2018


text:
---
domain: github.com
shortname: 21/HAEVENT
name: HA Event Subscription
status: draft
editor: Konstantin Nekrasov<EMAIL_ADDRESS>---

## Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC 2119](https://tools.ietf.org/html/rfc2119).

## Table of Contents

* [HA events](#ha-events)
* [Core ideas](#core-ideas)
* [Subscription procedure](#subscription-procedure)
* [HA event processing](#ha-event-processing)
  * [Notes](#notes)
  * [Corner case: RC re-election](#corner-case-rc-re-election)
* [References](#references)

## HA events

Hare is an integration point between Motr layer and HA. Hare must subscribe to so called HA Events to be able to react and translate them into Motr API terms if needed. This RFC describes how Hare subscribes to HA Events.

## Core ideas

1. Every hax process registers at Event Manager
   - Hare assumes that Event Manager will either create the corresponding topic or just return one if it exists. Hare doesn't need to verify if the topic already exists.
2. HA events will be processed by RC leader only
3. Last read offset will be stored in Consul KV
4. When RC leader is re-elected, the hax at the new RC leader node must start listening to the topic
   - And it should start from the offset stored in Consul KV

## Subscription procedure

![subscription](./ha-notifications-sequence.png)

  Diagram source

```plantuml
@startuml

autonumber

box "Node 1"
    participant hax
    participant EM as "Event Manager"
end box

box "Message Bus"
    participant Kafka
    participant Topic
end box

box "Node 2 (RC)"
    participant hax2 as "hax"
    participant EM2 as "Event Manager"
end box

hax -> EM: subscribe('hare', eventType)
EM -> EM: getOrCreateTopic()
EM -> Kafka: createChannel()
Kafka -> Topic **: create
note over Topic
    Topic is specific to Hare
end note
Topic --> Kafka
Kafka --> EM
EM --> hax: topic name

hax2 -> EM2: subscribe('hare', eventType)
EM2 -> EM2: getOrCreateTopic()
EM2 --> hax2: topic name
hax2 -> Topic: listen
note over hax2
    Only RC leader listens
    to the Topic
end note
@enduml
```

## HA event processing

![subscription](./ha-notifications-receival-sequence.png)

  Diagram source

```plantuml

@startuml

autonumber 11

box "Message Bus"
    participant Kafka
    participant Topic
end box

box "Node 2 (RC)"
    participant hax2 as "hax"
    participant EM2 as "Event Manager"
end box

database kv as "Consul KV"

activate hax2
activate Topic
activate kv
hax2 -> Topic: listen
Topic --> hax2:
activate hax2

note over hax2
    Only RC leader listens
    to the Topic
end note

Topic -> hax2: new event
hax2 -> hax2: process HA event
hax2 -> kv: store last confirmed offset
note right of kv
    The offset is specific to Topic.
    The offset is shared between all hax
    processes in the cluster.
    When RC changes, new RC will read out
    the Message Bus from the last confirmed offset.
end note
kv --> hax2
hax2 -> Topic: commit
@enduml
```

### Notes

1. The speed of the message processing will highly depend on the performance of Consul KV PUT operation
   - It means that ideally, Hare needs to use one topic only. In other words, we need to be able to subscribe to a number of HA events with a single call. According to the current design, HA allows to subscribe to specific events only; the more event types we're interested in, the more separate topics will be created. This should be changed.
2. At this moment, hax doesn't get notification "RC Leader is re-elected". I believe we can implement it by subscribing a watcher on leader key or something.
   - Alternative approach would be to use proto-rc script instead of hax. But that will make the processing chain longer and thus slower and more fragile:
     - `Kafka -> proto-rc script -> HTTP request -> hax`
     - `Kafka -> proto-rc script -> EQ -> Consul -> hax` (even longer)

### Corner case: RC re-election

When RC leader changes (e.g. due to HW issue) it may happen that previous RC didn't finish processing a message from MessageBus. So the message was read out from the queue but it's processing wasn't finished.

The problem can be alleviated by the approach as follows.

1. RC Hax receives a message from MessageBus
1. RC Hax double checks "am I still RC" (if yes, then proceed, otherwise just ignore the message and stop listening MessageBus)
1. Process the message
   - If processing means sending out messages to Motr world, then send everything needed and wait until Motr side confirms delivery.
1. RC Hax double checks "am I still RC" (if yes, then proceed, otherwise just ignore the message and stop listening MessageBus)
1. And only after that update "last confirmed" offset in Consul KV

**Potential problems**

1. During point \[3\] either a failure can occur to RC node or RC leader can be just re-elected. When another RC starts working, the same message will be taken for processing and Motr messages will be repeated.
   - Suggestion:
     1. Instead of one state "Message confirmed" add two states "Last taken offset" and "Last confirmed ofset" (analogy: 2-phase commit in databases)
     2. When another RC starts working it can at least see if the offset was taken by another RC previously. There is a chance that that RC is alive and it is in the middle of processing that message. Probably the current RC must delay for some time until assuming that another RC is not healthy and the message must be processed once again.

## References

1. HA Event Manager LLD ([link](https://seagatetechnology.sharepoint.com/:w:/s/gteamdrv1/tdrive1224/EQG6wcfCtJJFvG-1QLTqAL8BmyKZYAbkE33TFKepvd6kuA?e=xkGIh1))
2. HA HLD ([link](https://seagatetechnology.sharepoint.com/:w:/r/sites/gteamdrv1/tdrive1224/_layouts/15/Doc.aspx?sourcedoc=%7B2C9E9047-0884-4F26-AF0D-A2CD2B51995E%7D&file=CORTX_HA_HLD.docx&action=default&mobileredirect=true))

repository: max-seagate/cortx-hare

repo_name: max-seagate/cortx-hare


text:
# Ansible Role Oh My ZSH

![](https://github.com/ctorgalson/ansible-role-oh-my-zsh/workflows/Molecule%20Test/badge.svg)

This is a basic Ansible role to enable and configure Oh My Zsh on Ubuntu or
MacOS. It should also work on many other \*nix variants. It performs the
following tasks:

- Install and minimally configure Zsh:
  - make sure it exists,
  - set it as the default shell for the user specified by the role.
- Install Oh My Zsh for each specified user (in `~/.oh-my-zsh` by default).
- Configure (Oh My) Zsh by optionally creating a `.zshrc` file for each
  specified user.
- Alternately, configure Zsh by adding a block of lines to individual
  users' `.zshrc` files.

## Role variables

| Variable name  | Default value | Description |
|----------------|---------------|-------------|
| `omz_install_zsh` | `false` | Defines whether or not the role should attempt to install Zsh. |
| `omz_user` | `[]` | The user to install/configure (Oh My) Zsh for. See below for its properties. |
| `omz_user.name` | `-` | The name of the user. |
| `omz_user.group` | `-` | The group of the user |
| `omz_user.settings` | `-` | Extra settings (as a mult-line string) such as variable exports or aliases to add to the user's `.zshrc` file. Only used if `omz_zshrc_create` is `true`. |
| `omz_git_repository` | `https://github.com/robbyrussell/oh-my-zsh.git` | The git repo to clone Oh My Zsh from. |
| `omz_install_directory` | `.oh-my-zsh` | The name of the directory to clone Oh My Zsh into. |
| `omz_zshrc_create` | `true` | Whether or not to create `.zshrc`. If `true`, will create `.zshrc` from a template. |
| `omz_zshrc_template` | `templates/zshrc.zsh-template.j2` | The template used to create the user's `.zshrc` file when `omz_zshrc_create` is `true`. |
| `omz_zshrc_backup` | `true` | Whether or not to create backup the existing `.zshrc` files when the role changes it. |
| `omz_zsh_theme` | `robbyrussell` | See `templates/zshrc.zsh-template`. |
| `omz_case_sensitive` | `false` | See `templates/zshrc.zsh-template`. |
| `omz_hyphen_insensitive` | `false` | See `templates/zshrc.zsh-template`. |
| `omz_disable_auto_update` | `false` | See `templates/zshrc.zsh-template`. |
| `omz_update_zsh_days` | `13` | See `templates/zshrc.zsh-template`. |
| `omz_disable_ls_colors` | `false` | See `templates/zshrc.zsh-template`. |
| `omz_disable_auto_title` | `false` | See `templates/zshrc.zsh-template`. |
| `omz_enable_correction` | `false` | See `templates/zshrc.zsh-template`. |
| `omz_completion_waiting_dots` | `false` | See `templates/zshrc.zsh-template`. |
| `omz_disable_untracked_files_dirty` | `false` | See `templates/zshrc.zsh-template`. |
| `omz_hist_stamps` | `mm/dd/yyyy` | See `templates/zshrc.zsh-template`. |
| `omz_zsh_custom` | `$ZSH/custom` | See `templates/zshrc.zsh-template`. |
| `omz_plugins` | `[]` | A list of Oh My Zsh plugins to enable. |

## Role task files

### `main.yml`: task coordination

This file includes files that peform specific subsets of tasks.

### `zsh.yml`: Zsh setup

This task installs and sets zsh as the default shell for a user.

#### Variables used

- `omz_user`

### `oh-my-zsh-install.yml`: Oh My Zsh installation

This task clones the Oh My Zsh repository into the user directory of each
specified user and sets the appropriate permissions on the directory.

#### Variables used

- `omz_user`
- `omz_install_directory`
- `omz_git_repository`
- `omz_install_path`

### `oh-my-zsh-zshrc.yml`: Oh My Zsh configuration

This task creates the user a `.zshrc` file containing global values for various
Oh My Zsh options based on [the `.zshrc` template in the oh-my-zsh repository](https://raw.githubusercontent.com/robbyrussell/oh-my-zsh/master/templates/zshrc.zsh-template).
The task can be configured to back up any existing `.zshrc` file.

This task only runs when `omz_zshrc_create` is set to `true`.

#### Variables used

- `omz_user`
- `omz_zshrc_template`
- `omz_zshrc_backup`

### `zsh-zshrc.yml`: final Zsh configuration

This task adds individual lines to the `.zshrc` file. This is useful for adding
Zsh settings on an already-existing `.zshrc` file without creating it
from scratch.

This task only runs when `omz_zshrc_create` is set to `false`.

#### Variables used

- `omz_user`
- `omz_zshrc_backup`

## Sample playbook

    ---
    - name: Playbook
      hosts: all
      become: true

      vars:
        omz_install_zsh: true
        users:
          - name: "lorem"
            group: "lorem"
            settings: ""
          - name: "ipsum"
            group: "ipsum"
            settings: |
              export PATH="/usr/local/sbin:$path"
              alias l="ls -AF"
      tasks:
        - name: Run ansible-role-oh-my-zsh.
          include_role:
            name: "ansible-role-oh-my-zsh"
          vars:
            omz_user: "{{ item }}"
            # Only create `.zshrc` for user 'lorem'; item.settings will be
            # appended to `.zshrc` for the user 'ipsum'.
            omz_zshrc_create: "{{ (item.name == 'lorem') | ternary(true, false) }}"
            omz_plugins:
              - "autojump"
              - "git"
          with_items: "{{ users }}"

repository: ctorgalson/ansible-role-oh-my-zsh

gha_language: Python

repo_name: ctorgalson/ansible-role-oh-my-zsh


text:
# googlecontroller
GITHUB EXCLUSIVE KNOWLEDGE FOR SERVE_MEDIA HERE
Library for Python 3.9+ to push text message or audio file with the Google Home.Originally made by - Thomas Deblock (@tdeblock) I have expanded most of the code quite a bit little of the original remains, but the idea came from Mr.Deblock.This is a sorta v2 for the googlehomepush module they made.

## Installation

finally a pypi!!!!!!!
```
pip install googlecontroller
```
## Depending On

PyChromeCast
Pyngrok

## How to use

``` python
from googlecontroller import GoogleAssistant
from googlecontroller.http_server import serve_file # for local files
host = "ip"
home = GoogleAssistant(host=host)
home.say("test")
home.play("http://www.hubharp.com/web_sound/BachGavotteShort.mp3, opentunnel = 0") # we are doing opentunnel as this is the first time doing it
home.play("http://www.hubharp.com/web_sound/BachGavotteShort.mp3) # you do this after the first opening

#When serving media NEVER USE A \ ONLY USE /
#when doing your first home.serve_media you have to include a 3rd variable, opentunnle!For that 1 first time you have to set it manually to a 0!
#opentunnel = 0
#YOU MUST USE A DELAY IF DOING MULTIPLE IN A ROW FINE TUNE AS YOU SEE FIT
#like this home.serve_media("YourMedia.mp3", "C:/Users/YOU!/Music/", opentunnel)
#**then never use that variable again**
home.serve_media("YourMedia.mp3", "C:/Users/YOU!/Music/") # 1st is the name of the media, second is the full path to media location!
home.volume(100)
home.volume(0)

```
### .say(text, speed,ignore, lang)

Push a message on Google home

- `text` is the test message to say
- `speed` is the rate of speed of the message ranges from 0.000+ as slowest to 1 as normal speed.
- `ignore` ignore if audio is playing and play it regardless if ignore=True and only play if not playing if ignore=False or is not specified.
- `lang` the text language, default value is 'en' to change it have lang = 'language' as described in google translate en-Us, es (spanish), ect

### .play(url, ignore, contentType = 'audio/mp3'):

Push a sound to Google home
- `url` an audio file URL
- `ignore` ignore if audio is playing and play it regardless if ignore=True and only play if not playing.If ignore is not specified it will be set to False.
- `contentType` the audio file content type

### .volume(volumelevel):
- `volumelevel` the volume level from 0-100 (must be a integer) Example: home.volume(5) or home.volume(volumelevel=5). If you want to take it as user input you can do home.volume(volumelevel=input()).

## Maintainers
- Dray-Cyber

repository: dray-cyber/googlecontroller

repo_name: dray-cyber/googlecontroller


text:
---
layout: post
title: "A Projected Gradient Descent Method for CRF Inference allowing End-To-End Training of Arbitrary Pairwise Potentials"
date: 2018-01-02 10:31:27
categories: arXiv_CV
tags: arXiv_CV Segmentation Semantic_Segmentation Inference Deep_Learning Prediction Gradient_Descent
author: Måns Larsson, Anurag Arnab, Fredrik Kahl, Shuai Zheng, Philip Torr
mathjax: true
---

* content
{:toc}

##### Abstract
Are we using the right potential functions in the Conditional Random Field models that are popular in the Vision community? Semantic segmentation and other pixel-level labelling tasks have made significant progress recently due to the deep learning paradigm. However, most state-of-the-art structured prediction methods also include a random field model with a hand-crafted Gaussian potential to model spatial priors, label consistencies and feature-based image conditioning.
 In this paper, we challenge this view by developing a new inference and learning framework which can learn pairwise CRF potentials restricted only by their dependence on the image pixel values and the size of the support. Both standard spatial and high-dimensional bilateral kernels are considered. Our framework is based on the observation that CRF inference can be achieved via projected gradient descent and consequently, can easily be integrated in deep neural networks to allow for end-to-end training. It is empirically demonstrated that such learned potentials can improve segmentation accuracy and that certain label class interactions are indeed better modelled by a non-Gaussian potential. In addition, we compare our inference method to the commonly used mean-field algorithm. Our framework is evaluated on several public benchmarks for semantic segmentation with improved performance compared to previous state-of-the-art CNN+CRF models.

##### Abstract (translated by Google)
我们是否在视觉社区中流行的条件随机场模型中使用了正确的潜在函数？由于深度学习的范式，语义分割和其他像素级标签任务近来取得了重大进展。然而，大多数最先进的结构化预测方法还包括具有手工制作的高斯势的随机场模型，以对空间先验，标签一致性和基于特征的图像调节进行建模。
 在本文中，我们通过开发一个新的推理和学习框架来挑战这个观点，这个框架可以学习仅仅依赖于图像像素值和支持的大小而受到限制的CRF成对势。考虑标准空间和高维双向内核。我们的框架是基于CRF推断可以通过投影的梯度下降来实现的，因此可以很容易地集成到深度神经网络中，以便进行端到端的训练。经验证明，这种学习的潜力可以提高分割的准确性，某些标签类的相互作用确实更好地模拟非高斯的潜力。此外，我们将我们的推理方法与常用的平均场算法进行比较。我们的框架在几个公共的基准评估语义分割，与先前的CNN + CRF模型相比，性能有所提高。

##### URL
[http://arxiv.org/abs/1701.06805](http://arxiv.org/abs/1701.06805)

##### PDF
[http://arxiv.org/pdf/1701.06805](http://arxiv.org/pdf/1701.06805)


text:
This provides some examples of how some of the HHVM configuration options can be used to affect how a program is run. This is a work in progress and more examples will be added.

## Forcing Hack Mode

Imagine you have a program like this

@@ examples-examples/force.php @@

Notice that we are putting parameter and return types in a file that starts with `<?php`.

If we run the program like this:

```
hhvm force.php
```

we would get a fatal error because we are trying to use type hints in a non-Hack code file. However, with `hhvm.force_hh`, we can override that fatal.

```
hhvm -d hhvm.force_hh=true force.php
```

and get the expected output of `5`.

repository: watashi/user-documentation

gha_license_id: NOASSERTION

repo_name: watashi/user-documentation


text:
# Core API Tutorial
This tutorial shows how to take an existing training script and integrate it
with the Determined platform in a series of steps.  To keep the focus of the
tutorial on the Core API mechanics, this tutorial is based on a script that
increments an integer, rather than training any real model.

## Files
* **0_start.py**: A starting script with no Determined integrations at all.
* **0_start.yaml**: A barebones config to run the `0_start.py` script on
  a Determined cluster.
* **1_metrics.py**: Modify `0_start.py` to report metrics to Determined.
* **1_metrics.yaml**: A config to run `1_metrics.py`
* **2_checkpoints.py**: Modify `1_metrics.py` to save and load to and from
  checkpoint storage.  This also introduces preemption support.
* **2_checkpoints.yaml**: A config to run `2_checkpoints.py`.
* **3_hpsearch.py**: Modify `2_checkpoints.py` to participate in the
  hyperparameter search offered by the Determined master.
* **3_hpsearch.yaml**: Use an adaptive ASHA search to optimize a hyperparameter.
* **4_distributed.py**: Introduce distributed training to our script.
* **4_distributed.yaml**: Run `4_distributed.py` across many nodes.

## To Run
If you have not yet installed Determined, installation instructions can be found
under `docs/install-admin.html` or at https://docs.determined.ai/latest/index.html

Run stage 0 with the following command:

    det -m <master host:port> experiment create -f 0_start.yaml .

The other stages can be run by specifying the appropriate config file.

repository: determined-ai/determined

gha_language: Go

repo_name: determined-ai/determined


text:
# Alert Logic IAM Management

These scripts provide a programatic method to create, update, or delete your Alert Logic IAM roles in order to deploy Alert Logic products into your AWS account. In order to use these scripts, you must have the requirements listed below installed. You must also setup your aws credentials and config files as described [here](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html).

## IAM Policy Management Utility

**Requirements**

Python >= 2.6

boto3

requests

argparse

### Usage:

Help:
```
python manage_policy.py --help
usage: manage_policy.py [-h] --policy-name POLICY_NAME [--account ACCOUNT]
                            [--create] [--update]
                            [--delete-version DELETE_VERSION]
                            [--list-versions] [--minimal]

Alert Logic IAM Policy Management

optional arguments:
  -h, --help            show this help message and exit
  --policy-name POLICY_NAME IAM policy name to create/update
  --account ACCOUNT     AWS account number
  --create              Create IAM policy
  --update              Update existing IAM policy
  --delete-version DELETE_VERSION IAM policy version to delete
  --list-versions       List existing IAM policy versions
  --minimal             Apply minimal IAM policy
```

Create minimal permission policy:
```
python manage_policy.py --policy-name alertlogic-policy --create --minimal
Minimal permission option Allows you to maintain full control over the changes in your deployment, and requires you to perform any necessary actions manually.
Continue applying minimal permissions policy? (y/n): y
Are you sure you want to create policy alertlogic-policy? (y/n): y
Successfully created IAM Policy: arn:aws:iam::1234567890:policy/alertlogic-policy
```

Create full permission policy:
```
python manage_policy.py --policy-name alertlogic-policy --create
Full permission option allows Alert Logic to make all the necessary changes to your AWS account.
Continue applying full permissions policy? (y/n): y
Are you sure you want to create policy alertlogic-policy? (y/n): y
Successfully created IAM Policy: arn:aws:iam::1234567890:policy/alertlogic-policy
```

Update existing policy to minimal permission:
```
python manage_policy.py --policy-name alertlogic-policy --update --account<PHONE_NUMBER> --minimal
Are you sure you want to update IAM Policy: alertlogic-policy? (y/n): y
Successfully updated IAM Policy: arn:aws:iam::1234567890:policy/alertlogic-policy
```

Update existing policy to full permission:
```
python manage_policy.py --policy-name alertlogic-policy --update --account<PHONE_NUMBER>
Are you sure you want to update IAM Policy: alertlogic-policy? (y/n): y
Successfully updated IAM Policy: arn:aws:iam::1234567890:policy/alertlogic-policy
```

List policy versions:
```
python manage_iam.py --policy-name alertlogic-policy --account<PHONE_NUMBER> --list-versions
Existing policy versions: ['v8', 'v7', 'v6', 'v5', 'v4']
Default policy version: v8
```

Delete policy by version:
```
python manage_iam.py --policy-name alertlogic-policy --account<PHONE_NUMBER> --delete-version v4
Are you sure you want to delete policy alertlogic-policy version v4? (y/n): y
Successfully deleted IAM Policy version: v4
```

## IAM Role Management Utility

### Usage:

Help:
```
python manage_role.py --help
usage: manage_role.py [-h] --role-name ROLE_NAME
                          [--access-type {log,us,uk}]
                          [--external-id EXTERNAL_ID]
                          [--policy-arn POLICY_ARN] [--create] [--delete]
                          [--detach-policy] [--attach-policy]

Alert Logic IAM Role Management

optional arguments:
  -h, --help            show this help message and exit
  --role-name ROLE_NAME IAM role name to create
  --access-type {log,us,uk} Options: ['log'(CloudTrail and S3 log collection only), 'us', 'uk']
  --external-id EXTERNAL_ID Alert Logic Customer ID. To find your Customer ID, in
                        the Alert Logic console, click Settings > Support Information.
  --policy-arn POLICY_ARN Alert Logic IAM Policy previously created.
  --create              Create IAM Role
  --delete              Delete IAM Role
  --detach-policy       Detach IAM Policy from role
  --attach-policy       Attach IAM Policy from role
```

Create role and attach existing policy:
```
python manage_role.py --role-name alertlogic-role  --external-id<PHONE_NUMBER> --policy-arn arn:aws:iam::123456789012345:policy/alertlogic-policy --create --access-type us
Alert Logic Customer ID(External ID):<PHONE_NUMBER>
IAM Role to be created: alertlogic-role
IAM Policy Arn to be attached: arn:aws:iam::123456789012345:policy/alertlogic-policy
Alert Logic account for cross-account access: us:<PHONE_NUMBER>12345
Create new IAM Role alertlogic-role and apply policy: arn:aws:iam::123456789012345:policy/alertlogic-policy? (y/n): y
Successfully created IAM Role: alertlogic-role
```

Delete role:
```
python manage_role.py --role-name alertlogic-role --delete
Are you sure you want to delete role alertlogic-role? (y/n): y
Successfully deleted IAM Role: alertlogic-role
```

Attach policy to existing role:
```
python manage_role.py --role-name alertlogic-role --policy-arn arn:aws:iam::1234567890:policy/alertlogic-policy --attach-policy
Are you sure you want to attach policy arn:aws:iam::1234567890:policy/alertlogic-policy to IAM Role: alertlogic-role? (y/n): y
Successfully attached arn:aws:iam::1234567890:policy/alertlogic-policy to IAM Role: alertlogic-role
```

Detach policy to existing role:
```
python manage_role.py --role-name alertlogic-role --policy-arn arn:aws:iam::1234567890:policy/alertlogic-policy --detach-policy
Are you sure you want to detach policy arn:aws:iam::1234567890:policy/alertlogic-policy from IAM Role: alertlogic-role? (y/n): y
Successfully detached arn:aws:iam::1234567890:policy/alertlogic-policy from IAM Role: alertlogic-role
```

Contributing
------------

1. Fork the repository on Github
2. Create a named feature branch (like `add_component_x`)
3. Write your change
4. Submit a Pull Request using Github

License and Authors
-------------------
License:
Distributed under the MIT license.

Authors:
Justin Early<EMAIL_ADDRESS>

repository: alertlogic/alertlogic-iam-role-update

repo_name: alertlogic/alertlogic-iam-role-update


text:
## Authentication with Facebook App Native Login Dialog

### How to run the sample?

1. Download [Facebook SDK for iOS](https://developers.facebook.com/docs/ios/)
2. Copy `FacebookSDK.framework` folder to the root of the sample.
3. Run.

#### Use your own API keys
Sample should work out of the box because it is using DEMO API keys. If you want to use your own api keys, perform the following steps:

* Update `Auth0Client.AppNativeLoginSample-Info.plist` file with your Facebook App settings:
	* __FacebookAppID:__ {YOUR_FACEBOOK_CLIENT_ID}
	* __FacebookDisplayName:__ {Display Name you configured in the Facebook Dashboard}
	* __URL types -> Item 0 -> URL Schemes -> Item 0:__ fb{YOUR_FACEBOOK_CLIENT_ID}

* Update `LoginViewController.m` file with your Auth0 settings:
```Objective-c
NSString * const auth0_domain = @"{YOUR_AUTH0_DOMAIN}"; // e.g. contoso.auth0.com
NSString * const auth0_clientId = @"{YOUR_AUTH0_CLIENT_ID}";
```

gha_language: Objective-C


text:
#My dotfiles and preferences

When booting a new working station, we should set it up quickly. Dotfiles are great, but i've also listed my sublimetext preferences and packages etc...

## Step by step

- Install [those fonts](https://www.dropbox.com/s/sc3qqjm37bugjaa/fonts.zip?dl=0)
- Intall [iTerm](http://iterm2.com/)
- Then, [Sublime Text](http://www.sublimetext.com/3)
- Open iTerm and `cd`
- Symlink _subl_ : `ln -s "/Applications/Sublime Text.app/Contents/SharedSupport/bin/subl" ~/bin/subl`
- Install [homebrew](http://brew.sh/)
- Install homebrew formulae : `./brew.sh`
- Execute the .osx in your terminal : `./.osx`
- Reboot
- Install [ohmyzsh](http://ohmyz.sh/) : `curl -L http://install.ohmyz.sh | sh`
- If needed, set the default shell to zsh : `chsh -s /bin/zsh`(http://cl.ly/image/2H2d3M1m2b35))
- Install [zsh-pure](https://github.com/sindresorhus/pure) (rtfm dude)
- Copy `.gitconfig`, `.gitignore` & `.zshrc` in your `~/`

## OS X Apps I couldn't live without

- [Spectacle](http://spectacleapp.com/)
- [1Password](https://agilebits.com/onepassword)
- [Sip](http://theolabrothers.com/sip/) / [Frank deLoupe](http://jumpzero.com/frank/)
- [Euclid](http://euclidapp.com/)
- [ImageOptim](https://imageoptim.com/fr.html)
- [Mou](http://25.io/mou/)
- [Name Changer](http://mrrsoftware.com/namechanger/)
- [Sketch](http://bohemiancoding.com/sketch/)
- [iTerm](http://iterm2.com/)
- [Sublime Text](http://www.sublimetext.com/3)
- [VirtualBox](https://www.virtualbox.org/wiki/Downloads)
- My mouse's [drivers](http://steelseries.com/support/downloads)
- [VLC](https://www.videolan.org/vlc/)
- [Wunderlist](https://www.wunderlist.com/fr/)
- [Smooth mouse](http://smoothmouse.com/)

## Sublime text packages

After installing sublime text (3), you'll need to add it the [package control extension](https://sublime.wbond.net/installation), then install those great packages :

- [Theme Soda](https://github.com/buymeasoda/soda-theme/)
- [Alignment](http://wbond.net/sublime_packages/alignment)
- [PHP-Twig](https://github.com/Anomareh/PHP-Twig.tmbundle)
- [AutoFileName](https://github.com/BoundInCode/AutoFileName)
- [Sass](https://github.com/nathos/sass-textmate-bundle)
- [SCSS](https://github.com/MarioRicalde/SCSS.tmbundle)
- [Sass Snippets](https://github.com/sublimebrasil/sublime-snippets-sass)
- [LESS](https://sublime.wbond.net/packages/LESS)
- [Autoprefixer](https://github.com/sindresorhus/sublime-autoprefixer)
- [Bracket Highlighter](https://github.com/facelessuser/BracketHighlighter)
- [Color Highlighter](https://sublime.wbond.net/packages/Color%20Highlighter)
- [CSS3](https://github.com/y0ssar1an/CSS3)
- [CSS3 Snippets](https://github.com/joshnh/CSS-Snippets)
- [DocBlockr](https://github.com/spadgos/sublime-jsdocs)
- [SidebarGit](https://github.com/SublimeText/SideBarGit)
- [Emmet](https://github.com/sergeche/emmet-sublime)
- [Emmet CSS Snippets](https://p233.github.io/Emmet-Css-Snippets-for-Sublime-Text-2/)
- [Github Color Theme](https://github.com/AlexanderEkdahl/github-sublime-theme)
- [HTML5](https://github.com/mrmartineau/HTML5)
- [HTMLAttributes](https://github.com/agibsonsw/HTMLAttributes)
- [jQuery](https://github.com/SublimeText/jQuery)
- [JavascriptNext](https://github.com/Benvie/JavaScriptNext.tmLanguage)
- [JSHint](https://github.com/uipoet/sublime-jshint)
- [Markdown Editing](https://github.com/SublimeText-Markdown/MarkdownEditing)
- [Sublime linter](http://www.sublimelinter.com/en/latest/)
- [Trailing spaces](https://github.com/SublimeText/TrailingSpaces)

For the _Markdown Editing_ package, you'll need to edit the user settings for MultiMarkdown, GFM Markdown and Standard Markdown, adding those lines to the corresponfing files :

```
{
    "color_scheme": "Packages/MarkdownEditing/MarkdownEditor-Dark.tmTheme"
}
```

repository: eomusoru/dotfiles

repo_name: eomusoru/dotfiles


text:
![Forking](https://i.stack.imgur.com/yPKXU.png)
## What is a fork in Github?

A fork is a copy of a repository. Forking a repository allows you to freely experiment with changes without affecting the original project.
## When is forking used? (example)
* Proposing new changes.

* Using someone else's project as the start point.

## Sources:
* https://docs.github.com/en/github/getting-started-with-github/fork-a-repo


text:
comcast.pypi
============

A role for provisioning a minimal pypi-server

Role Variables
--------------

* `pypi_server_version`: 'The version of the pypiserver wheel to install and
  use. Default: `1.2.0`.'
* `pypi_user`: 'The user running the pypi-server service. It will be created if
  it doesn't exist. Default: `pypi`.'
* `pypi_group`: 'The group of which pypi_user is a member. Default: `pypi`.'
* `pypi_server_port`: 'The port in which the pypi-server will listen on.
  Default: `7974`.'
* `pypi_home_dir`: 'The home directory of the pypi user. Default: `/srv/pypi`.'
* `pypi_work_dir`: 'The workspace directory of the pypi-server. Default: `{{
  pypi_home_dir }}`.'
* `pypi_packages_directory`: 'The directory that will store uploaded python
  packages. Default: `{{ pypi_work_dir }}/packages`.'
* `pypi_log_file`: 'The log file the pypi-server will write to. Defautlt: `{{
  pypi_work_dir }}/pypi-server.log`.'
* `pypi_init_script_dir`: 'The directory that will store the pypi-server init
  script (sys-v only). Default: `/etc/init.d`.'
* `pypi_server_pid_file`: 'The location of the . Default: `{{ pypi_work_dir
  }}/pypi.pid`.'
* `pypi_requirements_template`: 'Template listing the required packages for
  the pypi-server. Default: `templates/pypi-server-requirements.txt.j2`.'
* `pypi_server_authenticate`: Comma-separated list of actions to authenticate
  a client, ex.: `download,list,update`. Default: `update`.
* `htaccess_dir`: 'The location of the generated `.htaccess` file. Default: `{{
  pypi_home_dir }}`.'
* `enable_anonymous_auth`: 'A boolean value that determines whether to use
  username/password authentication for uploads to the pypi-server. Note:
  Enabling this is not recommended as it will allow anyone to upload artifacts
  to pypi. Default: `false`.'
* `htaccess_username`: 'Username to use when authenticating to the pypi-server
  (only used when `enable_anonymous_auth is false`). Default: `test`.'
* `htaccess_password`: 'Password to use when authenticating to the pypi-server
  (only used when `enable_anonymous_auth is false`). Default: `test`.'

Dependencies
------------

None

Example Playbook
----------------

Including an example of how to use your role (for instance, with variables
passed in as parameters) is always nice for users too:

```yaml
    - hosts: servers
      roles:
         - role: comcast.pypi
```

Testing
-------

The following tools are required for testing this role.

1. [Virtualbox](https://www.virtualbox.org/wiki/Downloads)
1. [Vagrant](https://www.vagrantup.com/downloads.html)
1. [Python 2.7](https://www.python.org/downloads/release/python-2712/)
1. [virtualenv](https://pypi.org/project/virtualenv/)

Create test environment:

```bash
virtualenv --python=$(which python2.7) .venv
source .venv/bin/activate
pip install -r test-reqs.txt
```

Run tests:

```bash
molecule test
```

Test against a specific platform:

```bash
molecule test --platform centos/7
molecule test --platform debian/jessie64
```

Test uploads:

Any python package can be used to test uploading. In this case BeautifulSoup4
is used as an example.

1. Stand up local pypi server

```bash
molecule converge
```
This should leave the pypi-server running on `http://localhost:8080` with 0
packages stored.

2. Clone a Python source repo

```bash
git clone https://github.com/getanewsletter/BeautifulSoup4.git
```

3. Create/modify the `.pypirc` file in your home directory and add the
`htaccess_username` and `htaccess_password` credentials if using authentication.

The `.pypirc` file should look similar to this:

> Note: If using the playbook defaults for the htaccess_username and
htaccess_password then the credentials should match what is in the playbook.yml

```
[distutils]
index-servers =
  local

[local]
repository: http://localhost:8080
username: <htaccess_username>
password: <htaccess_password>
```
4. Upload python package

From the Python source code repo execute the following command.

```bash
python setup.py sdist upload -r local
```

This should upload the python package to the pypi-server specified in the
`.pypirc` file using the credentials specified in the `[local]` block.

Test download:

To test package downloads from the local pypi-server the command below can be
executed after successfully uploading a package. Following the upload example
beautifulsoup4 can be downloaded as follows:

> Change version_number for the example below to match the version of the
package that was uploaded.

```bash
pip install --extra-index-url http://localhost:8080/ beautifulsoup4==<version_number>
```

License
-------

[Apache 2.0](LICENSE)

Author Information
------------------

* [Luis Ortiz][lortiz]
* [Elliot Weiser][elliotweiser]
* [Comcast][comcast]

[comcast]: https://github.com/Comcast
[elliotweiser]: https://github.com/elliotweiser
[lortiz]: https://github.com/skippyPeanutButter

repository: DalavanCloud/ansible-role-pypi

repo_name: DalavanCloud/ansible-role-pypi


text:
#### Integrations

##### WithSecure Event Collector

- Fixed a library incompatibility by downgrading the Docker image to: *demisto/python3:<IP_ADDRESS>956*.

repository: demisto/content

repo_name: demisto/content


text:
## Change log
----------------------

Version 1.1
-------------

ADDED:

- added usage area to the README.md file

CHANGED:

- update of gradle to new version 6.7
- update of test dependency file-worker to new version 5.7
- javadoc extended and optimized

Version 1
-------------

ADDED:

- initial version of project
- new CHANGELOG.md file created
- gradle as build system

Notable links:
[keep a changelog](http://keepachangelog.com/en/1.0.0/) Don’t let your friends dump git logs into changelogs

repository: jimmyMaci/sign-and-verify

repo_name: jimmyMaci/sign-and-verify


text:
# Professional README Generator

  [license badge](https://img.shields.io/badge/license-MIT-brightgreen)

  ![Language Badge](https://img.shields.io/static/v1?label=language&message=Javascript&color=brightgreen)

  # Description
  It generates a professional read me based off of your inputs

  # Table of Contents
  - [Installation](#installation)
  - [Credit](#credit)
  - [Usage](#usage)
  - [Contributing](#contributing)
  - [license](#license)
  - [Badges](#badges)
  - [Features](#features)
  - [Tests](#tests)
  - [Questions](#questions)

  # Installation
  Have fun

  # Usage
  Have a whole lot of fun

  # Credits
  Xandromus

  # license
  [License badge](https://choosealicense.com/licenses/mit/)

  # Badges
  [Language Badge](https://img.shields.io/static/v1?label=language&message=undefined&color=brightgreen)

  # Features
  More fun stuff

  # Contributing
  DO NOT ABUSE THE CODE... EVER

  # Tests
  A grip of tests

  # Questions
  Github: https://github.com/TannerCarter <br>
  For any questions or concerns, please reach out to me via email<EMAIL_ADDRESS>


text:
# TODO

### Master-detail relationships

This type of relationship has one row in the master table and zero or more detail rows in one or more other table(s). A
foreign key that points to the master table can be annotated in its description with `@Detail` annotation.
For this type of hierarchy CRUD code for detail classes will not be generated. Persistence operations will be provided
by the repository for the master class.

We are using a somewhat unusual approach with JSON as encoding for detail persistence and fetching. That way we can
use a single round-trip for all data in this master-detail hierarchy.

**Example:**

An `Invoice` has one or more `InvoiceItem`s. Every `InvoiceItem` can have one or more `ItemTax` entries.
Both `InvoiceItem` and `ItemTax` have `@Detail` annotation on foreign keys to parent table.

Generated query for fetching a hierarchy in a `invoice -> item -> item_tax` relationship via JSON:

```sql
  SELECT row_to_json(master)
  FROM (
           SELECT invoice.*,
                  CASE WHEN count(items_detail) = 0
                       THEN ARRAY []::JSON[]
                       ELSE array_agg(items_detail.item)
                  END AS items,
                  CASE WHEN count(invoice_compensation_detail) = 0
                       THEN ARRAY []::JSON[]
                       ELSE array_agg(invoice_compensation_detail.compensation)
                  END AS compensations,
                  CASE WHEN count(invoice_tax_detail) = 0
                       THEN ARRAY []::JSON[]
                       ELSE array_agg(invoice_tax_detail.tax)
                  END AS invoice_taxes
           FROM invoice
                LEFT OUTER JOIN (
                    SELECT items_detail_inner.invoice_id, row_to_json(items_detail_inner) item
                     FROM (SELECT invoice_item.*,
                                  CASE WHEN count(item_tax_detail) = 0
                                       THEN ARRAY []::JSON[]
                                       ELSE array_agg(item_tax_detail.tax)
                                  END AS taxes
                         FROM invoice_item
                              LEFT OUTER JOIN (
                                  SELECT item_tax.item_id, row_to_json(item_tax) tax
                                    FROM item_tax
                              ) item_tax_detail ON invoice_item.id = item_tax_detail.item_id
                         GROUP BY invoice_item.id
                        ) items_detail_inner
                ) items_detail ON items_detail.invoice_id = invoice.id
                LEFT OUTER JOIN (
                    SELECT invoice_compensation.invoice_id, row_to_json(invoice_compensation) compensation
                     FROM invoice_compensation
                ) invoice_compensation_detail ON invoice.id = invoice_compensation_detail.invoice_id
                LEFT OUTER JOIN (
                    SELECT invoice_tax.invoice_id, row_to_json(invoice_tax) tax
                      FROM invoice_tax
                ) invoice_tax_detail ON invoice_tax_detail.invoice_id = invoice.id
           GROUP BY invoice.id
       ) master
  WHERE master.id = 37;
```

Persisting such entities is done in layers: first the top-level entities are persisted, then one level below, etc. This
is so that we get autogenerated keys that the lower-level entities need to refer to.

### Enums

Do Postgres enums have any value compared to lookup tables? This makes it a question of having enum values present at
compile time or loaded from the database.

### Lookup tables

Every application will have one or more lookup tables (not for enums but enums could fall under this category). Data
that doesn't change often but is needed often. These tables can be annotated with the `@Lookup` annotation. Apart from
the CRUD operations on these tables we can then generate code that loads data for all these tables at once. This can be
invoked on every transaction and can be improved with versioning so that the data is loaded on app start and then served
from cache unless some of it is updated.

### Events

Events are just normal tables that have a table annotation `@Event`. Events represent an immutable log of changes that
were already done so the generated repository will not have any update or delete methods - only create and query methods.

We will not impose any structure on the event tables but some incrementing id and / or timestamp would be useful to have
there.

### Entity versioning

You can track entity versions by adding a `@Versioning` annotation on the column that will be used to
keep the entity version. Supported column type is `timestamp(tz)`. You can set the value for this column
any way you like, the generated repository will always ignore the set value and use `now()` for all `INSERT` and
`UPDATE` operations.

### Optimistic concurrency (Version checking)
On a table that has a `@Versioning` column you can add a `@VersionCheck` annotation on the table-level comment. The
generated repository will have code that checks on update if the latest version in the table is the one that was used to
produce that new entity version. The return value for this operation will contain `Either` the `VersionCheckFailed`
error or the success value.
 - **TODO**: work on this some more.

### Entity history

When a table has `@Versioning` and `@VersionCheck` enabled you can also add `@History` annotation on the table-level
comment. This will turn all `UPDATE` queries into `INSERT` and all queries will be generated so that they always select
the latest version of the entity. The table in question should have a composite primary key (identity+version).

repository: pgmirror/pgmirror

repo_name: pgmirror/pgmirror


text:
Events in Ground DB II
======================

Events in Ground DB is emitted via a regular event emitter and the Meteor Tracker.

### Event / State

Ground DB emits events and states, the event state `loaded` happens only
once - but since it's a state `db.once('loaded');` will call back immediately if the
event has already been emitted.

* "loaded" -> `{ count }`

### Reactive variables

To get info about the read / write state Ground DB currently uses Tracker to emit
reactive events for progress.

* `db.pendingWrites`
* `db.pendingReads`

Api for the read/write states
* `progress()` -> `{ index, total, percent }`
* `isDone()` -> `Boolean`