text:
# API Review 08/19/2021

## IOperation support for interpolated string handler conversions

**NeedsWork** | [#roslyn/54718](https://github.com/dotnet/roslyn/issues/54718#issuecomment-902250198)

## API Review

We've generally agreed that the interface should expose the Append calls as children directly, rather than trying to add the information to `IInterpolatedTextOperation` or `IInterpolatedContentOperation`. We don't want to expose the data on `IBinaryOperation` directly: we feel it's surprising and a violation of layering, and that the `IInterpolatedStringOperation` should be the top node. We have a few options we're considering:

* We collapse all the parts into a single `IInterpolatedStringOperation`. While this is the most faithful to the semantics of the language, we're concerned about the ergonomics of not having `IOperation` nodes that correspond to nested `BinaryExpressionSyntax`es or `InterpolatedStringExpressionSyntax`es, which this approach would necessitate.
* We have dedicated `IBinaryContentOperation` or similar nodes that represent the binary operations themselves, and are part of the `Parts` array on the root `IInterpolatedStringOperation` node. While this is more complex and more of a "syntactic" view, it will alleviate the concerns about missing explicit syntax nodes.

A smaller team will explore this space and update the design.

One other note is that we should have a flag on `IInterpolatedStringAppendOperation` that indicates whether the content is a literal or an interpolation placeholder.

repository: dotnet/apireviews

repo_name: dotnet/apireviews


text:
# PLShortVideoKit 1.3.0 to 1.4.0 API Differences

## General Headers

```
PLSAVAssetExportSession.h
```

- *Added* @property (assign, nonatomic) BOOL isExportMovieToPhotosAlbum;

```
PLSGifComposer.h
```

- *Added* @property (assign, nonatomic) CGFloat interval;
- *Added* - (void)cancelComposeGif;

```
PLShortVideoRecorder.h
```

- *Added* @property (readwrite, nonatomic) PLSVideoRecoderRateType recoderRate;

```
PLShortVideoTranscoder.h
```

- *Added* @property (assign, nonatomic) BOOL isExportMovieToPhotosAlbum;

```
PLSTypeDefines.h
```

- *Added* NS_ENUM(NSInteger, PLSVideoRecoderRateType)

```
Added Class PLSMovieComposer
```

repository: pili-engineering/PLShortVideoKit

branch_name: refs/heads/master

extension: md

gha_language: Objective-C

license_type: permissive

repo_name: pili-engineering/PLShortVideoKit


text:
# Developer Notes

Information which are useful for developers and maintainers of the AceTime
library.

## Zone DB Files

As explained in the README.md, the AceTime library comes with 2 versions of the
[TZ Data](https://www.iana.org/time-zones):

* `src/ace_time/zonedb/*`: files used by the `BasicZoneProcessor` class
* `src/ace_time/zonedbx/*`: files used by the `ExtendedZoneProcessor` class

There are 2 main files in these directories:

* `zone_infos.h', `zone_infos.cpp`
* `zone_policies.h`, `zone_policies.cpp`

The format and meaning of these files are probably best explained in the header
files:

* `src/ace_time/internal/ZoneInfo.h`
* `src/ace_time/internal/ZonePolicy.h`

Some of the tricky encoding schemes were created to preserve resolution, while
keeping the binary size of the data structures as small as possible. For
example, these files are able to support DST transitions with a 1-minute
resolution, but the DST offsets have only a 15-minute resolution. Fortunately,
these limitations are sufficient to represent all time zones since about 1972.

Other bits of information are added by the code generator as comments in the
generated files and some of these are explained below.

### `zone_infos.h`

This file contains one entry of the `basic::ZoneInfo` or
`extended::ZoneInfo` data structure for each supported time zone.

The zone identifier is named `kZone{region}_{city}`, where the
`{region}_{city}` comes directly from the TZ Database. Some minor character
transformations are applied to create an valid C++ identifier. For example, all
dashes `-` are converted to underscore `_`.

The `kTzDatabaseVersion` string constant identifies the version of the TZ
Database that was used to generate these files, e.g. "2019a".

The `kZoneContext` variable points to an instance of `common::ZoneContext`
which identifies the `startYear` and `endYear` of the current database. These
will normally be 2000 and 2050 respectively, but a custom version of the
zonedb files could be generated whose startYear and endYear could be smaller (to
produce smaller ZoneDB files).

Near end of the `zone_info.h` file, we list the zones which were
deliberately excluded by the tool.

Also at the end of the `zone_info.h` file, there may be warnings about
known inaccuracies for a particular zone.

### `zone_infos.cpp`

Each zone entry in the `zone_info.cpp` contains a comment section that
describes some metadata about the given entry. For example, the entry for
`zonedb::kZoneEraAmerica_Los_Angeles` contains the following, showing how
much memory it will consume on the 8-bit Arduino controllers and the 32-bit
Arduino controllers:

* `// Zone name: America/Los_Angeles`
* `// Era count: 1`
* `// Strings: 24`
* `// Memory (8-bit): 43`
* `// Memory (32-bit): 53`

### `zone_policies.h`

An entry in `zone_info.cpp` may refer to a zone policy defined in
`zone_policies.h`. For example, the `kZoneAmerica_Los_Angeles` has a pointer
to a `kPolicyUS` data structure which is defined in `zone_policies.h`.

Each policy entry starts with a comment secion that contains some metadata
about the policy. For example:
```
// Policy name: US
// Rule count: 5
// Memory (8-bit): 51
// Memory (32-bit): 55
```
Just like `zone_infos.cpp`, the Memory section describes the amount of static
RAM consumed by the particular `ZonePolicy` data structure (and associated
`ZoneRule`.

## Upgrading ZoneInfo Files to a New TZDB Version

About 2-4 times a year, a new TZDB version is released. Here are some notes
(mostly for myself) on how to create a new release after a new TZDB version is
available.

* Update the TZDB repo (https://github.com/eggert/tz). This should be a
  sibling to the `AceTime` repo (since it will cause the least problems
  for various internal scripts):
    * `$ cd ../tz`
    * `$ git pull`
    * Check that the correct tag is pulled (e.g. `2020c` tag if that's the
      version that we want to upgrade to).
* Update the Hinnant date repo (https://github.com:HowardHinnant/date). This
  should be a sibling to the `AceTime` repo:
    * `$ cd ../date`
    * `$ git pull`
* Verify that `AceTime` and Hinnant `date` agree with each other using the
  same TZDB version.
    * BasicHinnantDateTest
        * `$ cd tests/validation/BasicHinnantDateTest`
        * Update the `TZ_VERSION` variable in the `Makefile` with the commit tag
          of the new TZDB version. (e.g. `TZ_VERSION = 2020c`).
        * `$ make clean`
        * `$ make`
        * `$ ./BasicHinnantDateTest.out | grep failed`
        * There should be no failures: `TestRunner summary: 268 passed, 0
          failed, 0 skipped, 0 timed out, out of 268 test(s).`
    * ExtendedHinnantDateTest
        * `$ cd tests/validation/ExtendedHinnantDateTest`
        * Update the `TZ_VERSION` variable in the `Makefile` with the commit tag
          of the new TZDB version. (e.g. `TZ_VERSION = 2020c`).
        * `$ make clean`
        * `$ make`
        * `$ ./ExtendedHinnantDateTest.out | grep failed`
        * There should be no failures: `TestRunner summary: 387 passed, 0
          failed, 0 skipped, 0 timed out, out of 387 test(s).`
* Update the various zone info files:
    * `src/ace_time/zonedb`
        * `$ cd src/ace_time/zonedb`
        * Edit the `Makefile` and update the `TZ_VERSION`.
        * `$ make`
    * `src/ace_time/zonedbx`
        * `$ cd src/ace_time/zonedbx`
        * Edit the `Makefile` and update the `TZ_VERSION`.
        * `$ make`
    * `tools/zonedbpy`
        * `$ cd tools/zonedbpy`
        * Edit the Makefile and update the `TZ_VERSION`.
        * `$ make`
* Update the CHANGELOG.md.
* Commit the changes to git
    * `$ git add ...`
    * `$ git commit -m "..."`

There are 6 other validation tests in `tests/validation` that compare AceTime
with other 3rd party librarties (Python pytz, Python dateutil, and Java date).
Unfortunately, they all seem to use the underlying TZDB version provided by the
Operating System, and I have not been able to figure out how to manually update
this dependency manually. When a new TZDB is released, all of these other tests
will fail until the underying timezone database of the OS is updated.

## Release Process

* Update and commit the version numbers in various files:
    * `src/AceTime.h`
    * `README.md`
    * `USER_GUIDE.md`
    * `docs/doxygen.cfg`
    * `library.properties`
    * `CHANGELOG.md`
    * Bump the version numbers in the unit tests:
        * `BasicZoneProcessorTest/BasicZoneProcessorTest.ino`
        * `ExtendedZoneProcessorTest/ExtendedZoneProcessorTest.ino`
    * `$ git commit -m "..."`
* Update and commit the Doxygen docs. This is done as a separate git commit
  because the Doxygen changes are often so large that they obscure all other
  important changes to the code base:
    * `$ cd docs`
    * `$ make clean`
    * `$ make`
    * `$ git add .`
    * `$ git commit -m "..."`
* Merge the `develop` branch into `master`.
    * Normally I do this with a PR on GitHub to keep an audit trail.
    * Go to https://github.com/bxparks/AceTime.
    * Create a Pull Request from `develop` to `master`.
    * Approve and merge the PR.
* Create a new Release.
    * Go to https://github.com/bxparks/AceTime
    * Click on "Releases"
    * Click on "Draft a new release"
    * Enter a tag version (e.g. `v1.2`), targeting the `master` branch.
    * Enter the release title.
    * Enter the release notes. I normally just copy and paste the latest changes
      from `CHANGELOG.md`.
    * Click Publish release.

license: MIT

repository: joesan/AceTime

repo_name: joesan/AceTime


text:
# twit-bot
A twitter bot that uses the twit package to perform a variety of tasks.

## What Can The Bot Do?
While the bot can quickly be modified to run different tasks, the bot is currently setup to find all twitter users who have the name 'Jimmy' (or are somehow associated with the name) and follow them at randomly spaced intervals.

The bot is also capable of automatically re-following other users and automatically liking tweets in which the bot was mentioned. In addition, the bot can tweet a random message at each of the users it follows.

## Installing
Because this bot is set up as an NPM package, you can simply run `npm install` in the repository path to install all of the dependencies.

## Configurations
You will need to fill out the config.js file in the lib directory, which looks like this:

```javascript
module.exports = {
    consumer_key:         '',
    consumer_secret:      '',
    access_token:         '',
    access_token_secret:  '',
    timeout_ms:           60*1000,  // optional HTTP request timeout to apply to all requests.
}
```
To get those keys and tokens, you need to register a Twitter Application on the Twitter website.

## Running The bot
The bot can be run by simply running 'npm start' in the repository path.

## Example Account
This particular bot has been operating the Twitter account named JimmyBigBoots.

repository: rknightly/twit-bot

repo_name: rknightly/twit-bot


text:
# Medium Editor for Rails

[![Gem Version](https://badge.fury.io/rb/medium-editor-rails.png)](http://badge.fury.io/rb/medium-editor-rails)
[![Code Climate](https://codeclimate.com/github/marjinal1st/medium-editor-rails.png)](https://codeclimate.com/github/marjinal1st/medium-editor-rails)
[![Dependency Status](https://gemnasium.com/marjinal1st/medium-editor-rails.svg)](https://gemnasium.com/marjinal1st/medium-editor-rails)

This gem integrates [Medium Editor](https://github.com/yabwe/medium-editor) with Rails asset pipeline.

## Version

The latest version of Medium Editor bundled by this gem is [5.22.0](https://github.com/yabwe/medium-editor/releases)

## Installation

Include **medium-editor-rails** in your Rails project's Gemfile:

```ruby
gem 'medium-editor-rails'
```

And then execute:

```bash
bundle install
```

## Configuration

Include javascript file in **app/assets/javascripts/application.js**:

```javascript
//= require medium-editor
```

Include stylesheet file in **app/assets/stylesheets/application.css**:

```scss
*= require medium-editor/medium-editor
```

You can also include themes from **themes** folder, example:

```scss
*= require medium-editor/themes/flat
```

For more themes, visit: [Medium Editor Themes](https://github.com/yabwe/medium-editor/wiki/Themes)

## Using Medium Editor with Rails

You need to initialize Medium Editor with any selector of div, example:

```html
<div class="editable"></div>

<script>
  var editor = new MediumEditor('.editor', {
    // options go here
  });
</script>
```

For extra options, visit: [Medium Editor Options](https://github.com/yabwe/medium-editor#mediumeditor-options)

## Contributing

1. Fork it ( http://github.com/marjinal1st/medium-editor-rails/fork )
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Add some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
5. Create new Pull Request

repository: rubentrf/medium-editor-rails

branch_name: refs/heads/develop

gha_language: Ruby

repo_name: rubentrf/medium-editor-rails


text:
### interpretability

Go back to the [[AI Glossary]]

The degree to which a model's predictions can be readily explained. Deep models are often non-interpretable; that is, a deep model's different layers can be hard to decipher. By contrast, linear regression models and wide models are typically far more interpretable.

url: https://raw.githubusercontent.com/KGBicheno/KGB_Agora/70ea95c9d38ab2f31abbd510691bb4c050a6cb3b/Artificial Intelligence/Introduction to AI/Week 3 - Introduction/Definitions/Interpretability.md

branch_name: refs/heads/main


text:
# dslink-javascript-homekit

A DSLink for exposing a HomeKit accessory to the local network, ultimately to
use Siri and other HomeKit apps with DSA.

Allows DSLink users to create HomeKit accessories, services, and characteristics
that mimic those available in the HAP HomeKit specification.

*In development. Link is feature complete but bugs may occur.*

### Documentation

Documentation for the DSLink can be found in **api.md**.

### Quick Start

###### Automatic installation:

- Bundle HomeKit with your Get DSA package.
- Download HomeKit from the DSLink Manager (DGLux).
- Install it in the DSLink Manager from git.

###### Manual installation:

```sh
git clone https://github.com/IOT-DSA/dslink-javascript-homekit.git
cd dslink-javascript-homekit
npm install
node dist/index.js --broker https://www.broker.com/conn
```

We've tested with third-party HomeKit apps in the past, but now recommend you
use the official Home app included within iOS 10.

Unless you change it from the link, the link will display in HomeKit as
*HomeKit + DSA*, and the pincode will be 465-46-465.

### License

The code in this repository is licensed under the Apache License (copy found
in **LICENSE.md**) unless stated otherwise.

repository: IOT-DSA/dslink-javascript-homekit

repo_name: IOT-DSA/dslink-javascript-homekit


text:
# Assignment 2: Markov Chains

In this assignment, we'll be looking at language models as a great example of Markov chains.
Please make sure to install both [`numpy`](https://numpy.org) and [`nltk`](http://www.nltk.org), since we'll be using those for the implementations.

## Trump vs. Obama

Surely, you're aware that the 45th President of the United States (@POTUS45) was an active user of Twitter, until (permanently) banned on Jan 8, 2021.
You can still enjoy his greatness at the [Trump Twitter Archive](https://www.thetrumparchive.com/), where you can download a JSON file linked under FAQs (or see Moodle materials).
We will be using original tweets only, so make sure to remove all retweets.

Another fan of Twitter was Barack Obama (@POTUS43 and @POTUS44), who used the platform in a rather professional way.
There are multiple ways to get the data, but the easiest is to download the [files from Kaggle](https://www.kaggle.com/jayrav13/obama-white-house) (or see Moodle materials).

Please also download the current POTUS (Joe Biden) tweets [from Kaggle](https://www.kaggle.com/rohanrao/joe-biden-tweets) (or see Moodle materials); we will be using those for testing.

Before you get started, please download the files; you can put them into the `res` folder.

In this assignment, you will be doing some Twitter-related preprocessing and training n-gram models to be able to distinguish between Tweets of Trump and Obama.
We will be using NLTK, more specifically it's [`lm`](https://www.nltk.org/api/nltk.lm.html) module.
For details, please go to [potus.py](src/potus.py).

## Theses Inspiration

Imagine you'd have to write another thesis, and you just can't find a good topic to work on.
Well, n-grams to the rescue!
Download the `theses.txt` data set from the Moodle page which consists of ca. 1,000 theses topics chosen by students in the past.

Pay extra attention to preprocessing: how would you handle hyphenated words and acronyms/abbreviations?

In this assignment, you will be sampling from n-grams to generate new potential thesis topics.
For more details, please go to [theses.py](src/theses.py).


text:
# winston-child

[![GitTip](http://img.shields.io/gittip/alexgorbatchev.svg)](https://www.gittip.com/alexgorbatchev/)
[![Dependency status](https://david-dm.org/alexgorbatchev/winston-child.svg)](https://david-dm.org/alexgorbatchev/winston-child)
[![devDependency Status](https://david-dm.org/alexgorbatchev/winston-child/dev-status.svg)](https://david-dm.org/alexgorbatchev/winston-child#info=devDependencies)
[![Build Status](https://secure.travis-ci.org/alexgorbatchev/winston-child.svg?branch=master)](https://travis-ci.org/alexgorbatchev/winston-child)

[![NPM](https://nodei.co/npm/winston-child.svg)](https://npmjs.org/package/winston-child)

Facilitates meta object inheritance for [Winston] loggers.

## Installation

    npm install winston-child

## Usage Example

    var winston = require('winston');
    var WinstonChild = require('winston-child').WinstonChild;

    var logger = new winston.Logger({
      transports: [
        new winston.transports.Console()
      ]
    });

    logger = new WinstonChild(logger, {foo: 'bar'});

    logger.info('hello!');
    // info: hello! foo=bar

    logger.info('context!', {local: 123});
    // info: context! local=123, foo=bar

    logger = logger.child({childValue: 'hello'});

    logger.info('hello!');
    // info: hello! foo=bar, childValue=hello

    logger.info('context!', {local: 123});
    // info: context! local=123, foo=bar, childValue=hello

## Testing

    npm test

## License

The MIT License (MIT)

Copyright 2014 Alex Gorbatchev

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

 IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

[Winston]: https://github.com/flatiron/winston

repository: alexgorbatchev/winston-child

gha_language: CoffeeScript

gha_license_id: MIT

repo_name: alexgorbatchev/winston-child


text:
# pacioli

Automatically tags EBS, ENI and other resources under an auto-scaling group's purview in an AWS region, since the ASG service decides not to propagate tags to any resources except EC2 instances.

repository: scottbrown/pacioli

repo_name: scottbrown/pacioli


text:
# Train-Scheduler

Schedule your next train and see how long you have to wait until it arrives!

* Powered by Javascript, jQuery, Moment.JS, and Firebase!

Further design and updates to come!

## Getting Started

Follow the deployed project link below to utilize the application.

### Deployed Project Link
<!-- make a link to the deployed site -->

[Train Scheduler](https://sajeelmalik.github.io/Train-Scheduler)

### Image Preview of Train Scheduler
<!-- take a picture of the image and add it into the readme  -->
![Train Scheduler Preview]( "Train Scheduler")

## Prerequisites

The page can be run from any browser, preferably on Google Chrome!

## Technology Used

* **HTML5**
* **CSS3**
* **Javascript** - the primary scripting logic powering the game
* **jQuery** - the robust scripting library for Javascript
* [**Firebase**](https://firebase.google.com/) - the online infrastructual database utilized
* [**Moment.js**](https://momentjs.com/docs) - a date-time Javascript library to enable ease of time manipulation
* [**Bootstrap CDN v4.1.0**](https://getbootstrap.com/docs/4.1/getting-started/introduction/) - the open-source web styling framework used
* [**Google Fonts**](https://fonts.google.com/) - an interactive library of licensed fonts

# Code Snippets
<!-- put snippets of code inside ``` ``` so it will look like code -->
<!-- if you want to put blockquotes use a > -->

Dynamic generation of HTML elements via jQuery is an extremely powerful tool to instantaneously adjust the webpage to the user's interaction with the interface. Here, jQuery interacts with Firebase to push new user entries onto the cloud.

```
  $("#submit").on("click", function (event) {
        event.preventDefault();

        var name = $("#name").val().trim();
        var destination = $("#destination").val().trim();
        var trainTime = $("#trainTime").val().trim();
        console.log(trainTime);
        var frequency = $("#frequency").val().trim();

        database.ref().push({
            name: name,
            destination: destination,
            trainTime: trainTime,
            frequency: frequency
        })
        $("#form").trigger("reset");
    });

```

# Learning points
<!-- Learning points where you would write what you thought was helpful -->
* Moment JS allows developers to interact more fluidly with a generally difficult-to-deal-with user input - time
* Firebase is extremely useful for long-term cloud storage of user data

## Authors

* **Sajeel Malik** - *Initial work* - [GitHub](https://github.com/sajeelmalik)

## License

This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details

repository: sajeelmalik/Train-Scheduler

repo_name: sajeelmalik/Train-Scheduler


text:
---
date: 2000-01-01T00:00:00+00:00
title: DRONE_COMMIT_BRANCH
author: bradrydzewski
---

Provides the target branch for the push or pull request. This value may be empty for tag events.

```
DRONE_COMMIT_BRANCH=master
```

repository: drone/docs

gha_language: SCSS

gha_license_id: NOASSERTION

repo_name: drone/docs


text:
# Weather forecast

* Output of the current weather
* Hourly forecast
* Forecast for 8 days
* Updating the weather every 30 minutes and changing the location

## Installation

`yarn install && yarn run build`

## Development

`yarn install && yarn run dev`

## License

Weather forecast is open-sourced software licensed under the [MIT license](http://opensource.org/licenses/MIT).

repository: Katochimoto/weather

repo_name: Katochimoto/weather


text:
# pod-monitoring

A React component for the Pod projects monitoring

## Installation

Use npm to install the package:

```
npm i pod-monitoring
```

## Usage

### Summary

```jsx
import MonitoringHealthCheck from 'pod-monitoring';

function MonitoringPage() {
  const yourApiAddress = 'https://your-project-api-address.ir/status';

  return (
    <MonitoringHealthCheck apiAddress={yourApiAddress}>
  );
}
```

### Step By Step Guide

First, create a page as a monitoring
```jsx
// ...
  <Route path="/monitoring" component={MonitoringPage} />
// ...
```

Second, in the MonitoringPage component, import MonitoringHealthCheck from pod-monitoring.
```jsx
import MonitoringHealthCheck from 'pod-monitoring';

function MonitoringPage() {
  const yourApiAddress = 'https://your-project-api-address.ir/status';

  return (
    <MonitoringHealthCheck apiAddress={yourApiAddress}>
  );
}
```

That's it. It's all the things that you should be done.

### Notice:
- Don't insert any content in the monitoring page.
- Use a public web service for monitoring.
- Use a public route in your front app.
- The maximum delay time for viewing content is 100 milliseconds.
- If your front app and backend service is up, you should see `frontend:successed,backend:successed`

## LICENSE

MIT

repository: talebkhah/pod-monitoring

repo_name: talebkhah/pod-monitoring


text:
![imgix logo](https://assets.imgix.net/imgix-logo-web-2014.pdf?page=2&fm=png&w=200&h=200)

# Luminous [![Build Status](https://travis-ci.org/imgix/luminous.svg?branch=master)](https://travis-ci.org/imgix/luminous) [![Slack Status](http://slack.imgix.com/badge.svg)](http://slack.imgix.com)

A simple, lightweight, no-dependencies JavaScript image lightbox.

* [Installation](#installation)
* [Usage](#usage)
* [Options / Defaults](#options-defaults)
* [Browser Support](#browser-support)
* [Theming](#theming)
* [Meta](#meta)

## Installation

* **NPM**: `npm install luminous-lightbox`
* **Bower**: `bower install luminous`
* **Manual**: [Download](https://github.com/imgix/luminous/archive/master.zip) and use `dist/luminous.min.js` or `dist/luminous.js`

## Usage

Once you've installed Luminous via one of the above methods, you're ready to get started. There are no dependencies, so you can just start making cool stuff. Check out the [announcement blog post](http://blog.imgix.com/2016/01/06/better-lightbox-zoom-viewer-with-imgix.html) for a demo, or take a peek here: https://imgix.github.io/luminous. Here's an example of the most basic possible implementation:

``` html
<a href="http://assets.imgix.net/dog.png?w=1600">
  <img alt="A dog!" src="http://assets.imgix.net/dog.png?w=400">
</a>
```

``` javascript
new Luminous(document.querySelector('a'));
```

## Options / Defaults

Here's an example of using Luminous with a custom configuration. All of the listed options are displayed with their default value.

``` javascript
var options = {
	// Prefix for generated element class names (e.g. `my-ns` will
	// result in classes such as `my-ns-lightbox`. Default `lum-`
	// prefixed classes will always be added as well.
	namespace: null,
	// Which attribute to pull the lightbox image source from.
	sourceAttribute: 'href',
	// Which attribute to pull the caption from, if any.
	captionAttribute: null,
	// The event to listen to on the _trigger_ element: triggers opening.
	openTrigger: 'click',
	// The event to listen to on the _lightbox_ element: triggers closing.
	closeTrigger: 'click',
	// Allow closing by pressing escape.
	closeWithEscape: true,
	// Automatically close when the page is scrolled.
	closeOnScroll: false,
	// A selector defining what to append the lightbox element to.
	appendToSelector: 'body',
	// If present (and a function), this will be called
	// whenever the lightbox is opened.
	onOpen: null,
	// If present (and a function), this will be called
	// whenever the lightbox is closed.
	onClose: null,
	// When true, adds the `imgix-fluid` class to the `img`
	// inside the lightbox. See https://github.com/imgix/imgix.js
	// for more information.
	includeImgixJSClass: false,
	// Add base styles to the page. See the "Theming"
	// section of README.md for more information.
	injectBaseStyles: true,
};

new Luminous(document.querySelector('a'), options);
```

## Theming

By default, Luminous injects an extremely basic set of styles into the page. You will almost certainly want to extend these basic styles for a prettier, more usable experience that matches your site. There is an included basic theme that may meet your needs, or at least give a good example of how to build out your own custom styles. The `namespace` option can be used as a way to easily apply different themes to specific instances of Luminous.

If you need to do something very out of the ordinary, or just prefer to include the default styles in CSS yourself, you can pass `injectBaseStyles: false` when instantiating a new instance of Luminous. Please note that if you disable the included base styles, you will still need to provide an animation for `.lum-lightbox.lum-opening` and `.lum-lightbox.lum-closing` (this can be a "noop" style animation, as seen in the base styles source).

## Browser Support

We support the latest version of Google Chrome (which [automatically updates](https://support.google.com/chrome/answer/95414) whenever it detects that a new version of the browser is available). We also support the current and previous major releases of desktop Firefox, Internet Explorer, and Safari on a rolling basis. Mobile support is tested on the most recent minor version of the current and previous major release for the default browser on iOS and Android (e.g., iOS 9.2 and 8.4). Each time a new version is released, we begin supporting that version and stop supporting the third most recent version.

## Meta

Luminous was made by [imgix](http://imgix.com). It's licensed under the BSD 2-Clause license (see the [license file](https://github.com/imgix/luminous/blob/master/license.md) for more info). Any contribution is absolutely welcome, but please review the [contribution guidelines](https://github.com/imgix/luminous/blob/master/contributing.md) before getting started.

repository: jackmcdade/luminous

repo_name: jackmcdade/luminous


text:
---
title: SIGINT 2013 crypto300 satisfaction
author: slipper
layout: post
permalink: /sigint-2013-crypto300-satisfaction/
categories:
  - Uncategorized
---
<http://slipper.tk/writeup/2013/07/11/sigint-2013-crypto-satisfaction/>

# [SIGINT 2013 crypto300 satisfaction][1]

Description

    Something fishy going on:
    you need this
    <IP_ADDRESS> 2000
    This flag does not have a SIGINT_ prefix!

[server.rb][2]

[bf.rb][3]

The server only runs properly RSA signed brainfuck code.

    def valid?(code, cert)
      return false unless code =~ /\A[\[\]+\-\. ]+\Z/
      return false unless cert =~ /\A\d+\Z/
      sig = cert.to_i
      crc = crc32(code)
      test = crc.to_bn.mod_exp($d, $n)
      return sig == test
    end

      if(valid_cert)
        puts "got valid code: #{code.inspect}"
        ruby_code = bfdo(code)
        puts "got some ruby code: #{ruby_code.inspect}"
        client.puts eval(ruby_code)

If the brainfuck code which we submit only contains `+-[].`, then the sever will calculate the crc of the code and check if it matches the signature.
Then how does `crc32()` work? The code in the script seems different from classic [CRC32][4].

    def crc32(str)
      mask = 0x04C11DB7
      crc = 0
      str.bytes.each do |byte|
        byte.to_s(2).rjust(8,"0").each_char do |bit|
          h_crc_bit = ( (( crc & 0x80_00_00_00) != 0) ? 1 : 0)
          h_byte_bit = bit.to_i
          if  h_crc_bit == h_byte_bit
            crc = ((crc << 1)^mask) & 0xffffffff
          else
            crc = (crc << 1) & 0xffffffff
          end
        end
      end
      return crc
    end

Standard CRC32 is the remainder of a polynomial division of the target string. In practice it employs the finite field GF(2). XOR is equivalent to division in GF(2).
As a hash function, CRC32 is easy to collision.
In the code above, XOR was taken when `h_crc_bit == h_byte_bit`.
However, in standard CRC32, it should be `h_crc_bit == 1 && h_byte_bit == 1`
Maybe the new crc32 is more likely to collision.

To study this algorithm, I rewrite it in C++.

    unsigned int CRC32_bit(unsigned char data[], int len)
    {
            unsigned int r = 0;
            unsigned int s;
            for (int i = 0; i < len; i++) {
                    s = data[i] << 24;
                    for (int j = 0; j < 8; j++) {
                            if (!((r ^ s) & 0x80000000))
                            //instead of if (r & 0x80000000)
                                    r = (r << 1) ^ POLY;
                            else
                                    r = r << 1;
                            s = s << 1;
                    }
            }
            return r;
    }

In this problem, it seems that we can't sign the code because we don't know the constants in `./rsa_keys.rb`.
The basic idea is very simple. If we keep the `crc=0` or `crc=1`, then signature is always the same with crc(0 or 1). Now we just need to find something excutable whose crc is 0 or 1.

First we should write a brainfuck program whose output is a legal ruby code, then append some useless character to the code to make its crc be 0 or 1.

A simple brainfuck programmer is like this

    def bf(s):
            p = 0
            q = 0
            K = 48
            ret = ''
            now = 0
            for c in s:
                if ord(c) < K:
                        if (now != 0):
                                ret += '<'
                                now = 0
                        while (p  ord(c)):
                                p -= 1
                                ret += '-'
                else:
                        if (now != 1):
                                ret += '>'
                                now = 1
                        while (q  ord(c)):
                                q -= 1
                                ret += '-'
                ret += '.'
        return ret

My code is kind of ugly but it does work. (The brainfuck code length is limited to 1000)

Given a crc sum and a byte we can calculate the next crc sum directly.

    inline unsigned int extend(unsigned int r, unsigned char s)
    {
            return (r <> 24) ^ s];
    }

`crc_table` is a pre-calculated table.

If the crc sum is 0, then \`r <> 8;
x |= (i ^ s) << 24;
if (extend(x, s) == r) {
//some code
}
}
}

But charset that we can use is limited, so we don't need to worry about too many solutions.

I used a bidirectional search algorithm to speed up my search in my code.
[crc32.cpp][5]

It works well even I only use `+-` to generate collisions.

    import subprocess
    from time import sleep
    from socket import *

    ...

    if __name__ == '__main__' :
            s = socket(AF_INET, SOCK_STREAM)
            s.connect(('<IP_ADDRESS>', 2000))
            print s.recv(1024)
            #raw = bf("""client.puts File.readlines("/etc/passwd")#""")
            #raw = bf("""client.puts File.readlines("./the_flag.rb")#""")
            raw = bf("""client.puts File.readlines("./rsa_keys.rb")#""")

            p = subprocess.Popen("./crc32", stdin = subprocess.PIPE, stdout = subprocess.PIPE)
            p.stdin.write(str(len(raw)) + "\n")
            p.stdin.write(raw + "\n")
            suffix = p.stdout.read()

            payload = raw + suffix

            print len(payload)
            print payload

            s.send(payload + "\n")
            s.send("1\n")
            sleep(5)
            print s.recv(4096)

[go.py][6]
Then I read [./the_flag.rb][7] and [./rsa_keys.rb][8].

The FLAG is `goozbartouuu`

 [1]: http://slipper.tk/writeup/2013/07/11/sigint-2013-crypto-satisfaction/
 [2]: https://github.com/5lipper/CTF-Challenges/blob/master/SIGINT2013/crypto/satisfaction/server.rb
 [3]: https://github.com/5lipper/CTF-Challenges/blob/master/SIGINT2013/crypto/satisfaction/bf.rb
 [4]: http://en.wikipedia.org/wiki/Crc32
 [5]: https://github.com/5lipper/CTF-Challenges/blob/master/SIGINT2013/crypto/satisfaction/crc32.cpp
 [6]: https://github.com/5lipper/CTF-Challenges/blob/master/SIGINT2013/crypto/satisfaction/go.py
 [7]: https://github.com/5lipper/CTF-Challenges/blob/master/SIGINT2013/crypto/satisfaction/the_flag.rb
 [8]: https://github.com/5lipper/CTF-Challenges/blob/master/SIGINT2013/crypto/satisfaction/rsa_keys.rb


text:
# jetson-stats-grafana-dashboard

## Introduction
This project is a Grafana dashboard, driven by a Prometheus metrics collector, for monitoring NVIDIA Jetson devices [Xavier NX, Nano, AGX Xavier, TX1, TX2] operating autonomously in a `headless` fashion (i.e. no connected display and associated user-input devices).

![image info](./grafana_dashboard_panel.png)

---

## Prerequisites

The following hardware and software should already be configured and operational:

- NVIDIA Jetson device
- A host running [Grafana](https://grafana.com/)
- A host running [Prometheus](https://prometheus.io/)

---

## Installation

### Jetson Device

1. Ensure `jetson_stats` is installed on your Jetson device. Full instructions can be found on the project [GitHub repo](https://github.com/rbonghi/jetson_stats)

2. Next, install the Prometheus metrics collector

	```bash
	nvidia@agx-desktop $ pip install prometheus-client
	nvidia@agx-desktop $ sudo cp jetson_stats_prometheus_collector.py /usr/local/bin/
	```

3. Install the Prometheus metrics collector as a system background `systemd` service

	```bash
	nvidia@agx-desktop $ sudo cp jetson_stats_prometheus_collector.service /etc/systemd/system/
	nvidia@agx-desktop $ sudo systemctl daemon-reload
	nvidia@agx-desktop $ sudo systemctl start jetson_stats_prometheus_collector
	nvidia@agx-desktop $ sudo systemctl status jetson_stats_prometheus_collector
	```

	**Alternatively**, you can install the `systemd` service for the current user as shown below:

	```bash
	nvidia@agx-desktop $ mkdir -p ~/.config/systemd/user
	nvidia@agx-desktop $ sudo cp jetson_stats_prometheus_collector.service /etc/systemd/system/
	nvidia@agx-desktop $ systemctl --user daemon-reload
	nvidia@agx-desktop $ systemctl --user start jetson_stats_prometheus_collector
	nvidia@agx-desktop $ systemctl --user status jetson_stats_prometheus_collector
	```

**Note:** The Prometheus metrics collector listens on port `8000` by default. If you wish to change this you will need to edit the `jetson_stats_prometheus_collector.service` file and change the `--port` argument to your required port number. You will also need to ensure to use the same port value later in this readme when referencing `REPLACEME_YOUR_JETSON_PROMETHEUS_COLLECTOR_PORT`.

### Grafana Dashboard

1. Install required Grafana plugins and restart the Grafana service

	```bash
	$ grafana-cli plugins install marcusolsson-dynamictext-panel
	$ sudo systemctl restart grafana-server.service
	```

### Prometheus Metrics Collection

1. Open the Prometheus config:

	```bash
	$ sudo nano /etc/prometheus/prometheus.yml
	```

2. Add metrics collection job config for the Jetson device

	```yml
	scrape_configs:
	  - job_name: 'nvidia_jetson'
	    static_configs:
	    - targets: ['REPLACEME_YOUR_JETSON_HOST_IP:REPLACEME_YOUR_JETSON_PROMETHEUS_COLLECTOR_PORT']
	```

	**Note:** Replace the `REPLACEME_XX` parts with the corresponding values.

3. Restart the Prometheus service for the changes to take effect

	```bash
	$ sudo systemctl restart /etc/systemd/system/prometheus.service
	```

### Grafana Dashboard Panel

1. From the Grafana Admin Dashboard select `+Create` -> `Import`

2. Open the file `jetson_stats_grafana_dashboard.json` and update the `__inputs` entry for the `DS_MY_PROMETHEUS` value and save the file

3. Next, copy and paste the contents of modified `jetson_stats_grafana_dashboard.json` into the text box named `Import via panel json` or select the file using the `Upload JSON file` button

4. The `jetson-stats-grafana-dashboard` should now be available

repository: adaouda/jetson-stats-grafana-dashboard

repo_name: adaouda/jetson-stats-grafana-dashboard


text:
# `ng-book`

- Controllers are instantiated only when they are needed and discarded when they are not. Every time we switch a route or reload a view, the current controller gets cleaned up by Angular.
- Services provide a method to keep data around for the lifetime of the app and communicate controllers in a consistent manner.
- *Services are singleton objects instantiated once per app and are loaded only when necessary.*

## Registering a Service

1. Use the `angular.module` `factory`:

    angular.module('myApp.services', [])
      .factory('githubService', function() {
        var serviceInstance = {};
        return serviceInstance;
      }
    });

This means that `githubService` is now registered with the AngularJS app using the name `githubService` as its name.

2. Bracket notation:

    angular.module('myApp.services', [])
      .factory('githubService', [function($http) {
      }]);

## Using Services

To use a service, we need to identify it as a dependency for the component where we're using it: a controller, a directive, a filter, or another service.

    angular.module('myApp', ['myApp.services'])
      .controller('ServiceController',
        function($scope, githubService){...

*If your application requires authentication from a back-end service, we might want to create a `SessionsService` that handles user authentication and holds onto a token passed by the back-end service.*

## Options for Creating Services

1. `factory()`

`factory` takes in two arguments, `getFn(function)`. This function gets run when angular creates the services.

    angular.module('myApp.services', [])
      .factory('githubService', [function($http) {
        return {
          'username': 'auser'
        }
      }]);

2. `service`

We can use this to register a constructor function for our service object.

    var Person = function($http) {
      this.getName = function(){
        return $http({
          method: 'GET',
          url: '/api/user'
        });
      };
    };

    angular.services('personService', Person);

The `service()` function will instantiate the instance using the `new` keyword when creating the instance.

3. `provider`

These are created through the `$provide` service, which is responsible for instantiating these providers at run time. A provider is an object with a `$get` method. The `$injector` calls the `$get` method to create a new instance of the service. `$provider` exposes several different API methods for creating a service, each with a different intended use.

The `provider()` method is responsible for registering services in the `$providerCache`. `factory` is shorthand for doing something through `provider`.

> Factory

    angular.module('myApp.services', [])
      .factory('githubService', [function($http) {
        return {
          'username': 'auser'
        }
      }]);

> Provider

    .provider('myService', {
      $get: function() {
        return {
          'username': 'auser'
        }
      }
    });

Use the `provider` method because it gives you the ability to configure service returned by the `.provider()` method using the Angular `.config()` function. We can inject a special attribute into the `config()` method.

*If you want to configure the services in the `config()` function, we must use `provider()` to define our service.*

4. `constant()`

We can register an existing value as a services that we can later inject into other parts of our app as a services.

    angular.module('myApp')
      .constant('apiKey', '123123123')

Injection:

    angular.module('myApp')
      .controller('MyController',
        function($scope, apiKey) {
          $scope.apiKey = apiKey;
        });

5. `value()`

[TODO]: THIS.

6. `decorator()`

[TODO]: THIS.

# Angular: Service vs. Provider vs. Factory
[link](http://stackoverflow.com/questions/15666048/angular-js-service-vs-provider-vs-factory)

*Services:* When declaring `serviceName` as an injectable argument, you will be provided with an instance of the function. You get `new FunctionYouPassedToService()`.

*Factories:*  You get the value that is returned when the function reference is invoked.

*Providers:* (`module.provider('providerName' ,function)`). When declaring `providerName` as an injectable argument you will be provided with `ProviderFunction().$get()`. They can be configured during the module configuration phase.

# AngularJS Step-by-Step: Services
[link](http://blog.pluralsight.com/angularjs-step-by-step-services)

When I say service, I really just refer to a simple object that does some sort of work. The main public members of a service are functions, not properties, especially since having properties suggests some sort of state.

SRP: The controller's job is to wire up the scope. If your controller is also responsible for making AJAX calls to fetch and update data, this is a violation of the SRP principle. Logic like that should be abstracted out into a separate service, then injected into the objects that need to use it.

DIP: Objects should depend on abstractions, not concretions. In JS, you can look at any parameter of any function (constructor or otherwise) as an abstractions, since you can pass in any object for that parameter so long as it has the members on it that are used within that method. The key here is the ability to use DI--the ability to inject into other objects.

license: Unlicense

repository: daryllxd/lifelong-learning

gha_language: Elixir

gha_license_id: Unlicense

repo_name: daryllxd/lifelong-learning


text:
## V1 Use Cases

This document contains the complete list of accepted use-cases for the v1
version of `service-catalog`.

## Use cases

1.  Sharing services:
    1.  (Blackbox services) As a SaaS provider that already runs a Service
        Broker, I want users of Kubernetes to be able to use my service
        via the Service Broker API, so that I can grow my user base to
        include users of Kubernetes
    2.  As the operator of an existing service running in Kubernetes, I want to
        be able to publish my services using a Service Broker, so that users
        external to my Kubernetes cluster can use my service

### Blackbox services

There are numerous SaaS providers that already operate Service Brokers today.
It should be possible for the operator of an existing Service Broker to
publish their services into the catalog and have them consumed by users of
Kubernetes.  This offers a new set of users to the service operator and offers
a wide variety of SaaS products to users of Kubernetes.

### Exposing existing services outside the cluster

It should be possible for service operators whose services are deployed in a
Kubernetes cluster to publish their services using a Service Broker.  This
would allow these operators to participate in the existing Service Broker
ecosystem and grow their user base accordingly.

repository: KurtStam/service-catalog

repo_name: KurtStam/service-catalog


text:
## Renderer2D

# Public Functions:
**static void Init()**
- Allocates new Renderer2DStorage
- Inits the VertexArray, VertexBuffer, and IndexBuffer
- The data for these are being made and set here but should be moved outside of this specific function

**static void Shutdown()*
- Deletes s_Data

**static void BeginScene(const OrthographicCamera& camera)**
- Each layer will call this in their OnUpdate
- Binds s_Data's shader and sets the shader's ViewProjectionMatrix to the camera's ViewProjectionMatrix

**static void EndScene()**
- Each layer will call this in their OnUpdate

**static void DrawQuad(const glm::vec2& position, const glm::vec2& size, const glm::vec4& color)**
- see below

**static void DrawQuad(const glm::vec3& position, const glm::vec2& size, const glm::vec4& color)**
- This is for flat color quads
- Sets s_Data's texture shader color, tiling factor, transform
- Binds the WhiteTexture shader
- Binds s_Data's VertexArray
- Issues RenderCommand::DrawIndexed call

**static void DrawQuad(const glm::vec2& position, const glm::vec2& size, const Ref<Texture2D>& texture, float tilingFactor = 1.0f, const glm::vec4& tintColor = glm::vec4(1.0f))**
- see below

**static void DrawQuad(const glm::vec3& position, const glm::vec2& size, const Ref<Texture2D>& texture, float tilingFactor = 1.0f, const glm::vec4& tintColor = glm::vec4(1.0f))**
- This is for textured quads
- Sets s_Data's texture shader tint color, tiling factor, transform
- Binds the given texture2D
- Binds s_Data's VertexArray
- Issues RenderCommand::DrawIndexed call

**static void DrawRotatedQuad(const glm::vec2& position, const glm::vec2& size, float rotation, const glm::vec4& color)**
- same as above but with a radian factor

**static void DrawRotatedQuad(const glm::vec3& position, const glm::vec2& size, float rotation, const glm::vec4& color)**
- same as above but with a radian factor

**static void DrawRotatedQuad(const glm::vec2& position, const glm::vec2& size, float rotation, const Ref<Texture2D>& texture, float tilingFactor = 1.0f, const glm::vec4& tintColor = glm::vec4(1.0f))**
- same as above but with a radian factor

**static void DrawRotatedQuad(const glm::vec3& position, const glm::vec2& size, float rotation, const Ref<Texture2D>& texture, float tilingFactor = 1.0f, const glm::vec4& tintColor = glm::vec4(1.0f))**
- same as above but with a radian factor

# Related but not in class
**struct Renderer2DStorage**
- Ref<VertexArray> QuadVertexArray
- Ref<Shader> TextureShader
- Ref<Texture2D> WhiteTexture

**static Renderer2DStorage* s_Data**


text:
OpenBuildings

## Brief

A global collection of building polygon data, open and free to use.

Currently just a basic list of potential sources to track openaddresses style

## Sources

### California

| Coverage | Source | Number | Date |
| -------- | ------ | ------ | ---- |
| San Francisco | `https://data.sfgov.org/Housing-and-Buildings/Building-Footprints/72ai-zege/data` | 177,023 | 2017 |
| Los Angeles County | `https://egis3.lacounty.gov/dataportal/2016/11/03/countywide-building-outlines-2014-update-public-domain-release/` | 3,000,000+ |  2014 |
| City of Los Angeles | `http://geohub.lacity.org/datasets/813fcefde1f64b209103107b26a8909f_0` | 1109238 | 2014 |

### Washington, DC

| Coverage | Source |
| -------- | ------ |
| Statewide | `https://maps2.dcgis.dc.gov/dcgis/rest/services/DCGIS_DATA/Facility_and_Structure/MapServer/1` |

### New York

| Coverage | Source |
| -------- | ------ |
| The City of New York | `https://data.cityofnewyork.us/api/geospatial/nqwf-w8eh?method=export&format=Original` |

### Maryland

| Coverage | Source |
| -------- | ------ |
| Allegany County | `http://alleganygis.allconet.org/allcogis/rest/services/Dashboards/BWPermits/MapServer/11` |
| Anne Arundel County | `http://www.aacounty.org/AACoOIT/GIS/aaco_buildings.zip` |
| Baltimore City | `https://maps.baltimorecity.gov/egis/rest/services/OpenBaltimore/Building_Footprint/MapServer/0` |
| Baltimore County | `https://bcgis.baltimorecountymd.gov/arcgis/rest/services/Facilities/Buildings/MapServer/2/query?outFields=*&where=1%3D1` |
| Calvert County | `http://calvertgis.co.cal.md.us/arcgis/rest/services/Basemaps/Streets_Base_Map/MapServer/10` |
| Caroline County | `` |
| Carroll County | `https://services.arcgis.com/Uf0DiYpD9NOFO5YH/ArcGIS/rest/services/Buildings_CarrollCounty/FeatureServer/0` |
| Cecil County | `http://cecilmaps.ccgov.org/arcgis/rest/services/Public_10_1/MapServer/5`, `http://cecilmaps.ccgov.org/arcgis/rest/services/Public_10_1/MapServer/6` |
| Charles County | `` |
| Dorchester County | `` |
| Frederick County | `` |
| Garrett County | `` |
| Harford County | `` |
| Howard County | `` |
| Kent County | `` |
| Montgomery County | `` |
| Prince George's County | `` |
| Queen Anne's County | `` |
| Saint Mary's County | `` |
| Somerset County | `` |
| Talbot County | `` |
| Washington County | `` |
| Wicomico County | `` |
| Worcester County | `` |

### Washington

| Coverage | Source |
| -------- | ------ |
| City of Seattle | `https://data.seattle.gov/download/y7u8-vad7/application%2Fzip` |

### Virginia
| Coverage | Source |
| -------- | ------ |
| Statewide | `http://gismaps.vita.virginia.gov/arcgis/rest/services/VA_Base_layers/VA_Building_Footprints/MapServer/0` |

repository: trescube/openbuildings

repo_name: trescube/openbuildings


text:
# For Testers
If you want to go for a quick test drive of the SparkFHE code within the Apache Spark environment, please visit and follow these instructions, https://github.com/SpiRITlab/SparkFHE-Examples/wiki

# For Developers
If you are a developer working on the SparkFHE-Examples code, you can use the following instructions to develop new example code. You will also need to setup the Apache Spark environment, because it will download and install all dependencies. If you haven't done so, please visit and follow these instructions, https://github.com/SpiRITlab/SparkFHE-Examples/wiki

Note, there are two pom files which you can use to compile or package:
```bash
pom-devel.xml                   # will use the existing shared lib within ./libSparkFHE/lib
pom.xml                         # will download and refresh the C++ shared lib from our repo
```

## 1. Soft-link the libSparkFHE folder
Note, change ${PATH_TO_spark-3.0.0-SNAPSHOT-bin-SparkFHE} to where you install "spark-3.0.0-SNAPSHOT-bin-SparkFHE"
```bash
cd SparkFHE-Examples
ln -s ${PATH_TO_spark-3.0.0-SNAPSHOT-bin-SparkFHE}/libSparkFHE/ ./
```

## 2. Get SparkFHE-Addon
```bash
./mvn -f pom-devel.xml scm:checkout
```

## 3. Compile the example code
The example code will use the shared libraries downloaded while setting up the Apache Spark environment.
```bash
./mvn -f pom-devel.xml -U clean compile
```
Note, without '-f pom-devel.xml', maven will download a new shared library from our repo and overwrite the existing one in libSparkFHE/lib.

## 4. Run the demo code
### Run it with a script
```bash
cd scripts
bash myMavenExampleRun.bash
```

OR run it step-by-step
### Step 1. Generate necessary key pair and example ciphertexts (only needed to run once)
```bash
# this will generate the example key pair
./mvn -f pom-devel.xml exec:java -Dexec.mainClass=spiritlab.sparkfhe.example.basic.KeyGenExample -Dexec.args="local"

# this will generate some ciphertexts
./mvn -f pom-devel.xml exec:java -Dexec.mainClass=spiritlab.sparkfhe.example.basic.EncDecExample -Dexec.args="local gen/keys/my_public_key.txt gen/keys/my_secret_key.txt"
```
### Step 2. Run examples: Test different FHE operations on example ciphertexts and vectors of ciphertexts
```bash
# this will perform some basic FHE operations
./mvn -f pom-devel.xml exec:java -Dexec.mainClass=spiritlab.sparkfhe.example.basic.BasicOPsExample -Dexec.args="local  gen/keys/my_public_key.txt gen/keys/my_secret_key.txt"

# this will perform dot product calculation on vectors of encrypted numbers
./mvn -f pom-devel.xml exec:java -Dexec.mainClass=spiritlab.sparkfhe.example.basic.DotProductExample -Dexec.args="local gen/keys/my_public_key.txt gen/keys/my_secret_key.txt"
```

## Run JUnit5 tests
```bash
./mvn -f pom-devel.xml test
```

## Package into .jar
```bash
./mvn -f pom-devel.xml -U -DskipTests clean package
```

# Known issues
You may see the these [warnings](https://github.com/SpiRITlab/SparkFHE-Examples/issues/7). It maybe due to Spark's internal bugs.

repository: aamirjamal/SparkFHE-Examples

repo_name: aamirjamal/SparkFHE-Examples


text:
# ironSource.atom SDK for IOS

[![License][license-image]][license-url]
[![Docs][docs-image]][docs-url]
[![Build status][travis-image]][travis-url]
[![Coverage Status][coverage-image]][coverage-url]
[![Pods][pod-image]][pod-url]

atom-ios is the official [ironSource.atom](http://www.ironsrc.com/data-flow-management) SDK for the IOS platform.

- [Signup](https://atom.ironsrc.com/#/signup)
- [Documentation](https://ironsource.github.io/atom-ios/)
- [Installation](#installation)
- [Usage](#usage)
- [Change Log](#change-log)
- [Example](#example)

## Installation

### Installation from [CocoaPods](https://cocoapods.org/?q=atomsdk).
Add dependency in your pod file
```ruby
pod 'AtomSDK'
```

## Usage

### High Level SDK - "Tracker"
The Tracker is used for sending events to Atom based on several conditions
- Every 10 seconds (default)
- Number of accumulated events has reached 50 (default)
- Size of accumulated events has reached 64KB (default)
Case of server side failure (500) the tracker uses an exponential back off mechanism with jitter.
To see the full code check the [example section](#example)

Example of sending an event in Swift:
```swift
// initialize atom-sdk api object
var apiTracker_ = ISAtomTracker()
apiTracker_!.enableDebug(true)
apiTracker_!.setAuth("<YOUR_AUTH_KEY>")
apiTracker_!.setBulkSize(<BULK_COUNT>)
apiTracker_!.setBulkBytesSize(<MAX_BULK_SIZE_IN_BYTES>)
apiTracker_!.setEndPoint("https://track.atom-data.io/")

// track event
apiTracker_!.trackWithStream("<YOUR_STREAM_NAME>",
                                data: "{\"test\":\"test\"}")

// flush all data in tracker
apiTracker_!.flush()

```
Example of sending an event in Objective-C:
```objc
// initialize atom-sdk api object
ISAtomTracker* apiTracker_ = [[ISAtomTracker alloc] init];
[apiTracker_ enableDebug:true];
[apiTracker_ setBulkSize:<BULK_COUNT>];
[apiTracker_ setBulkBytesSize:<MAX_BULK_SIZE_IN_BYTES>];
[apiTracker_ setEndPoint:@"https://track.atom-data.io/"];

// track event
[apiTracker_ trackWithStream: @"<YOUR_STREAM_NAME>" data: @"{\"test\":\"test\"}"
                       token: @"<YOUR_AUTH_KEY>"];

// flush all data in tracker
[apiTracker_ flush];
```

### Low Level (Basic) SDK
The Low Level SDK has 2 methods:
- putEvent - Sends a single event to Atom
- putEvents - Sends a bulk (batch) of events to Atom

Example of sending an event in Swift:
```swift
// initialize atom-sdk api object
    var api_ = ISAtom()

    // print debug info in console
    api_!.enableDebug(true)
    api_!.setAuth("<YOUR_AUTH_KEY>")

    // send POST request
    api_!.putEventWithStream("<YOUR_STREAM_NAME>",
        data: "{\"test\":\"test\"}",
        callback: {response in

            let errorStr = (response.error == "") ? "nil" : "\"\(response.error)\""
            let dataStr = (response.data == "") ? "nil" : "\"\(response.data)\""
            let statusStr = "\(response.status)"
    })

    // send Bulk request
    api_!.putEventsWithStream("<YOUR_STREAM_NAME>",
                              arrayData: ["{\"test\":\"test\"}", "{\"test\":\"test\"}"],
                              callback: { response in

                                let errorStr = (response.error == "") ? "nil" : "\"\(response.error)\""
                                let dataStr = (response.data == "") ? "nil" : "\"\(response.data)\""
                                let statusStr = "\(response.status)"

                                let responseStr = "{\n\t\"error\": \(errorStr), " +
                                    "\n\t\"data\": \(dataStr), " +
                                    "\n\t\"status\": \(statusStr)\n}"
    })

    // check health of server
    api_!.health()
```
Example of sending an event in Objective-C:
```objc
// initialize atom-sdk api object
ISAtom* api_ = [[ISAtom alloc] init];

// print debug info in console
[api_ enableDebug:true];
[api_ setAuth: @"<YOUR_AUTH_KEY>"];

// send POST request
ISRequestCallback callback = ^(ISResponse* response) {
    // reponse data from server
};

[api_ putEventWithStream: @"<YOUR_STREAM_NAME>"
                    data: @"{\"test\":\"test\"}"
                callback: callback];

// send Bulk request
NSArray* data = [NSArray arrayWithObjects:@"{\"test\":\"test\"}", @"{\"test\":\"test\"}", nil];
[api_ putEventsWithStream: @"<YOUR_STREAM_NAME>"
                arrayData: data
                 callback: callback];

// check health of server
[api_ health];
```

## Swift version
There is a [legacy swift version](https://github.com/ironSource/atom-ios/tree/swift) of the sdk.
**it is considered beta and it is not officially supported.**

## Change Log

### v1.2.0 - Rewrite to Objective C
- Full re-write to Objective C

### v1.1.2
- Adding Cococapods release
- Changing CommonCrypto to Arcane
- Bugfix - fixing the case when there are multiple streams that need to be flushed
while there is a bad response from server (retry Qs).
- Increased coverage

### v1.0.0 - Base Swift version
- Basic features: putEvent, putEvents and Tracker

## Example
You can use our [example][example-url] for sending data to Atom:

![alt text][example]

## License
[MIT](LICENSE)

[docs-image]: https://img.shields.io/badge/docs-latest-blue.svg
[docs-url]: https://ironsource.github.io/atom-ios/
[pod-image]: https://img.shields.io/cocoapods/v/AtomSDK.svg
[pod-url]: https://cocoapods.org/?q=AtomSDK
[travis-image]: https://travis-ci.org/ironSource/atom-ios.svg?branch=master
[travis-url]: https://travis-ci.org/ironSource/atom-ios
[coverage-image]: https://coveralls.io/repos/github/ironSource/atom-ios/badge.svg?branch=master
[coverage-url]: https://coveralls.io/github/ironSource/atom-ios?branch=master
[license-image]: https://img.shields.io/badge/license-MIT-blue.svg
[license-url]: LICENSE
[example]: https://cloud.githubusercontent.com/assets/1713228/15971662/08129c62-2f43-11e6-980d-66d36a41f961.png "example"
[example-url]: atom-sdk/AtomSDKExample

repository: metricjar/atom-ios

repo_name: metricjar/atom-ios


text:
# bull-repl

Bull / BullMQ queue command line REPL.

### Version compatibility

- For bull 3.x.x use `bull-repl@latest` (0.x.x)
- For bullmq use `bull-repl@next` (1.x.x)

### Install

```
npm install bull-repl -g
```

### Run

```
bull-repl
```

### Overview help

```
help
```

```
  Commands:

    help [command...]                  Provides help for a given command.
    exit                               Exits application.
    connect [options] <queue>          Connect to bull queue
    connect-list                       List of saved connections
    connect-rm <name>                  Remove saved connection
    connect-save <name>                Save current connection
    connect-to <name>                  Connect to saved connection
    stats                              Count of jobs by type
    active [options]                   Fetch active jobs
    waiting [options]                  Fetch waiting jobs
    completed [options]                Fetch completed jobs
    failed [options]                   Fetch failed jobs
    delayed [options]                  Fetch delayed jobs
    pause [options]                    Pause current queue
    resume [options]                   Resume current queue from pause
    get <jobId...>                     Get job
    add [options] <data>               Add job to queue e.g. add '{"x": 1}'
    rm [options] <jobId...>            Remove job
    retry [options] <jobId...>         Retry job
    retry-failed [options]             Retry first 100 failed jobs
    promote [options] <jobId...>       Promote job
    fail [options] <jobId> <reason>    Move job to failed
    complete [options] <jobId> <data>  Move job to completed e.g. complete 1 '{"x": 1}'
    clean [options] <period>           Clean queue for period ago, period format - https://github.com/zeit/ms#examples
    logs [options] <jobId>             Get logs of job
    log <jobId> <data>                 Add log to job
    events-on                          Turn on logging of queue events
    events-off                         Turn off logging of queue events
```

### Connect help

```
connect --help
```

```
  Usage: connect [options] <queue>

  Connect to bull queue

  Options:

    --help                 output usage information
    --prefix <prefix>      Prefix to use for all queue jobs
    -h, --host <host>      Redis host for connection
    -p, --port <port>      Redis port for connection
    -d, --db <db>          Redis db for connection
    --password <password>  Redis password for connection
    -c, --cert <cert>      Absolute path to pem certificate if TLS used
    -u, --url <url>        Redis sentinel format URL
    -e, --exec <exec>      Exec command
```

### Fetch jobs by group help

```
active --help
```

```
  Usage: active [options]

  Fetch active jobs

  Options:

    --help                   output usage information
    -q, --query <query>      Query jobs via jq - https://stedolan.github.io/jq/manual/#Basicfilters. Notice, that bull data in root key e.g '[.root[] | select(.progress > 70)]'
    -t, --timeAgo <timeAgo>  Get jobs since time ago via https://github.com/zeit/ms#examples
    -s, --start <start>      Start index (pagination)
    -e, --end <end>          End index (pagination)
```

### Notes

- You can see help on each command, for example: `connect --help`
- You can predefine startup command, when run bull-repl. For example: `bull-repl connect my-queue`

repository: darky/bull-repl

gha_language: TypeScript

repo_name: darky/bull-repl


text:
# Team meeting Updates🔊

## Day 1 - 19 August 2021⏰

### Meeting Time : 3 p.m to 4 p.m  and   7:30 p.m to 8:30 p.m

#### Meeting Minutes - Select the Project, Team name and Distribute the Day 1 task among all team members.

## Day 2 - 20 August 2021⏰

### Meeting Time : 2 p.m to 3 p.m  and   7:30 p.m to 8:30 p.m

#### Meeting Minutes - Distribute the applications of the project to each member and UML Diagram.

## Day 3 - 21 August 2021⏰

### Meeting Time : 11 a.m to 12 p.m  and   5:30 p.m to 6:30 p.m

#### Meeting Minutes - Each member share there progress to respected application and disscus on Day 4 task to complete.

## Day 4 - 22 August 2021⏰

### Meeting Time : 11 a.m to 12 p.m  and   3:30 p.m to 4:30 p.m

#### Meeting Minutes - Some member completed their task and push their code in team ripo. Other were working on it and also disscus on Day 5 task to complete.

## Day 5 - 23 August 2021⏰

### Meeting Time : 4 p.m to 5 p.m  and   7:15 p.m to 8 p.m

#### Meeting Minutes - Members of the group added some feautures and updated the Test plan and Contributors list. All the members had connect session with the mentor for the feedback and what changes or improvements need to be done before final submissions.

## Day 6 - 24 August 2021⏰

### Meeting Time : 2 p.m to 3:30 p.m  and  7:15 to 7:30 p.m

#### Meeting Minutes - Today is the final day of our projet. All push there code in ripo and it is passing all the checks. We also made a working video of our project and have meeting with mentor to show the output our project.


text:
# About
Project from Udacity. <br>
Final project from 3rd Section
Original [GitHub project](https://github.com/udacity/CVND---Image-Captioning-Project)

# Aproach
* Started with a simple DecoderRNN of: 1 Embedding Layer -> LSTM -> Linear
* Tried to add an attention layer after LSTM: 1 Embedding Layer -> LSTM -> MultiHeadAttention -> Linear
* Tried to add an attention layer before LSTM: 1 Embedding Layer -> MultiHeadAttention -> LSTM -> Linear
* Found in the internet literature on best case performances and how they did it. Particularly found [GitHub Example](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Image-Captioning) as well as [this medium tutorial](https://medium.com/analytics-vidhya/image-captioning-with-attention-part-1-e8a5f783f6d3)
* Compared performance of all models with an excelent example from: [here](https://medium.com/analytics-vidhya/image-captioning-with-attention-part-1-e8a5f783f6d3)

# Instructions

## Submission Files
* :white_check_mark:`models.py`
* :white_check_mark:`1_Preliminaries.ipynb`
* :white_check_mark:`2_Training.ipynb`
* :white_check_mark:`3_Inference.ipynb`

## Project Rubric [link_original](https://review.udacity.com/#!/rubrics/1427/view)

### `models.py`

#### Specify the CNNEncoder and RNNDecoder
| Criteria       		|     Meets Specifications	        			            |
|:---------------------|:---------------------------------------------------------:|
|  :white_check_mark: `CNNEncoder`. |  The chosen CNN architecture in the `CNNEncoder` class in **model.py** makes sense as an encoder for the image captioning task.|
| :white_check_mark: `RNNDecoder`. |  The chosen RNN architecture in the `RNNDecoder` class in **model.py** makes sense as a decoder for the image captioning task.|

### 2_Training.ipynb

| Criteria       		|     Meets Specifications	        			            |
|:---------------------|:---------------------------------------------------------:|
| :white_check_mark:Using the Data Loader |  When using the `get_loader` function in **data_loader.py** to train the model, most arguments are left at their default values, as outlined in **Step 1** of **1_Preliminaries.ipynb**. In particular, the submission only (optionally) changes the values of the following arguments: `transform`, `mode`, `batch_size`, `vocab_threshold`, `vocab_from_file`. |
| :white_check_mark:Step 1, Question 1 |  The submission describes the chosen CNN-RNN architecture and details how the hyperparameters were selected. |
| :white_check_mark:Step 1, Question 2 |  The transform is congruent with the choice of CNN architecture. If the transform has been modified, the submission describes how the transform used to pre-process the training images was selected.|
| :white_check_mark:Step 1, Question 3 |  The submission describes how the trainable parameters were selected and has made a well-informed choice when deciding which parameters in the model should be trainable.|
| :white_check_mark:Step 1, Question 4 |  The submission describes how the optimizer was selected.|
| :white_check_mark:Step 2 |  The code cell in **Step 2** details all code used to train the model from scratch. The output of the code cell shows exactly what is printed when running the code cell. If the submission has amended the code used for training the model, it is well-organized and includes comments.|

### 3_Inference.ipynb

| Criteria       		|     Meets Specifications	        			            |
|:---------------------|:---------------------------------------------------------:|
| :white_check_mark:`transform_test` |  The transform used to pre-process the test images is congruent with the choice of CNN architecture. It is also consistent with the transform specified in `transform_train` in **2_Training.ipynb**.|
| :white_check_mark:Step 3 | The implementation of the `sample` method in the `RNNDecoder` class correctly leverages the RNN to generate predicted token indices.|
| :white_check_mark:Step 4 | The `clean_sentence` function passes the test in **Step 4**. The sentence is reasonably clean, where any `<start>` and `<end>` tokens have been removed.|
| :white_check_mark: Step 5 | The submission shows two image-caption pairs where the model performed well, and two image-caption pairs where the model did not perform well.|

## Bonus :boom::boom::boom:
* :white_check_mark: Use the validation set to guide your search for appropriate hyperparameters.
* :white_check_mark: Tinker with your model - and train it for long enough - to obtain results that are comparable to (or surpass!) recent research articles
* :exclamation: Implement beam search to generate captions on new images.

# Results

## Approach
From the varios rounds performed, I evaluated 3 different runs:
`v102_paramsDecoder_withoutFlipTrans_batch_size10_vocabThr3_embedSize512_hiddenSize1024_totEpochs10
v121_paramsDecoder_withoutFlipTrans_batch_size10_vocabThr3_embedSize512_hiddenSize1024_totEpochs3
v120_paramsDecoder_withoutFlipTrans_batch_size10_vocabThr3_embedSize512_hiddenSize1024_totEpochs3`

All using the benchmark of the model found [here](https://medium.com/analytics-vidhya/image-captioning-with-attention-part-1-e8a5f783f6d3)
`v120_paramsDecoder_withoutFlipTrans_batch_size10_vocabThr3_embedSize512_hiddenSize1024_totEpochs3`

## Final Results
**Benchmark** (`v200_paramsDecoder_withoutFlipTrans_batch_size10_vocabThr4_embedSize512_hiddenSize1024_totEpochs10`)
* Attention: 2 epochs:
> ratio: 0.9943724833842694; Bleu_1: 0.679; Bleu_2: 0.496; Bleu_3: 0.344; Bleu_4: 0.232; METEOR: 0.211; ROUGE_L: 0.491; CIDEr: 0.639
* Attention: 3 epochs:
> ratio: 1.0094033452608777; Bleu_1: 0.659; Bleu_2: 0.484; Bleu_3: 0.339; Bleu_4: 0.232; ; METEOR: 0.214; ROUGE_L: 0.488; CIDEr: 0.640

**Best and most Simple** (`v102_paramsDecoder_withoutFlipTrans_batch_size10_vocabThr3_embedSize512_hiddenSize1024_totEpochs10`)
* my simple model - 1 epochs:
> ratio: 1.009220942322141; Bleu_1: 0.587; Bleu_2: 0.383; Bleu_3: 0.244; Bleu_4: 0.157; METEOR: 0.182; ROUGE_L: 0.424; CIDEr: 0.507
* my simple model - 2 epochs:
> ratio: 1.0005286538264253; Bleu_1: 0.616; Bleu_2: 0.416; Bleu_3: 0.275; Bleu_4: 0.183; METEOR: 0.192; ROUGE_L: 0.442; CIDEr: 0.574
* my simple model - 3 epochs:
> ratio: 1.014850954868193; Bleu_1: 0.637; Bleu_2: 0.449; Bleu_3: 0.306; Bleu_4: 0.209; METEOR: 0.211; ROUGE_L: 0.466; CIDEr: 0.667
* my simple model - 4 epochs:
> ratio: 1.0104244476961306; Bleu_1: 0.636 Bleu_2: 0.448; Bleu_3: 0.305; Bleu_4: 0.209; METEOR: 0.210; ROUGE_L: 0.464; CIDEr: 0.669
* my simple model - 5 epochs:
> ratio: 1.012540160924407; Bleu_1: 0.632; Bleu_2: 0.446; Bleu_3: 0.306; Bleu_4: 0.211; METEOR: 0.213; ROUGE_L: 0.466; CIDEr: 0.668

**Attention Before LSTM** (`v121_paramsDecoder_withoutFlipTrans_batch_size10_vocabThr3_embedSize512_hiddenSize1024_totEpochs3`)
* my att first model - 1 epochs:
> ratio: 1.102866731062698; Bleu_1: 0.357; Bleu_2: 0.179; Bleu_3: 0.065; Bleu_4: 0.029; METEOR: 0.091; ROUGE_L: 0.285; CIDEr: 0.037
* my att first model - 2 epochs:
> ratio: 0.706783945336399; Bleu_1: 0.205; Bleu_2: 0.091; Bleu_3: 0.032; Bleu_4: 0.014; METEOR: 0.050; ROUGE_L: 0.174; CIDEr: 0.015

**Attention After LSTM** (`v120_paramsDecoder_withoutFlipTrans_batch_size10_vocabThr3_embedSize512_hiddenSize1024_totEpochs3`)
* my att aft model - 1 epochs:
> ratio: 0.33351825744295616; Bleu_1: 0.052; Bleu_2: 0.034; Bleu_3: 0.021; Bleu_4: 0.014; METEOR: 0.037; ROUGE_L: 0.101; CIDEr: 0.071
* my att aft model - 2 epochs:
> ratio: 0.21256804304037424; Bleu_1: 0.005; Bleu_2: 0.005; Bleu_3: 0.004; Bleu_4: 0.002; METEOR: 0.014; ROUGE_L: 0.034; CIDEr: 0.029

## Thoughts
My best model is clearly 1.0.2 (although simple), proved to be better than adding a multihead attention.
And performance wise is close to the most complicated one used to benchmark.


text:
# Kubernetes Agent architecture

## Issues the agent is trying to address

- Integrate a cluster, located behind a firewall or NAT, with GitLab. To
  learn more, read [issue #212810, Invert the model GitLab.com uses for Kubernetes integration by leveraging long lived reverse tunnels](https://gitlab.com/gitlab-org/gitlab/-/issues/212810).
- Access API endpoints in a cluster in real time. For an example use case, read
  [issue #218220, Allow Prometheus in K8s cluster to be installed manually](https://gitlab.com/gitlab-org/gitlab/-/issues/218220#note_348729266).
- Enable real-time features by pushing information about events happening in a cluster.
  For example, you could build a cluster view dashboard to visualize changes in progress
  in a cluster. For more information about these efforts, read about the
  [Real-Time Working Group](https://about.gitlab.com/company/team/structure/working-groups/real-time/).
- Enable a [cache of Kubernetes objects through informers](https://github.com/kubernetes/client-go/blob/ccd5becdffb7fd8006e31341baaaacd14db2dcb7/tools/cache/shared_informer.go#L34-L183),
  kept up-to-date with very low latency. This cache helps you:

  - Reduce or eliminate information propagation latency by avoiding Kubernetes API calls
    and polling, and only fetching data from an up-to-date cache.
  - Lower the load placed on the Kubernetes API by removing polling.
  - Eliminate any rate-limiting errors by removing polling.
  - Simplify backend code by replacing polling code with cache access. While it's another
    API call, no polling is needed. This example describes [fetching cached data synchronously from the front end](https://gitlab.com/gitlab-org/gitlab/-/issues/217792#note_348582537) instead of fetching data from the Kubernetes API.

## High-level architecture

The GitLab Kubernetes Agent and the GitLab Kubernetes Agent Server use
[bidirectional streaming](https://grpc.io/docs/what-is-grpc/core-concepts/#bidirectional-streaming-rpc)
to allow the connection acceptor (the gRPC server, GitLab Kubernetes Agent Server) to
act as a client. The connection acceptor sends requests as gRPC replies. The client-server
relationship is inverted because the connection must be initiated from inside the
Kubernetes cluster to bypass any firewall or NAT the cluster may be located behind.
To learn more about this inversion, read
[issue #212810](https://gitlab.com/gitlab-org/gitlab/-/issues/212810).

This diagram describes how GitLab (`GitLab RoR`), the GitLab Kubernetes Agent (`agentk`), and the GitLab Kubernetes Agent Server (`kas`) work together.

```mermaid
graph TB
  agentk -- gRPC bidirectional streaming --> kas

  subgraph "GitLab"
  kas[kas]
  GitLabRoR[GitLab RoR] -- gRPC --> kas
  kas -- gRPC --> Gitaly[Gitaly]
  kas -- REST API --> GitLabRoR
  end

  subgraph "Kubernetes cluster"
  agentk[agentk]
  end
```

- `GitLab RoR` is the main GitLab application. It uses gRPC to talk to `kas`.
- `agentk` is the GitLab Kubernetes Agent. It keeps a connection established to a
  `kas` instance, waiting for requests to process. It may also actively send information
  about things happening in the cluster.
- `kas` is the GitLab Kubernetes Agent Server, and is responsible for:
  - Accepting requests from `agentk`.
  - [Authentication of requests](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/identity_and_auth.md) from `agentk` by querying `GitLab RoR`.
  - Fetching agent's configuration from a corresponding Git repository by querying Gitaly.
  - Matching incoming requests from `GitLab RoR` with existing connections from
    the right `agentk`, forwarding requests to it and forwarding responses back.
  - (Optional) Sending notifications through ActionCable for events received from `agentk`.
  - Polling manifest repositories for [GitOps support](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/gitops.md) by communicating with Gitaly.

<i class="fa fa-youtube-play youtube" aria-hidden="true"></i>
To learn more about how the repository is structured, see
[GitLab Kubernetes Agent repository overview](https://www.youtube.com/watch?v=j8CyaCWroUY).

## Guiding principles

GitLab prefers to add logic into `kas` rather than `agentk`. `agentk` should be kept
streamlined and small to minimize the need for upgrades. On GitLab.com, `kas` is
managed by GitLab, so upgrades and features can be added without requiring you
to upgrade `agentk` in your clusters.

`agentk` can't be viewed as a dumb reverse proxy because features are planned to be built
[on top of the cache with informers](https://github.com/kubernetes/client-go/blob/ccd5becdffb7fd8006e31341baaaacd14db2dcb7/tools/cache/shared_informer.go#L34-L183).

## FAQ

- **Q**: Why do we need long-running connections? Cannot we just ask users to punch a hole in their firewall?

  **A**: Even if it was always possible, having an agent running in the cluster enables us to build more features, not just connecting such clusters to GitLab.

- **Q**: Why do we need long-running connections? Can we use polling instead?

  **A**: Polling will not allow for real-time access to the in-cluster APIs. For example, our metrics integration queries Prometheus API to get the data for dashboards. I.e. the request comes from GitLab, not from the Kubernetes side.

- **Q**: Can we push data about things happening in the cluster using REST API instead of using streaming?

  **A**: Yes, we could use REST API and push the data. But since we already need long-running connections (see above), why not utilize them? `agentk` uses gRPC which [multiplexes multiple logical channels](https://www.cncf.io/blog/2018/08/31/grpc-on-http-2-engineering-a-robust-high-performance-protocol/) onto a usually smaller number of TCP connections. There will be a smaller number of TCP connections if all communications are consolidated and happen over gRPC. See the link for technical details.

- **Q**: Can we put the cache into `kas` rather than into `agentk` to make it "dumber" / simpler per the principle above?

  **A**: Technically yes. However, that would mean `kas` would get an update for each event for object kinds `kas` runs informers for. An event contains the whole changed object. Number of events in active clusters may be significant. So, multiplied by the number of clusters, that means we'd have a lot of traffic between `kas` and `agentk`. There is no practical benefit for building it this way, only the downside of having a lot of useless traffic.

  Instead of the above we could have an up to date precomputed view on top of the cache in `kas`. `agentk` could make the calculations locally and push an update immediately to `kas` (which could push an event via ActionCable). For example, `agentk` could maintain a cache with [`Node`](https://kubernetes.io/docs/concepts/architecture/nodes/) objects and push the current number of nodes to `kas` each time there is a change. The UI then can fetch the number of nodes from `kas` via an API (via the main application or bypassing it).

- **Q**: Why use gRPC for `GitLab RoR` -> `kas` access? Why not REST API?

  **A**: To benefit from all the [good things gRPC provides or enables](https://grpc.io/faq/) and avoid any pitfalls of a hand-rolled client implementation. GitLab already uses gRPC to talk to Gitaly, it's an existing dependency.

- **Q**: Why poll Git repositories from `kas` and not from `agentk`?

  **A**: There are several reasons:

  - Allows the GitLab instance operator to configure polling frequency and any other relevant parameters. Makes it harder for the user to misuse/abuse the system.

  - Fewer knobs for the user to tweak means more straightforward configuration experience.

  - Makes it easier to implement pub/sub / push-based notifications about changes to the repo to reduce polling frequency.

  - It follows the "smart `kas`, dumb `agentk`" principle described above.

- **Q**: Why use Gitaly directly to fetch data from repositories instead of the usual "front door" (HTTPS or SSH)?

  **A**: There are several reasons:

  - Polling cost is higher for the "front door". For GitLab.com, HTTPS traffic access via the "front door" goes via CloudFlare, HAProxy, and [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse/). For SSH traffic - CloudFlare and GitLab Shell. Only then a request reaches Gitaly. Using the "front door" would mean more traffic for the infrastructure to handle. More load means higher running cost because more hardware/VMs and bandwidth is needed.

  - Accessing Gitaly via gRPC is a better development experience vs invoking Git for repository access. It's essentially the difference between calling a function vs invoking a binary. Better development experience means faster iterations.

- **Q**: If there is no middlemen between `kas` and Gitaly, how do we prevent `kas` overloading Gitaly?

  **A**: `kas` should rate limit itself. The following not mutually exclusive approaches are possible:

  - Per-`kas` instance rate limiting can be put in place using gRPC's rate limiting mechanisms.
  - Per-`agentk` rate limiting within `kas` can be implemented using some external state storage. E.g. [using Redis](https://redislabs.com/redis-best-practices/basic-rate-limiting/). Or, alternatively, each `kas` instance can enforce per-`agentk` limit just for itself - not as good, but much simpler to implement (no Redis needed).
  - Global rate limiting for all `kas` instances can also be implemented using Redis.

- **Q**: Should `kas` be part of GitLab Workhorse?

  **A**: See https://gitlab.com/gitlab-org/gitlab/-/issues/232064