text:
Instead of uploading the dropped files to an URL, this subclass of [Dropzone.js](http://www.dropzonejs.com/) publishes them to [IPFS](https://ipfs.io/) with [js-ipfs](https://github.com/ipfs/js-ipfs) (no running local nodes needed).

Both [ipfs](https://www.npmjs.com/package/ipfs) and [dropzone](https://www.npmjs.com/package/dropzone) are peer dependencies.

## Usage

```js
const IPFSDropzone = require('ipfs-dropzone')

let dz = IPFSDropzone('#files', {
  /*
    the name of the repo which will store the IPFS blocks in the browser.
    this name will be used by IPFS to create the IndexedDB databases.

    defaults to "ipfs-dropzone"
  */
  ipfsRepoName: 'filemap.xyz',

  /*
    ipfsPath is a function that takes the dropzone file object
    and returns a string that will be used as the path when calling
    https://github.com/ipfs/interface-ipfs-core/blob/master/SPEC/FILES.md#add

    defaults to (file) => file.name
  */
  ipfsPath: file => file.name

  // all other options you'll normally pass to dropzone go here.
})

dz.on('success', file => {
  /*
    everywhere the normal dropzone file object will now have a
    property called "ipfs" which contains the result to the call to
    https://github.com/ipfs/interface-ipfs-core/blob/master/SPEC/FILES.md#add
  */

  console.log('file published to ipfs: ' + file.ipfs[0].hash)

  /*
    your dropzone object will also have the property "node", which
    resolves to the IPFS node the dropzone object is using.
    it is a Promise because the IPFS node is only initialized at the
    moment of the first file upload.
  */
  dz.node.then(ipfs => {
    ipfs.files.cat(file.hash, (err, file) => {
      console.log(file.toString('utf-8'))
    })
  })
})
```

## Information

  * Most normal Dropzone features and events should work, but I'm not sure.
  * There's no way to cancel the process of adding a file.
  * I don't know how to publish this package in a way all JS transpilers and bundlers out there can understand. Please help me.
  * Please read the source if something is unclear. The source is really small.

## What is the LICENSE?

The MIT License.

repository: fiatjaf/ipfs-dropzone

repo_name: fiatjaf/ipfs-dropzone


text:
# Frequently Asked Questions (FAQs)

## Questions about the Flatland Challenge:
These are the most common questions regarding the [Flatland Challenge](https://www.aicrowd.com/challenges/flatland-challenge).
If your questions are not answered please check the [Forum](https://discourse.aicrowd.com/c/flatland-challenge?_ga=2.33753761.1627822449.1571622829-1432296534.1549103074) and post your question there.

### How can I win prizes in this challenge?
You can win prizes in different categories.

Best Solution Prize: Won by the participants with the best performing submission on our test set. Only your rankings from the Round 1 and Round 2 are taken into account. Check the leader board on this site regularly for the latest information on your ranking.

The top three submissions in this category will be awarded the following cash prizes (in Swiss Francs):

CHF 7’500.- for first prize

CHF 5’000.- for second prize

CHF 2’500.- for third prize

Community Contributions Prize: Awarded to the person/group who makes the biggest contribution to the community - done through generating new observations and sharing them with the community.

The top submission in this category will be awarded the following cash prize (in Swiss Francs): CHF 5’000.-

In addition, we will hand-pick and award up to five (5) travel grants to the Applied Machine Learning Days 2019 in Lausanne, Switzerland. Participants with promising solutions may be invited to present their solutions at SBB in Bern, Switzerland.

To check your eligibility please read the prizes section in the [rules](https://www.aicrowd.com/challenges/flatland-challenge/challenge_rules/68).

### What are the deadlines for the flatland challenge?
- The beta round starts on the 1st of July 2019 and ends on the 30th of July 2019
- Round 1 closed on Sunday, 13th of October 2019, 12 PM. UTC +1
- Round 2 closes on Sunday, 5th of January 2020, 12 PM. UTC +1

### How is the score of a submission computed?
The scores of your submission are computed as follows:

1. Mean number of agents done, in other words how many agents reached their target in time.
2. Mean reward is just the mean of the cummulated reward.
3. If multiple participants have the same number of done agents we compute a "nomralized" reward as follows:
```
normalized_reward =cumulative_reward / (self.env._max_episode_steps +self.env.get_num_agents()
```
The mean number of agents done is the primary score value, only when it is tied to we use the "normalized" reward to determine the position on the leaderboard.

### How do I submit to the Flatland Challenge?
Follow the instructions in the [starter kit](https://github.com/AIcrowd/flatland-challenge-starter-kit) to get your first submission.

### Can I use env variables with my controller?
Yes you can. You can access all environment variables as you please. We recommend you use a custom observation builder to do so as explained [here](http://flatland-rl-docs.s3-website.eu-central-1.amazonaws.com/03_tutorials.html#custom-observations-and-custom-predictors-tutorial).

### What are the time limits for my submission?
If there is no action on the server for 10 minutes the submission will be cancelled and a time-out error wil be produced.

If the submissions in total takes longer than 8 hours a time-out will occur.

### What are the parameters for the environments for the submission scoring?
The environments vary in size and number of agents as well as malfunction parameters. The upper limit of these variables for submissions are:
- `(x_dim, y_dim) <= (150, 150)`
- `n_agents <= 250` (this might be updated)
- `malfunction rates` this is currently being refactored

## Questions about the Flatland Repository:
This section provides you with information about the most common questions around the Flatland repository. If your question is still not answered either reach out to the contacts listed on the repository directly or open an issue by following these [guidlines](http://flatland-rl-docs.s3-website.eu-central-1.amazonaws.com/06_contributing.html).
### How can I get started with Flatland?
Install Flatland by running `pip install -U flatland-rl` or directly from source by cloning the flatland repository and running `python setup.py --install` in the repository directory.

These [Tutorials](http://flatland-rl-docs.s3-website.eu-central-1.amazonaws.com/03_tutorials.html) help you get a basic understanding of the flatland environment.
### How do I train agents on Flatland?
Once you have installed Flatland, head over to the [baselines repository](https://gitlab.aicrowd.com/flatland/baselines) to see how you can train your own reinforcement learning agent on Flatland.

Check out this [tutorial](https://gitlab.aicrowd.com/flatland/baselines/blob/master/torch_training/Getting_Started_Training.md?_ga=2.193077805.1627822449.1571622829-1432296534.1549103074) to get a sense of how it works.

### What is a observation builder and which should I use?
Observation builders give you the possibility to generate custom observations for your controller (reinfocement learning agent, optimization algorithm,...). The observation builder has access to all environment data and can perform any operations on them as long as they are not changed.
This [tutorial](http://flatland-rl-docs.s3-website.eu-central-1.amazonaws.com/03_tutorials.html#custom-observations-and-custom-predictors-tutorial) will give you a sense on how to use them.
### What is a predictor and which one should I use?
Because railway traffic is limited to rails, many decisions that you have to take need to consider future situations and detect upcoming conflicts ahead of time. Therefore, flatland provides the possibility of predictors that predict where agents will be in the future. We provide a stock predictor that assumes each agent just travels along its shortest path.
You can build more elaborate predictors and use them as part of your observation builder. You find more information [here](http://flatland-rl-docs.s3-website.eu-central-1.amazonaws.com/03_tutorials.html#custom-observations-and-custom-predictors-tutorial).
### What information is available about each agent?
Each agent is an object and contains the following information:

- `initial_position = attrib(type=Tuple[int, int])`: The initial position of an agent. This is where the agent will enter the environment. It is the start of the agent journey.
- `position = attrib(default=None, type=Optional[Tuple[int, int]])`: This is the actual position of the agent. It is updated every step of the environment. Before the agent has entered the environment and after it leaves the environment it is set to `None`
- `direction = attrib(type=Grid4TransitionsEnum)`: This is the direction an agent is facing. The values for directions are `North:0`, `East:1`, `South:2` and `West:3`.
- `target = attrib(type=Tuple[int, int])`: This is the target position the agent has to find and reach. Once the agent reaches this position its taks is done.
- `moving = attrib(default=False, type=bool)`: Because agents can have malfunctions or be stopped because their path is blocked we store the current state of an agent. If `agent.moving == True` the agent is currently advancing. If it is `False` the agent is either blocked or broken.
- `speed_data = attrib(default=Factory(lambda: dict({'position_fraction': 0.0, 'speed': 1.0, 'transition_action_on_cellexit': 0})))`: This contains all the relevant information about the speed of an agent:
    - The attribute `'position_fraction'` indicates how far the agent has advanced within the cell. As soon as this value becomes larger than `1` the agent advances to the next cell as defined by `'transition_action_on_cellexit'`.
    - The attribute `'speed''` defines the travel speed of an agent. It can be any fraction smaller than 1.
    - The attribute `'transition_action_on_cellexit'` contains the information about the action that will be performed at the exit of the cell. Due to speeds smaller than 1. agents have to take several steps within a cell. We however only allow an action to be chosen at cell entry.
- `malfunction_data = attrib(default=Factory(lambda: dict({'malfunction': 0, 'malfunction_rate': 0, 'next_malfunction': 0, 'nr_malfunctions': 0,'moving_before_malfunction': False})))`: Contains all information relevant for agent malfunctions:
    - The attribute `'malfunction` indicates if the agent is currently broken. If the value is larger than `0` the agent is broken. The integer value represents the number of `env.step()` calls the agent will still be broken.
    - The attribute `'next_malfunction'` will be REMOVED as it serves no purpose anymore, malfunctions are now generated by a poisson process.
    - The attribute `'nr_malfunctions'` is a counter that keeps track of the number of malfunctions a specific agent has had.
    - The attribute `'moving_before_malfunction'` is an internal parameter used to restart agents that were moving automatically after the malfunction is fixed.
- `status = attrib(default=RailAgentStatus.READY_TO_DEPART, type=RailAgentStatus)`: The status of the agent explains what the agent is currently doing. It can be in either one of these states:
    - `READY_TO_DEPART` not in grid yet (position is None)
    - `ACTIVE` in grid (position is not None), not done
    - `DONE` in grid (position is not None), but done
    - `DONE_REMOVED` removed from grid (position is None)

### Can I use my own reward function?
Yes you can do reward shaping as you please. All information can be accessed directly in the env.
### What are rail and schedule generators?
To generate environments for Flatland you need to provide a railway infrastructure (rail) and a set of tasks for each agent to complete (schedule).
### What is the max number of timesteps per episode?
The maximum number of timesteps is `max_time_steps = 4 * 2 * (env.width + env.height + 20)`
### What are malfunctions and what can i do to resolve them?
Malfunctions occur according to a Poisson process. The hinder an agent from performing its actions and update its position. While an agent is malfunctioning it is blocking the paths for other agents. There is nothing you can do to fix an agent, it will get fixed automatically as soon as `agent.malfunction_data['malfunction'] == 0` .
You can however adjust the other agent actions to avoid delay propagation within the railway network and keeping traffic as smooth as possible.

### Can agents communication with each other?
There is no communitcation layer built into Flatland directly. You can however build a communication layer outside of the Flatland environment if necessary.

## Questions about bugs
### Why are my trains drawn outside of the rails?
If you render your environment and the agents appear to be off the rail it is usually due to changes in the railway infrastructure. Make sure that you reset your renderer anytime the infrastructure changes by calling `env_renderer.reset().
`

license: MIT

branch_name: refs/heads/master

extension: md

gha_language: Python

gha_license_id: MIT

license_type: permissive


text:
A constant is a container containing a [value](value "wikilink"). This
value cannot be changed apart from initialization, hence the name
constant, as opposed to a [variable](variable "wikilink") of which the
value can vary. A constant can be of any [datatype](datatype "wikilink")
and can contain a value according to its datatype. For example, an
integer has whole numbers between -2\^31 (-2147483648) and +2\^31-1
(2147483647) (e.g. 23) , a float has a floating point number (e.g.
2.674) and a string has a series of characters (e.g. "Hello World!").

There is a number of [predefined
constants](predefined_constants "wikilink"). These variables are
predefined, which means they exist without the programmer making them.
They can be useful in a some [functions](functions "wikilink") or when
assigning them to some [global variables](global_variable "wikilink").

See Also
--------

-   [A list of predefined constants](constants "wikilink")
-   [Variable](Variable "wikilink")

<Category:General>

repository: DivHub/pixtudio-website

repo_name: DivHub/pixtudio-website


text:
[![Stars](https://img.shields.io/github/stars/laminlabs/readfcs?logo=GitHub&color=yellow)](https://github.com/laminlabs/readfcs)
[![codecov](https://codecov.io/gh/laminlabs/readfcs/branch/main/graph/badge.svg?token=6A5PYRX809)](https://codecov.io/gh/laminlabs/readfcs)
[![pypi](https://img.shields.io/pypi/v/readfcs?color=blue&label=pypi%20package)](https://pypi.org/project/readfcs)
[![doi](https://img.shields.io/badge/doi-10.56528%2Frfcs-lightgrey)](https://doi.org/10.56528/rfcs)

# readfcs: Read FCS files

_A lightweight open-source Python package that loads data and metadata from Flow Cytometry Standard (FCS) files into DataFrame and AnnData objects._

Read the [docs](https://lamin.ai/docs/readfcs), [report](https://lamin.ai/reports/2022/readfcs).

repository: laminlabs/readfcs

branch_name: refs/heads/main

repo_name: laminlabs/readfcs


text:
## Project Name
Ping Pong!

## Features
This is a page where a user inputs a number and the browser will count up to that number and insert ping, pong, and pingpong along the way.

![screenshot of project](pingpong.png)

## Specifications
For any number inputted, the program will count up to that number. For numbers divisible by 3, 5, and 15, the program will output ping, pong, and pingpong respectively.
For example:

1. For any number not divisible by 3, 5, and 15, print out the number:
  - Input: 2
  - Output: 1, 2

2. For numbers divisible by 3, output ping.
  - Input: 3
  - Output: 1, 2, ping

3. For numbers divisible by 5, output pong.
  - Input: 5
  - Output: 1, 2, ping, 4, pong

4. For numbers divisible by 15, output pingpong.
 - Input: 15
 - Output: 1, 2, ping, 4, pong, ping, 7, 8, ping, pong, 11, ping, 13, 14, pingpong

## Technologies
JavaScript, JQuery, BootStrap, HTML and CSS

## Usage
To view the site, you can go to https://mehequanna.github.io/epicodus-project3.

To use the code, you can clone the repository at [https://github.com/Mehequanna/epicodus-project3](https://github.com/Mehequanna/epicodus-project3).

1. Download the project.
2. Open index.html in your browser.
3. Have fun!

## Author
Stephen Emery

## License
This work can be used under the MIT License.
Copyright (c) 2016 Stephen Emery


text:
FreeFlix-Media-Hub & CoCo-POST-Chain Clients
==

The Cosmos IBC ecosystem is vibrant with many developers & projects working with the various modules, and at the moment IBC (mostly due to Cosmos GOZ & in general).

In our case too, team FreeFlix Media has worked on a utility that will continue to use Cosmos IBC at a later point in time.

Owing to the fact that most people that are aware of Cosmos IBC are developers (atleast at this point in time) & after receiving a low turn out for interaction with https://goz.freeflix.media, we have released 2 clients ```ffcli``` for **freeflix-media-hub** and ```cococli``` for **coco-post-chain**, along with an interactive script that takes a few inputs inputs from you at the beginning of the script and runs demonstrates interactions while you sit back and enjoy!

The clients are being released for 2 operating systems:

1) GNU/Linux (Ubuntu & Debian)
2) Mac OS

-----

### 1) GNU/Linux (Ubuntu & Debian)

#### Dependencies

1) **jq**

    ``` sudo apt-get install jq ```

2) **curl**

    ``` sudo apt-get install curl ```

#### Run the script

- Download the binaries:

  ```wget https://github.com/FreeFlixMedia/ibc/releases/download/v0.0.1-ibc-demo/ff_coco_script_linux_CLI.tar.xz```

- Extract the file to a folder and navigate into it & change permissions of the script file

    ```tar -xJvf ff_coco_script_linux_CLI.tar.xz```

- Give required permissions for the bash script to execute

    `sudo chmod 744 FF_COCO_IBC_demo.sh`

- Run the script file

  `bash ./FF_COCO_IBC_demo.sh`

NOTE: If you have any issues running this, please raise an issue [OR] write to info[at]freeflix[dot]media

---------

### 2) Mac OS

#### Dependencies

1) **jq**

    ``` brew install jq ```

2) **curl**

    ``` brew install curl ```

#### Run the script

- Download the binaries:

  ```wget https://github.com/FreeFlixMedia/ibc/releases/download/v0.0.1-ibc-demo/ff_coco_script_mac_CLI.tar.xz```

- Extract the file to a folder and navigate into it & change permissions of the script file

    ```tar -xJvf ff_coco_script_mac_CLI.tar.xz```

- Give required permissions for the bash script to execute

    `sudo chmod 744 FF_COCO_IBC_demo.sh`

- Run the script file

  `bash ./FF_COCO_IBC_demo.sh`

NOTE: If you have any issues running this, please raise an issue [OR] write to info[at]freeflix[dot]media

Important points to note:
---

0. Please install dependencies and make sure you have downloaded the binaries from the [releases page here](https://github.com/FreeFlixMedia/ibc/releases/tag/v0.0.1-ibc-demo).

1. The demo is the creation & exchange of assets between a **creator** (on FreeFlix Media Hub) and a **licensee** (on CoCo POST Chain).

2. The demo will ask you for your *Twitter handle* & *tweet IDs*. You do not need to enter your actual Tweet handle for this script to run. You can enter any set of characters/random string. Make sure it is unique enough & doesn't match with an existing asset_ID on the chain.

3. Once the above is entered and you'll be asked for a key to create accounts and that's it! You can sit back and drink coffee/even have pop-corn while the demo runs you through each of the activities on FreeFlix Media Hub and COCO POST Chain

What does the script demonstrate?
---

The script aims to demonstrate:

1. Asset creation WITHOUT licensing (pNFT #1) on **freeflix-media-hub**
2. Asset creation WITH licensing (pNFT #2) on **freeflix-media-hub** for licensing it to **coco-post-chain**. The NFT exists on both chains.
3. Asset creation WITH licensing with an automatic attribution to the owner

Licensing of an Asset listed on the marketplace
---

![Asset Licesing](https://raw.githubusercontent.com/FreeFlixMedia/ibc/master/goz/media/FreeFlixMedia_CosmicCompass_CosmosGOZ_ListedAssetExchangeFROM_FF_to_COCO.png)

Licensing of an unlisted Asset with automatic attribution to the asset owner
---

![Asset Licesing](https://raw.githubusercontent.com/FreeFlixMedia/ibc/master/goz/media/FreeFlixMedia_CosmicCompass_CosmosGOZ_UnlistedAssetExchange_FROM_COCO_2_FreeFlix_and_back.png)

A quick walk-through
---

The freeflix-media-hub is powered by the fungible token FFMT while the coco-post-chain is powered by the COCO token.

This is a 25 step journey that the script will take you through and we have condensed it into 8 steps.

Below are the events triggered during the demo:

1) An account is created on the freeflix-media-hub & queried for FFMT tokens. This account supports both FFMT tokens and COCO tokens
    - FF account Token Balance : 1000 FFMT || 0 COCO
    - 0 pNFTs

2) Asset #1 is created on the freeflix-media-hub
    - A pre-defined fee of 50 FFMT is utilized for the tx
    - FF account Token Balance : 950 FFMT || 0 COCO
    - 1 pNFT

3) Asset #2 is created on the freeflix-media-hub for licensing by a user on the coco-post-chain
    - Token Balance : 900 FFMT || 0 COCO
    - 2 pNFTs

4) An account is created on the coco-post-chain & queried for COCO tokens. This account supports both FFMT tokens and COCO tokens
    - COCO account Token Balance : 0 FFMT || 1000 COCO
    - 0 sNFTs

5) A user on the coco-post-chain pays **100 COCO token** and takes Asset #2 for licensing with a revenue share.
    - A pre-defined fee of 50 COCO is utilized for the tx
    - COCO account Token Balance : 0 FFMT || 850 COCO
    - 1 sNFT

6) A user on the coco-post-chain now creates an sNFT Asset #3 & pays 100 COCO for it. The notable point here is, this asset is yet to be created on the freeflix-media-hub. This is a demo of asset attribution, without the creator having to list all their assets.
    - A pre-defined fee of 50 COCO is utilized for the tx
    - COCO account Token Balance : 0 FFMT || 700 COCO
    - 2 sNFTs

7) Final balance for the account on freeflix-media-hub
    - FF account Token Balance : 900 FFMT || 200 COCO
    - NFT token balance : 3 pNFTs
        - 1 with licensing as FALSE
        - 2 with licensing as TRUE

8) Final Balance for the account on coco-post-chain
    - Token Balance : 0 FFMT || 700 COCO
    - NFT token balance : 2 sNFTs
        - both that have been licensed from the account on freeflix-media-hub

The entire process takes a few minutes to execute once you initiate it with the script. If the process is too fast you can wait for the script to end and scroll up to check everything that has happened.

To know more about the details of implementation, please visit our medium articles ([article #1](https://medium.com/freeflix/community-powered-linear-broadcast-using-cosmos-ibc-by-freeflix-media-cosmic-compass-coco-bfb00b4584b9) & [article #2](https://medium.com/freeflix/community-powered-linear-broadcast-using-cosmos-ibc-by-freeflix-media-cosmic-compass-coco-bfb00b4584b9)) for further reading.

repository: OmniFlix/ibc

repo_name: OmniFlix/ibc


text:
# Nanofix
  Zero-cost low-latency FIX message libraries in Rust

    Documentation
   |
    Website

## Overview
nanofix is a collection of tools written in rust that work with the [FIX Protocol](https://www.fixtrading.org/).
The tools range from a FIX message viewer, TBA

While there exists other rust based FIX crates there was a need to have access to lower level constructs to
build a suite of tools to work with the FIX protocol.

## Project Layout
The root project is intended for `nanofix-rs` library developers only.
Consumers should depend on the sub crates.

Sub crates:

* [`fixv`]: A command line FIX message log parser and viewer

## License

## TODO
* add functionality to filter (from and/or to) by MsgSeqNum
* add functionality to filter (from and/or to) by SendingTime
* amy other filters ??
* support config _(~/.fixv)_ to explicit set the different colors
* add support for excluding multi-byte MsgTypes

## Links
- [FIX Trading Community](https://www.fixtrading.org/)
- [FIX on Wikipedia](https://en.wikipedia.org/wiki/Financial_Information_eXchange)

### Downloading Programs
For example, to download the FIX Viewer program, type the following command:

```curl -L -o fixv.zip https://github.com/subnano/nanofix-rs/releases/download/0.1.1/fixv.zip && unzip fixv.zip -d ~/bin && rm fixv.zip```

repository: subnano/nanofix-rs

repo_name: subnano/nanofix-rs


text:
# Docker Image

![Docker](https://img.shields.io/github/actions/workflow/status/starudream/docker-image/docker.yml?label=docker&style=for-the-badge)
![License](https://img.shields.io/github/license/starudream/docker-image?style=for-the-badge)

## License

[Apache License 2.0](./LICENSE)

repository: starudream/docker-image

gha_language: Dockerfile

repo_name: starudream/docker-image


text:
# Heppia despreauxii (Mont.) Tuck. SPECIES

#### Status
ACCEPTED

#### According to
The Catalogue of Life, 3rd January 2011

#### Published in
Geneac. Lich. 46 (1872)

#### Original name
Solorina despreauxii Mont.

### Remarks
null

repository: mdoering/backbone

repo_name: mdoering/backbone


text:
## Fields

The operators `.` and `:` access public fields of
[data abstractions](../statements/#data),
[code abstractions](../statements/#code),
and
[native](../statements/#c-integration) structs:

```
Dot   ::= Loc `.´ (ID_int|ID_nat)
Colon ::= Loc `:´ (ID_int|ID_nat)
```

The expression `e:f` is a sugar for `(*e).f`.

`TODO: ID_nat to avoid clashing with Céu keywords.`

<!--
Example:

```
native do
    typedef struct {
        int v;
    } mystruct;
end
var _mystruct s;
var _mystruct* p = &s;
s.v = 1;
p:v = 0;
```
-->

repository: ceremcem/ceu

gha_license_id: NOASSERTION

repo_name: ceremcem/ceu


text:
---
layout: post
title: "Convert .ARW to jpeg and reduce size in a folder"
category: "memo"
date: 2018-05-31 10:09:57
tags:
- osx
---

# Convert all the .ARW images in a specific folder

```
cd path/to/images/folder
for i in *.ARW; do sips -s format jpeg -s formatOptions 70 "${i}" --out "${i%ARW}jpeg"; done
```

# Reduce all images

```
sips -Z 640 *.jpeg
```

# sources

- [Use sips to quickly, easily—and freely—convert image files](https://robservatory.com/use-sips-to-quickly-easily-and-freely-convert-image-files/)
- [Batch Resize Images Quickly in the OS X Terminal](https://lifehacker.com/5962420/batch-resize-images-quickly-in-the-os-x-terminal)

gha_language: CSS


text:
Overview
======
This little project was built to answare our need for a simple image proxy solution.
It is very light, using GD/iMagick through the image intervention library by Oliver Vogel.

Deployment
======
![alt text](https://github.com/goroomer/image-proxy/raw/master/src/docs/diagram.png "Solution implementation")

Performance
======
Response time
------
A 3MB file request:
- Check for cached version
- Get source CloudFront
- Fit: 350x350px
- Our CloudFront
- User
the average results under 500ms (95% of which is getting the image from source)

![alt text](https://github.com/goroomer/image-proxy/raw/master/src/docs/avg_latency.PNG "Response time(ms)")

CPU Load
------
With 3 t2.nano instances we utilize less then 10% cpu for 350K requests per day

Usage
======
     https://static.example.com/param_string/source_url
param_string
------
| Function       | String       |
| ------------- |:-------------:|
| Fit the original image to the given size     | fit:{width}x{height} |
| Set width      | w:{width}    |
| Set width & keep aspect ratio      | w:{width}c    |
| Set height      | h:{height}    |
| Set height & keep aspect ratio     | h:{height}c   |
| Crop  to given size (centered)    | c:{width}x{height}    |
| Set quality   | q:{percent}   |

Examples
------
Best fit to a 250x250 box, 100% quality

    https://static.example.com/fit:250x250,q:100/source_url

Crop to 250x250 box, 50% quality

    https://static.example.com/c:250x250,q:50/source_url

Set width to 250px and keep the height as is

    https://static.example.com/w:250/source_url

Set height to 250px and keep aspect ratio

    https://static.example.com/h:250c/source_url

Installation
======

The installation is really straight forward, you can use chef / docker if you know how.
or skip the overhead and deploy a plain installation with ec2 images and basic bash scripts

Ngnix configuration example:
------
```
server {
        listen 80 default_server;
        listen [::]:80 default_server;

        root /home/ubuntu/image-proxy/src;

        # Add index.php to the list if you are using PHP
        index index.php index.html index.htm index.nginx-debian.html;

        server_name _;

        location / {
                index index.php;
                rewrite ^/check$ / last;
                rewrite ^/(.*)/(.*\:.*) /?source_url=$2&params=$1 last;
        }
        location ~ \.php$ {
                include snippets/fastcgi-php.conf;
                fastcgi_pass unix:/run/php/php7.3-fpm.sock;
        }

}
```

Deploy From Comunity AMI:
------
    ami-aff007b9

Create a redundant AWS auto scaling deployment
------
- Create VPC with at lease 2 availability zones (for redundancy)
- Create launch configurations for the AMI(ami-aff007b9) and VPC
- Create an Auto Scaling Group for the Launch Configuration
- Create Scaling policies
- Create Target Group under LOAD BALANCING in the EC2 tab (set health check to path: /check )
- Create a Load Balancer with the Target Group
- Set your DNS for static.yourdoamin.com to the Load Balancer CNAME record
- Test your setup go to: static.yourdomain.com/fit:500x500,q:90/https://static.pexels.com/photos/211929/pexels-photo-211929.jpeg
- OPTIONAL: You can add a CDN to this setup: create a CloudFront distribution connected to your load balancer, and set you DNS to the CloudFront distribution CNAME

TODO
======
- Create an installation script
- Create sample nginx & php conf files
- Add some security features to the code
- Add Redis cache adapter

Thanks
======
- Oliver Vogel, for his intervention/image library http://image.intervention.io/

repository: goroomer/image-proxy

gha_language: PHP

repo_name: goroomer/image-proxy


text:
# HexagonalJsonApi

[![CircleCI](https://circleci.com/gh/subodhkhanduri1/hexagonal_json_api/tree/master.svg?style=shield)](https://circleci.com/gh/subodhkhanduri1/hexagonal_json_api/tree/master)

This is a very simple pattern for modeling Rails JSON API requests in different layers based on their different responsibilities:
* Use Case Services - Each service `executes` a single Controller action's work
* Validators - Services use one or more Validator objects to validate/authorize the request
* Presenters - Formats the result data returned by the Service execution
* Data Objects - Services encapsulate the result data in a single data object which can pass through the narrow interface between the Service and the Presenter.
* Responses - Encapsulates the final response to give back to the Controller

The app's business logic can be further abstracted out into logic classes that are invoked by the Services.

### Benefits:
* Each layer is segregated and can be tested independently.
* Narrow, Hardened interfaces between the different layers.
* Service and Presenter combinations can be changed easily.

## Installation

Add this line to your application's Gemfile:

```ruby
gem 'hexagonal_json_api'
```

And then execute:

    $ bundle

Or install it yourself as:

    $ gem install hexagonal_json_api

## Usage

TODO: Write usage instructions here

## Development

After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).

## Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/hexagonal_json_api. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.

## License

The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).

gha_language: Ruby


text:
Between 2008 and 2012,
archeologists excavated the rubble
of an ancient hospital in England.
In the process, they uncovered
a number of skeletons.
One in particular belonged
to a wealthy male
who lived in the 11th or 12th century
and died of leprosy between
the ages of 18 and 25.
How do we know all this?
Simply by examining some old,
soil-caked bones?
Even centuries after death,
skeletons carry unique features
that tell us about their identities.
And using modern tools and techniques,
we can read those features as clues.
This is a branch of science known as
biological anthropology.
It allows researchers to piece together
details about ancient individuals
and identify historical events
that affected whole populations.
When researchers uncover a skeleton,
some of the first clues they gather,
like age and gender,
lie in its morphology,
which is the structure, appearance,
and size of a skeleton.
Bones, like the clavicle,
stop growing at age 25,
so a skeleton with a clavicle
that hasn't fully formed
must be younger than that.
Similarly, the plates in the cranium
can continue fusing up to age 40,
and sometimes beyond.
By combining these with some
microscopic skeletal clues,
physical anthropologists can estimate
an approximate age of death.
Meanwhile, pelvic bones reveal gender.
Biologically, female pelvises are wider,
allowing women to give birth,
where as males are narrower.
Bones also betray the signs
of ancient disease.
Disorders like anemia leave their traces
on the bones.
And the condition of teeth
can reveal clues
to factors like diet and malnutrition,
which sometimes correlate with wealth
or poverty.
A protein called collagen can give us
even more profound details.
The air we breathe,
water we drink,
and food we eat
leaves permanent traces
in our bones and teeth
in the form of chemical compounds.
These compounds contain measurable
quantities called isotopes.
Stable isotopes in bone collagen
and tooth enamel varies among mammals
dependent on where they lived
and what they ate.
So by analyzing these isotopes,
we can draw direct inferences regarding
the diet and location of historic people.
Not only that, but during life,
bones undergo a constant cycle
of remodeling.
So if someone moves from one place
to another,
bones synthesized after that move
will also reflect the new isotopic
signatures of the surrounding environment.
That means that skeletons can be used
like migratory maps.
For instance, between 1-650 AD,
the great city of Teotihuacan in Mexico
bustled with thousands of people.
Researchers examined the isotope ratios
in skeletons' tooth enamel,
which held details of their diets
when they were young.
They found evidence for significant
migration into the city.
A majority of the individuals
were born elsewhere.
With further geological
and skeletal analysis,
they may be able to map where
those people came from.
That work in Teotihuacan is also
an example of how bio-anthropologists
study skeletons in cemeteries
and mass graves,
then analyze their similarities
and differences.
From that information, they can learn
about cultural beliefs,
social norms,
wars,
and what caused their deaths.
Today, we use these tools to answer
big questions about how forces,
like migration and disease,
shape the modern world.
DNA analysis is even possible in some
relatively well-preserved ancient remains.
That's helping us understand how diseases
like tuberculosis
have evolved over the centuries
so we can build better treatments
for people today.
Ancient skeletons can tell us a
surprisingly great deal about the past.
So if your remains are someday
buried intact,
what might archeologists
of the distant future learn from them?

repository: prewittdavon/TEDVis

repo_name: prewittdavon/TEDVis


text:
# Medieval life simulator
---
Feel free to add anything you want. Just expand this village by creating new buildings, characters etc. and have fun!
Please write script using p5.js library. You can read more [here](https://p5js.org/reference/).

All pull request probably will be added.

## Running
Check your bowser and disable adblocker and mining blocker plugin. If you have any troubles with running script, choose one of this solutions:
* Use local server (eg. XAMPP)
* Use Firefox
* Follow [this](https://github.com/processing/p5.js/wiki/Local-server) instructions

repository: ravecampus/Medieval-life-simulator

repo_name: ravecampus/Medieval-life-simulator


text:
# Python Set Up

In this guide, we will need to run some python scripts in some of the steps.

## Installation

First of all, if we do not have python, we need to install it in our computer. This can be done from the [official website](https://www.python.org/downloads/). The installation should also download pip on your computer.
For Linux computers, python is installed by default. However, pip is not installed by default. You can check if they have been downloaded in your machine by running the following commands in the terminal:

```bash
python3 --version
pip3 --version
```

To install pip in Linux run the following commands:

```bash
sudo apt-get update
sudo apt-get -y install python3-pip
```

If the command above doesn’t work run:

```bash
curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"
python3 get-pip.py --user
```

After the installation, I recommend using Visual Studio Code as an editor. Download it from the [official website](https://code.visualstudio.com/) as well.

## Setting the Environment

1. Open Visual Studio Code, go to the extension tab in the left panel and search python in the search bar. Install the extension.

    ![python extension](./media/python_extension.png)

1. Create a new folder AzureFarmBeats to store all the python scripts. Go to the files tab and click on Open Folder. Open the folder you just created.

    ![files tab](./media/files_tab.png)

1. Open a terminal in Visual Studio Code by clicking terminal at the top and then new terminal.

    ![new terminal](./media/new_terminal.png)

Now we will create a virtual environment in python to store all the python packages.

### Linux/Mac

1. Run the following command to install python virtual environments:

    ```bash
    python3 -m pip install --user virtualenv
    ```

2. Create a new virtual environment use the command below:

    ```bash
    python3 -m venv .venv
    ```

    This will create a folder named `.venv` in your current workspace.

3. Close the terminal and open a new one to activate the environment. Alternatively, you can use the following command to do so:

    ```bash
    source ./.venv/bin/activate
    ```

4. Create a new file named `requirements.txt` with the following content:

    ```txt
    msal
    requests
    azure-eventhub
    asyncio
    ```

5. Use the following command to install the libraries in the `requirements.txt` file:

    ```bash
    pip3 install -r requirements.txt
    ```

### Windows

1. Run the following command to install python virtual environments:

    ```bash
    py -m pip install --user virtualenv
    ```

2. Create a new virtual environment use the command below:

    ```bash
    py -m venv .venv
    ```

    This will create a folder named `.venv` in your current workspace.

3. Close the terminal and open a new one to activate the environment. Alternatively, you can use the following command to do so:

    ```bash
    .\env\Scripts\activate
    ```

4. Create a new file named `requirements.txt` with the following content:

    ```txt
    msal
    requests
    azure-eventhub
    asyncio
    ```

5. Use the following command to install the libraries in the `requirements.txt` file:

    ```bash
    pip3 install -r requirements.txt
    ```

## Running scripts

To run the python scripts, use the commands below:

- On macOS and Linux:

    ```bash
    python nameOfFile.py
    ```

- On Windows:

    ```cmd
    py nameOfFile.py
    ```

    Where nameOfFile is the name of the script you want to run.

-------
Go to next step: [Create Telemetry Client](./Create_telemetry_client.md)

repository: microsoft/FarmBeats-UniversityCommunity

repo_name: microsoft/FarmBeats-UniversityCommunity


text:
---
description: How to display validation errors in your UI.
---

# Displaying Errors

The validation controller maintains the active list of validation results which can be iterated to display the errors in UI.

```markup
<ul>
  <li repeat.for="result of validationController.results">
    <template if.bind="!result.valid">${result}</template>
  </li>
</ul>
```

{% embed url="https://stackblitz.com/edit/au2-validation-display-errors?ctl=1" caption="" %}

There are also some out-of-the-box components that can be used to display the errors. These are discussed in the following sections.

## `validation-errors` custom attribute

This custom attribute can be used to bind the errors for children the target elements.

```markup
<div validation-errors.from-view="nameErrors"> <!--binds all errors for name to the "nameErrors" property-->
  <input value.bind="person.name & validate">
  <div>
    <span repeat.for="error of nameErrors">${error.result.message}</span>
  </div>
</div>
<div validation-errors.from-view="ageErrors"> <!--binds all errors for age to the "ageErrors" property-->
  <input value.bind="person.age & validate">
  <div>
    <span repeat.for="error of ageErrors">${error.result.message}</span>
  </div>
</div>
```

Note that this in itself does not show any error, unless errors are iterated to be bound with the view. An example can be seen below.

{% embed url="https://stackblitz.com/edit/au2-validation-validation-errors-ca?ctl=1" caption="" %}

A point to note is that multiple validation targets can also be used for a single `validation-errors` custom attribute, and the errors for multiple targets will be captured the same way.

```markup
<div validation-errors.from-view="errors"> <!--binds all errors for name, and age to the "errors" property-->
  <input value.bind="person.name & validate">
  <input value.bind="person.age & validate">
  <div>
    <span repeat.for="error of errors">${error.result.message}</span>
  </div>
</div>
```

The usage of this custom element can be deactivated by using `UseSubscriberCustomAttribute` configuration option.

```typescript
import { ValidationHtmlConfiguration } from '@aurelia/validation-html';
import Aurelia from 'aurelia';

Aurelia
  .register(ValidationHtmlConfiguration.customize((options) => {
    // customization callback
    options.UseSubscriberCustomAttribute = false;
  }))
  .app(component)
  .start();
```

This is useful if you have a custom attribute of the same name, and want to use that over this out-of-the-box custom attribute.

### `validation-container` custom element

The `validation-container`custom element also has similar goal of capturing the validation errors for the children target elements. Additionally, it provides a template to display the errors as well. This helps in reducing the boilerplate created by the `validation-errors` custom attribute. For example, using this custom element, displaying the errors reduces to the following.

```markup
<validation-container>
  <input value.bind="person.name & validate">
</validation-container>
<validation-container>
  <input value.bind="person.age & validate">
</validation-container>
```

{% embed url="https://stackblitz.com/edit/au2-validation-validation-container-ce?ctl=1" caption="" %}

There are couple of important points to note about the examples shown above. The first validation target shown in the example uses the default template of the custom element. This custom element template is based on two `slot`s as shown below.

```markup
<slot>
  <!--meant for validation target-->
</slot>
<slot name='secondary'>
  <!--here goes error-->
</slot>
```

The results of using the default template may not also suite your app or esthetics. However content can be injected into the slot from Light DOM \(in case you are unfamiliar with the concepts, you are encouraged to give this [excellent article](https://developers.google.com/web/fundamentals/web-components/shadowdom#styling) a read\) as shown in the example. Although traditionally the default slot is meant for the validation target\(s\), it can also be used to inject the error template, as shown in the example above.

It is quite understandable that the CSS-containment of the Shadow DOM can come in the way of styling the custom element as per your need. It can be argued that this can be facilitated using CSS variables extensively. However, there is a far easy alternative to reach the same goal is offered by facilitating the customization of the whole template. To this end, use the `SubscriberCustomElementTemplate` configuration option.

{% embed url="https://stackblitz.com/edit/au2-validation-validation-container-ce-custom-template?ctl=1" caption="" %}

There is another aspect of this configuration option. When a `null`, `undefined`, or '' \(empty string\) is used as the value for this configuration option, it deactivates the usage of this custom element. This is in sense similar to the `UseSubscriberCustomAttribute` configuration option.

### `ValidationResultPresenterService`

Unlike the previous two approaches, this is a standalone service that manipulates the DOM directly. That it adds elements to DOM for every new errors and removes elements from DOM that are associated with old errors.

To use this, you need to instantiate it and register it with the validation controller.

```typescript
import { IValidationController, ValidationResultPresenterService } from '@aurelia/validation';

export class MyApp {
  private presenter: ValidationResultPresenterService;

  public constructor(
     @newInstanceForScope(IValidationController) private validationController: IValidationController,
  ) {
      this.presenter = new ValidationResultPresenterService();
      this.validationController.addSubscriber(this.presenter);
  }
}
```

{% embed url="https://stackblitz.com/edit/au2-validation-validation-validationresultpresenterservice?ctl=1" caption="" %}

The error rendering process can be completely overridden in the child classes. The use methods for overriding are described below.

* `add`: this adds a new error to DOM. Override this if you want to completely change the process of adding new errors.
* `remove`: this removes an old error from the DOM. Override this if you want to completely change the process of removing old errors.
* `getValidationMessageContainer`: As the name suggests it provides container element with respect to current target. The default behavior is to look for an element with the attribute `validation-result-container` that is contained by the parent element of the current target element. If there is none found a `div` is created with the attribute and appended to the parent element.
* `showResults`: This is the method that appends the errors to the container. By default a `span` with the error message is added for every errors whereas the valid results are skipped.

To avoid direct DOM manipulation, it is highly encouraged to use the previously mentioned custom attribute, and custom element.

> One commonality across these components is that all these are different implementations of the [`ValidationResultsSubscriber` interface](validation-controller.md#addSubscriber-and-removeSubscriber).

repository: aurelia/aurelia

gha_language: TypeScript

repo_name: aurelia/aurelia


text:
# Marriage Contracts

Smart contract to get marriage on the blockchain!

## Getting Started
Simply download the required packages from the client folder.

```
$ npm install
```

The contracts folder keeps a copy of the contract in Solidity. The migrations folder keeps Truffle deployment files.

### Prerequisites
You must download Truffle in order to compile and migrate updated contracts.

```
$ npm install -g truffle
```

The Ganache GUI was also used during the development process -> Ganache

## Deployment

Prerequisite: [create an Infura Account](https://infura.io/register)

Create an .env file with these two variable INFURA_API_KEY and MNENOMIC

```
$ truffle compile
```

After that you can choice what network (`ropsten`, `kovan`, `rinkeby`, `main`)  you can send the contract

```
$ truffle migrate --network choice
```

gha_language: JavaScript


text:
The `import pcap` command uses [libpcap](https://www.tcpdump.org) to read
network packets from a trace or an interface.

VAST automatically calculates the [Community
ID](https://github.com/corelight/community-id-spec) for PCAPs for better
pivoting support. The extra computation induces an overhead of approximately 15%
of the ingestion rate. The option `--disable-community-id` disables the
computation completely.

The PCAP import format has many additional options that offer a user interface
that should be familiar to users of other tools interacting with PCAPs. To see
a list of all available options, run `vast import pcap help`.

Here's an example that reads from the network interface `en0` cuts off packets
after 65535 bytes.

```bash
sudo vast import pcap --interface=en0 --cutoff=65535
```


text:
# Getting started with Gatsby Cloud and Kontent

Learn how to connect Gatsby Cloud with Kentico Kontent based Gatsby site.

:warning: This page has been moved to official Gatsby documentation here:
* <https://www.gatsbyjs.com/docs/kontent/getting-started/>

repository: SeanDYoko/image-glitch

repo_name: SeanDYoko/image-glitch


text:
## Android Groen Buffel ##

**Connect your Android device with ADB over network in just a one click.**

![Groen Buffel](https://raw.github.com/yuv4ik/android-groen-buffel/master/artwork.png)

### How it works? ###

![How it works](https://raw2.github.com/yuv4ik/android-groen-buffel/master/how%20it%20works.png)

### License ###

Copyright (C) 2014 yuv4ik

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.


text:
The enclosing scripts can do two things:
1. Read a database table and create an XML file in the Prime Data folder.
2. Create a MERGE statement per XML file in preparation to load the data into the database.

#### Running Templates
The .tt templates can run be run by right-clicking on the .tt file and selecting _Run Custom Tool_.

### Create XML Data
In order to create an XML document for prime data follow the following steps.

1. Open Templates\\1 - Create XML File\\SQLConnection.tt and verify the connection settings. If using Windows Authentication ensure GetSecureLogin() = true; the GetSQLUsername() and GetSQLPassword() values are not necessary.
2. Open Templates\\Table Metadata.xml and add a row to the table(s) and add the table(s) to the XML. Ensure that you add the type(prime) and the indexkey (usually the primary key).
3. If running only one table then open the Templates\\1 - Create XML File\\Create XML Data.tt and change the tableName variable to the table that you wish to run. Then run the Create XML Data.tt file.
4. If refreshing the data for all of the tables then run the Templates\\1 - Create XML File\\Generate Prime Data XML.tt.

### Create MERGE Statements
This step will create the Scripts\\Post-Deployment\PrimeDataMerge.sql. In order to create the MERGE statements to deploy the data from XML to the database follow these steps.

1. Open Templates\\2 - Create Merge\\SQLConnection.tt and verify the connection settings. If using Windows Authentication ensure GetSecureLogin() = true; the GetSQLUsername() and GetSQLPassword() values are not necessary.
2. Run the Templates\\2 - Create Merge\\Generate Merge Statements.tt.

repository: bubdm/ReferenceDataTemplates

filename: Template Readme.md

repo_name: bubdm/ReferenceDataTemplates


text:
# udev Rules

In order to access the various USB-connected devices on the car reliably and easily, each can be mapped to a specific endpoint in `/dev` using custom udev rules.

After installing the rules file, the following devices will be symlinked to `/dev/car`:

- `/dev/car/camera` - Logitech BRIO (UVC device)
- `/dev/car/vesc` - VESC (TTY device)
- `/dev/car/razor` - SparkFun Razor IMU (TTY device)

## Installation

Run `rules-install.sh`, or copy the included rules file to `/etc/udev/rules.d/`.
Reboot for the new rules to take effect.

repository: RTSYork/rtscar

repo_name: RTSYork/rtscar


text:
# Jenkins-slave
As jenkins Slave

**Base slave**: inlucde `python3-dev`,`pip3`

- Android slave include: Android SDK / Nodejs / Ionic / Cordova
- Dotnet slave include: .Net Core SDK / powershell
- C/Cpp slave include: build-essential / qmake / automake
- Golang slave inclue: golang tool

license: Unlicense

repository: idoop/jenkins-slave

gha_license_id: Unlicense

repo_name: idoop/jenkins-slave


text:
# javascript-injectorr
Chrome extension for injecting javascript to the webpage you are visiting. It can be used as a handy tool to make your surfing a bit of more fun. I personally use it mainly during the test automation.

## Install
You can install the extension from [chrome web store](https://chrome.google.com/webstore/detail/javascript-injector/ejnccfcackblkelbafbolcpjfpcmbplg).

## How to use
Let's say you want to run a small piece of javascript in your favorite web site https://duckduckgo.com/ every time you visit it.

* Open the website on your chrome:
![alt text](https://raw.githubusercontent.com/hex0cter/js-injector/master/publisher/screenshots/SS1.png "open the website")

* Click on the javascript-injector extension right to the address bar, and fill in the javascript below
```javascript
document.body.style.backgroundColor = "lightblue";
```

![alt text](https://raw.githubusercontent.com/hex0cter/js-injector/master/publisher/screenshots/SS2.png "fill in the javascript")

* Click on update

From now on, whenever you visit the website, on any of your devices (as long as you log into chrome using the same account), the snippet will be executed automatically.
![alt text](https://raw.githubusercontent.com/hex0cter/js-injector/master/publisher/screenshots/SS3.png "the snippet will be executed")

## License

This code is free to use under the terms of the MIT license.

## Contribution

You are more than welcome to raise any issues [here](https://github.com/hex0cter/js-injector/issues), or create a Pull Request.

repository: Nobleraj/js-injector

repo_name: Nobleraj/js-injector


text:
# Long Running Tasks

Currently, in Vitess, long running tasks are used in a couple places:

* `vtworker` runs tasks that can take a long time, as they can deal with a lot
  of data.

* `vttablet` runs backups and restores that can also take a long time.

In both cases, we have a streaming RPC API that starts the job, and streams some
status back while it is running. If the streaming RPC is interrupted, the
behavior is different: `vtworker` and `vttablet backup` will interrupt their
jobs, whereas `vttablet restore` will try to keep going if it is past a point of
no return for restores.

This document proposes a different, but common design for both use cases.

## RPC Model

We introduce a model with three different RPCs:

* Start the job will be a synchronous non-streaming RPC, and just starts the
  job. It returns an identifier for the job.

* Getting the job status is a streaming RPC. It takes the job identifier, and
  streams the current status back. This job status can contain log entries
  (regular messages) or progress (percentage, N/M completion, ...).

* Canceling a running job is its own synchronous non-streaming RPC, and takes
  the job identifier.

For simplicity, let's try to make these the same API for 'vtworker' and
'vttablet'. Usually, a single destination can only run a single job, but let's
not assume that in the API. If a destination process cannot run a job, it should
return the usual `RESOURCE_EXHAUSTED` canonical error code.

These RPCs should be grouped in a new API service. Let's describe it as usual in
`jobdata.proto` and `jobservice.proto`. The current `vtworkerdata.proto` and
`vtworkerservice.proto` will eventually be removed and replaced by the new
service.

Let's use the usual `repeated string args` to describe the job. `vtworker`
already uses that.

So the proposed proto definitions:

``` proto

# in jobdata.proto

message StartRequest {
  repeated string args = 1;
}

message StartResponse {
  string uid = 1;
}

message StatusRequest {
  string uid = 1;
}

// Progress describes the current progress of the task.
// Note the fields here match the Progress and ProgressMessage from the Node
// display of workflows.
message Progress {
  // percentage can be 0-100 if known, or -1 if unknown.
  int8 percentage = 1;

  // message can be empty if percentage is set.
  string message = 2;
}

// FinalStatus describes the end result of a job.
message FinalStatus {
  // error is empty if the job was successful.
  string error = 1;
}

// StatusResponse can have any of its fields set.
message StatusResponse {
  // event is optional, used for logging.
  logutil.Event event = 1;

  // progress is optional, used to indicate progress.
  Progress progress = 2;

  // If final_status is set, this is the last StatusResponse for this job,
  // it is terminated.
  FinalStatus final_status = 3;
}

message CancelRequest {
  string uid = 1;
}

message CancelResponse {
}

# in jobdata.service

service Job {
  rpc Start (StartRequest) returns (StartResponse) {};

  rpc Status (StatusRequest) returns (stream StatusResponse) {};

  rpc Cancel (CancelRequest) returns (CancelResponse) {};
}
```

## Integration with Current Components

### vtworker

This design is very simple to implement within vtworker. At first, we don't need
to link the progress in, just the logging part.

vtworker will only support running one job like this at a time.

### vttablet

This is also somewhat easy to implement within vttablet. Only `Backup` and
`Restore` will be changed to use this.

vttablet will only support running one job like this at a time. It will also
take the ActionLock, so no other tablet actions can run at the same time (as we
do now).

### vtctld Workflows Integration

The link here is also very straightforward:

* When successfully starting a remote job, the address of the remote worker and
  the UID of the job can be checkpointed.

* After that, the workflow can just connect and update its status and logs when
  receiving an update.

* If the workflow is aborted and reloaded somewhere else (vtctld restart), it
  can reconnect to the running job easily.

* Canceling the job is also easy, just call the RPC.

### Comments

Both vtworker and vttablet could remember the last N jobs they ran, and their
status. So when a workflow tries to reconnect to a finished job, they just
stream a single `StatusResponse` with a `final_status` field.

repository: vitessio/vitess

gha_language: Go

repo_name: vitessio/vitess


text:
## Working with Configurations using the Gradle Kotlin DSL

Each configuration `create()`ed under the `jooq { configurations { } }` block presents the script author with an opportunity to customize a tree of JAXB elements drawn
from `jooq-meta` which encodes the [JOOQ Codegen XML Schema](https://www.jooq.org/xsd/jooq-codegen-3.18.0.xsd).

For each configuration so configured, this tree is serialized to a temporary XML file and passed as a command-line argument to `org.jooq.codegen.GenerationTool` for code
generation.

Before your Gradle Kotlin DSL script configures it, each configuration's JAXB tree is pre-populated as follows:

```groovy
        // all types appearing below are from package org.jooq.meta.jaxb
        return new Configuration()
            .withJdbc(new Jdbc())
            .withGenerator(new Generator()
                .withStrategy(new Strategy())
                .withDatabase(new Database())
                .withGenerate(new Generate())
                .withTarget(new Target()
                    .withDirectory(null)
                )
            )
```

The act of customizing this tree can be made DSL-like using the `apply()` extension-method from the Kotlin standard library:

```kotlin
import org.jooq.meta.jaxb.Logging
jooqConfiguration.apply {
    // this: org.jooq.meta.jaxb.Configuration
    logging = Logging.WARN  // simple assignment of single-valued property
    jdbc.apply {  // 'jdbc' property that was previously instantiated for you
        // this: org.jooq.meta.jaxb.Jdbc
        driver = "org.h2.Driver"
        url = "jdbc:h2:~/test;AUTO_SERVER=TRUE"
    }
}
```

However, for collection-valued properties, they are initially-empty, and you must:

1. Instantiate a set of JAXB elements of the correct type
2. Set (or add) them as a collection

In the example below, `forcedTypes` is an initially-empty `List<org.jooq.meta.jaxb.ForcedType>` member of `org.jooq.meta.jaxb.Database` that
is replaced with a collection of two elements:

```kotlin
import org.jooq.meta.jaxb.ForcedType
jooqConfiguration.apply {
    generator.apply {
        database.apply {
            forcedTypes = listOf(
                ForcedType().apply {
                    name = "varchar"
                    includeExpression = ".*"
                    includeTypes = "JSONB?"
                },
                ForcedType().apply {
                    name = "varchar"
                    includeExpression = ".*"
                    includeTypes = "INET"
                }
            )
        }
    }
}
```

Familiarity with the [JOOQ Codegen XML Schema](https://www.jooq.org/xsd/jooq-codegen-3.18.0.xsd) and
the [`org.jooq.meta.jaxb` package](https://github.com/jOOQ/jOOQ/tree/version-3.18.4/jOOQ-meta/src/main/java/org/jooq/meta/jaxb)
will inform your configuration efforts.

repository: etiennestuder/gradle-jooq-plugin

gha_language: Groovy

repo_name: etiennestuder/gradle-jooq-plugin


text:
---
title: feature-sets
authors:
  - "@patrickdillon"
reviewers:
  - "@wking, installer, upgrades"
  - "@bparees, feature sets"
approvers:
  - "@sdodson"
  - "@zaneb"
api-approvers: # In case of new or modified APIs or API extensions (CRDs, aggregated apiservers, webhooks, finalizers). If there is no API change, use "None"
  - "@deads2k"
creation-date: 2022-09-07
last-updated: 2022-09-07
tracking-link: # link to the tracking ticket (for example: Jira Feature or Epic ticket) that corresponds to this enhancement
  - https://issues.redhat.com/browse/CORS-2253
---

# Installer Feature Sets

## Summary

This enhancement proposes adding support for feature sets to `openshift-install`.
Users would be able to enable feature sets at install time. The Installer would support all feature sets in the OpenShift API,
and write the selected Feature Set to the cluster.

## Motivation

Feature gates have been standard in OpenShift since at least 4.1, but have been absent from the Installer.
There are many observable benefits of the practice, but the primary motivations are:

* Align the Installer with OpenShift release practices and graduation criteria, i.e. enable QE & CI testing
of gated features
* Improve Installer developer experience by removing concerns about exposing non-GA features during active development
* Allow a mechanism for pre-release user testing

We have an immediate use case for this as we are targeted to release a tech preview feature in 4.12.

### User Stories

As a member of OpenShift concerned with the release process (TRT, dev, staff engineer, maybe even PM), I want to opt in to
pre-release features so that I can run periodic testing in CI and obtain a signal of feature quality.

As quality engineering, I want to opt in to pre-release features so that I can test nightly and other release images
both in CI & locally.

As a customer, I want to opt in to pre-release features so that I can test and verify proof of concepts.

As an Installer developer, I want to protect new API fields with a feature gate so that I can build features without concern
that an incomplete feature would be exposed to customers.

### Goals

The goals of this proposal are:
* To provide an API that will allow users to enable feature sets in the Installer
* Allow the Installer to enable feature sets in the cluster
* Ensure the proposed implementation adheres to best practices for feature sets

### Non-Goals

None

## Proposal

### Workflow Description

Example workflow demonstrating feature set validation and use:

**Developer** contributes code to the openshift-install codebase.

**Admin** is the user performing an install.

1. Developer adds feature `foo` to openshift-install. Feature `foo`
includes field `bar` in the install config; feature `foo` is part of the Tech Preview Feature Set.
2. Admin specifies `bar: baz` in install-config.yaml (but `featureSet` is empty).
3. `openshift-install` returns an error: `the TechPreviewNoUpgrade feature set must be enabled to use this field`
4. Admin adds `featureSet: TechPreviewNoUpgrade` to install-config.yaml
5. Installer generates `FeatureGate` manifest
6. Installer emits warning message indicating the enabled feature set when provisioning infrastructure
7. Cluster installs successfully with TechPreview cluster

### API Extensions

[OpenShift API Feature Sets](https://github.com/openshift/api/blob/master/config/v1/types_feature.go)
would become part of the install config API:

```go

// FeatureSet enables features that are not part of the default feature set.
	// +optional
	FeatureSet configv1.FeatureSet `json:"featureSet,omitempty"`
```

### Implementation Details/Notes/Constraints [optional]

#### Install Config

The basic implementation through the install config is relatively straight forward. Gated features are protected
by validation:

```go

	if c.FeatureSet == nil || *c.FeatureSet != types.TechPreviewNoUpgrade {
		errMsg := "the TechPreviewNoUpgrade feature set must be enabled to use this field"

    if installConfig.Foo != nil && len(installConfig.Foo.Bar) > 0 {
			allErrs = append(allErrs, field.Forbidden(field.NewPath("foo", "bar"), errMsg))
		}
	}
```

Package `types` defines the install config and represents the API offered by the Installer. Most notably this
API is consumed/used by hive. For this reason, it is important that feature sets be part of the install config.

#### CLI Flag (future work)

On the other hand, the install config is an asset and gated features could potentially be exercised
before the install config is validated (or perhaps the feature is completely independent from the install config). For this reason, it would
be a good idea to provide a flag:

```shell
$ openshift-install create cluster --feature-set TechPreviewNoUpgrade
```

This flag, if provided, would also populate the value in the install config; and conflicts between the flag and install config
would throw an error.

While there are clear use cases for a flag, there are no immediate use cases and the implementation of the Installer (with
its separation between command-line flags and assets) does not lead to a clear-cut design. Implementing this functionality
is certainly possible, but I propose we defer implementation for this until it is determined to be necessary.

#### openshift-install explain

`explain` provides a command-line reference for the install config:

```shell
$ openshift-install explain installconfig | tail -n 6
    pullSecret <string> -required-
      PullSecret is the secret to use when pulling images.

    sshKey <string>
      SSHKey is the public Secure Shell (SSH) key to provide access to instances.
```

`explain` should handle feature-gated fields appropriately, either by skipping them (not printing them) or clearly marking
them as gated features. The initial implementation will simply include
comments denoting fields that are feature gated.

### Risks and Mitigations

I am not aware of any significant risks to introducing this feature.

### Drawbacks

I am not aware of any significant drawbacks to introducing this feature.

## Design Details

### Open Questions [optional]

N/A

### Test Plan

Unit testing the new tech preview functionality and observing existing e2e tests should be sufficient.

### Graduation Criteria

n/a

#### Dev Preview -> Tech Preview

#### Tech Preview -> GA

#### Removing a deprecated feature

### Upgrade / Downgrade Strategy

Upgrades will be blocked based on the feature set. There is
discussion about whether it makes sense to block a cluster
upgrade based on Installer-only features, i.e. if the changes
are only in the Installer and don't affect the cluster, why
block upgrades?

The upgrade status of the cluster is based on the feature set,
so the current implementation would have the ability to accomodate
an Installer-only tech preview if we were able to add a new feature
set for that purpose. For discussion: if we added an `InstallerTechPreview` or `TechPreviewInstallOnly` feature set,
it could be possible that such a feature set would allow upgrades.

There could be a use case for supporting both feature sets in the
Installer: `InstallerTechPreview` would be for install-only
features, while `TechPreview` may apply to features that affect
both the Installer and the cluster.

The addition of such an `InstallerTechPreview` would need to be
added in a follow up enhancement.

### Version Skew Strategy

There is a small risk in unreleased Installers that there may be
a mismatch between the API versions in the installer and the cluster.
The Installer would need to be kept in sync with the OpenShift API
to make sure it is aware of current feature sets, which are updated
rarely.

### Operational Aspects of API Extensions

n/a

#### Failure Modes

Feature gating itself is straightforward install-config validation, so there are no predictable, significant failures.

#### Support Procedures

This should already be consistent with standard tech preview practices.

## Implementation History

## Alternatives

repository: openshift/enhancements

repo_name: openshift/enhancements


text:
# Gene Ontology enrichment analysis

The [Gene Ontology](http://www.geneontology.org/) (GO) project provides structured description for genes known biological information at different levels. The controlled vocabularies (fixed words) describe gene function at 3 levels:
- biological processes
- molecular functions
- cellular components

These terms are species independent and hierarchically structured (see more information [here](http://geneontology.org/page/ontology-documentation)).

Enrichment analysis is a test to see a small subset of genes when sampled from large set of genes (reference set), what is the probability that small subset of genes (or statistically large proportion of subset genes) belong to a functional category as opposed to a randomly sampled subset of genes. This is normally done using either hypergeometric test (test without replacement) or using binomial test (with replacement).

## GO enrichment using Ontologizer
Before you begin, you need to have 2 important files.
  1.  This is easily obtainable from the official Gene Ontotology Consortium website (`go.obo` file from the [downloads page](http://geneontology.org/page/download-ontology))
  2. **Mapping file:** this file describes the association of genes of the organism with the GO terms (this can be generated using Blast2Go- by loading swissprot blast xml files and interproscan and exporting it in .anno file or by any [other ways](http://ontologizer.de/howto/). You can also get them from official webpage if your organism is [listed](http://geneontology.org/page/download-annotations)).

Once you have these 2 files, you can run Ontologizer. Ontologizer is a command line tool which very effecient when you have large number of list to analyze. You need to [download](http://ontologizer.de/cmdline/Ontologizer.jar) the `jar` file to run this
The command:

```
java -jar Ontologizer.jar \
   --association association.anno \
   --go gene_ontology.obo \
   --studyset your_input_list.txt \
   --population population.txt \
   --calculation Parent-Child-Union \
   --mtc Westfall-Young-Single-Step \
   --dot 0.05 \
   --resamplingsteps 1000
```

Here the `population.txt` is basically the full list of genes that you have in anno file. The only time you need to change this is when you have a different background set to test. You can also play around the other settings like `--mtc`, `--resamplingsteps` to optimize. With the `--dot` it will also generate a `dot` file, that can be used with `GraphViz` to show the pathway where these genes are enriched.

The `dot` file is created after the run is complete. You can convert this to `png` format to view them easily:

```
dot -Tpng input.dot -o output.png
```

## GO enrichment using BiNGO

## More information

* Ashburner, M., Ball, C.A., Blake, J.A., Botstein, D., Butler, H., Cherry, J.M., Davis, A.P., Dolinski, K., Dwight, S.S., Eppig, J.T. and Harris, M.A., 2000. [Gene ontology: tool for the unification of biology](https://pubmed.ncbi.nlm.nih.gov/10802651). Nature genetics, 25(1), pp.25-29.
* Gene Ontology Consortium, 2019. [The gene ontology resource: 20 years and still GOing strong](https://pubmed.ncbi.nlm.nih.gov/30395331). Nucleic acids research, 47(D1), pp.D330-D338.

repository: ISUgenomics/bioinformatics-workbook

gha_language: HTML

repo_name: ISUgenomics/bioinformatics-workbook


text:
# Git [pre-commit](http://pre-commit.com/) hook for managing Terraform lock files

A pre-commit hook for managing Terraform lock (`.terraform.lock.hcl`) files.

See also: https://github.com/pre-commit/pre-commit

## How to install

### Using pre-commit-hooks with pre-commit

Add this to your `.pre-commit-config.yaml`

```yaml
-   repo: https://github.com/davidcaste/pre-commit-terraform-lock
    rev: <VERSION> # Get the latest from: https://github.com/davidcaste/pre-commit-terraform-lock/releases
    hooks:
    -   id: check-tf-lock-file-without-zh-entries
    # -   id: ...
```

### Hooks available

#### `check-tf-lock-file-without-zh-entries`
Prevent adding a `zh:` entry to a `.terraform.lock.hcl` file.

repository: davidcaste/pre-commit-terraform-lock

repo_name: davidcaste/pre-commit-terraform-lock


text:
# Style #7 - Infinite Mirror

Constraints:

- All, or a significant part, of the problem is modelled by
  induction. That is, specify the base case (n_0) and then the n+1
  rule

Possible names:

- Infinite mirror
- Inductive
- Recursive

[Example Code](07-infinite-mirror.rs)


text:
# RAMdisk Device

Caution: This page may contain information that is specific to the legacy
version of the driver framework (DFv1). Also the workflows documented on
this page may only be specific to the Fuchsia source checkout
(`fuchsia.git`) environment.

## Overview

In this section, we'll examine a simplified RAM-disk driver.

This driver introduces:

*   the block protocol's **query()** and **queue()** ops
*   Virtual Memory Address Regions ([VMAR](/docs/reference/kernel_objects/vm_address_region.md)s)
    and Virtual Memory Objects ([VMO](/docs/reference/kernel_objects/vm_object.md)s)

The source is in `//examples/drivers//ramdisk/demo-ramdisk.c`.

As with all drivers, the first thing to look at is how the driver initializes itself:

```c
static zx_status_t ramdisk_driver_bind(void* ctx, zx_device_t* parent) {
    zx_status_t status = ZX_OK;

    // (1) create the device context block
    ramdisk_device_t* ramdev = calloc(1, sizeof((*ramdev)));
    if (ramdev == NULL) {
        return ZX_ERR_NO_MEMORY;
    }

    // (2) create a VMO
    status = zx_vmo_create(RAMDISK_SIZE, 0, &ramdev->vmo);
    if (status != ZX_OK) {
        goto cleanup;
    }

    // (3) map the VMO into our address space
    status = zx_vmar_map(zx_vmar_root_self(), 0, ramdev->vmo, 0, RAMDISK_SIZE,
                         ZX_VM_FLAG_PERM_READ | ZX_VM_FLAG_PERM_WRITE, &ramdev->mapped_addr);
    if (status != ZX_OK) {
        goto cleanup;
    }

    // (4) add the device
    device_add_args_t args = {
        .version = DEVICE_ADD_ARGS_VERSION,
        .name = "demo-ramdisk",
        .ctx = ramdev,
        .ops = &ramdisk_proto,
        .proto_id = ZX_PROTOCOL_BLOCK_IMPL,
        .proto_ops = &block_ops,
    };

    if ((status = device_add(parent, &args, &ramdev->zxdev)) != ZX_OK) {
        ramdisk_release(ramdev);
    }
    return status;

    // (5) clean up after ourselves
cleanup:
    zx_handle_close(ramdev->vmo);
    free(ramdev);
    return status;
}

static zx_driver_ops_t ramdisk_driver_ops = {
    .version = DRIVER_OPS_VERSION,
    .bind = ramdisk_driver_bind,
};

ZIRCON_DRIVER_BEGIN(ramdisk, ramdisk_driver_ops, "zircon", "0.1", 1)
    BI_MATCH_IF(EQ, BIND_PROTOCOL, ZX_PROTOCOL_MISC_PARENT),
ZIRCON_DRIVER_END(ramdisk)

```

At the bottom, you can see that this driver binds to a `ZX_PROTOCOL_MISC_PARENT` type of
protocol, and provides `ramdisk_driver_ops` as the list of operations supported.
This is no different than any of the other drivers we've seen so far.

The binding function, **ramdisk_driver_bind()**, does the following:

1.  Allocates the device context block.
2.  Creates a [VMO](/docs/reference/kernel_objects/vm_object.md).
    The [VMO](/docs/reference/kernel_objects/vm_object.md)
    is a kernel object that represents a chunk of memory.
    In this simplified RAM-disk driver, we're creating a
    [VMO](/docs/reference/kernel_objects/vm_object.md) that's `RAMDISK_SIZE`
    bytes long.
    This chunk of memory **is** the RAM-disk — that's where the data is stored.
    The [VMO](/docs/reference/kernel_objects/vm_object.md)
    creation call, [**zx_vmo_create()**](/docs/reference/syscalls/vmo_create.md),
    returns the [VMO](/docs/reference/kernel_objects/vm_object.md) handle through
    its third argument, which is a member in our context block.
3.  Maps the [VMO](/docs/reference/kernel_objects/vm_object.md) into our address space with
    [**zx_vmar_map()**](/docs/reference/syscalls/vmar_map.md).
    This function returns a pointer to a
    [VMAR](/docs/reference/kernel_objects/vm_address_region.md)
    that points to the entire
    [VMO](/docs/reference/kernel_objects/vm_object.md) (because
    we specified `RAMDISK_SIZE` as the mapping size argument) and gives us read and
    write access (because of the `ZX_VM_FLAG_PERM_*` flags).
    The pointer is stored in our context block's `mapped_addr` member.
4.  Adds our device with **device_add()**,
    just like all the examples we've seen above.
    The difference here, though is that we see two new members: `proto_id` and
    `proto_ops`.
    These are defined as "optional custom protocol" members.
    As usual, we store the newly created device in the `zxdev` member of our
    context block.
5.  Cleans up resources if there were any problems along the way.

For completeness, here's the context block:

```c
typedef struct ramdisk_device {
    zx_device_t*    zxdev;
    uintptr_t       mapped_addr;
    uint32_t        flags;
    zx_handle_t     vmo;
    bool            dead;
} ramdisk_device_t;
```

The fields are:

Type            | Field         | Description
----------------|---------------|----------------
`zx_device_t*`  | zxdev         | the ramdisk device
`uintptr_t`     | mapped_addr   | address of the [VMAR](/docs/reference/kernel_objects/vm_address_region.md)
`uin32_t`       | flags         | device flags
`zx_handle_t`   | vmo           | a handle to our [VMO](/docs/reference/kernel_objects/vm_object.md)
`bool`          | dead          | indicates if the device is still alive

### Operations

Where this device is different from the others that we've seen, though,
is that the **device_add()**
function adds two sets of operations; the "regular" one, and an
optional "protocol specific" one:

```c
static zx_protocol_device_t ramdisk_proto = {
    .version = DEVICE_OPS_VERSION,
    .message = ramdisk_message,
    .get_size = ramdisk_getsize,
    .unbind = ramdisk_unbind,
    .release = ramdisk_release,
};

static block_protocol_ops_t block_ops = {
    .query = ramdisk_query,
    .queue = ramdisk_queue,
};
```

The `zx_protocol_device_t` one handles control messages (**ramdisk_message()**), device size
queries (**ramdisk_getsize()**), and device cleanups (**ramdisk_unbind()** and
**ramdisk_release()**).

<!--- should I discuss the ioctls, or were they to have been removed as part of the simplification? -->

The `block_protocol_ops_t` one contains protocol operations particular to the
block protocol.
We bound these to the device in the `device_add_args_t` structure (step (4) above through
the `.proto_ops` field.
We also set the `.proto_id` field to `ZX_PROTOCOL_BLOCK_IMPL` — this is what
identifies this driver as being able to handle block protocol operations.

Let's tackle the trivial functions first:

```c
static zx_off_t ramdisk_getsize(void* ctx) {
    return RAMDISK_SIZE;
}

static void ramdisk_unbind(void* ctx) {
    ramdisk_device_t* ramdev = ctx;
    ramdev->dead = true;
    device_unbind_reply(ramdev->zxdev);
}

static void ramdisk_release(void* ctx) {
    ramdisk_device_t* ramdev = ctx;

    if (ramdev->vmo != ZX_HANDLE_INVALID) {
        zx_vmar_unmap(zx_vmar_root_self(), ramdev->mapped_addr, RAMDISK_SIZE);
        zx_handle_close(ramdev->vmo);
    }
    free(ramdev);
}

static void ramdisk_query(void* ctx, block_info_t* bi, size_t* bopsz) {
    ramdisk_get_info(ctx, bi);
    *bopsz = sizeof(block_op_t);
}
```

**ramdisk_getsize()** is the easiest — it simply returns the size of the resource, in bytes.
In our simplified RAM-disk driver, this is hardcoded as a `#define` near the top of the file.

Next, **ramdisk_unbind()** and **ramdisk_release()** work together.
When the driver is being shut down, the **ramdisk_unbind()** hook is called.
It sets the `dead` flag to indicate that the driver is shutting down (this is checked
in the **ramdisk_queue()** handler, below).
It's expected that the driver will finish up any I/O operations that are in progress (there
won't be any in our RAM-disk), and it should call
**device_unbind_reply()**
to indicate unbinding is complete.

Sometime after **device_unbind_reply()** is called,
the driver's **ramdisk_release()** will be called.
Here we unmap the [VMAR](/docs/reference/kernel_objects/vm_address_region.md),
with [**zx_vmar_unmap()**](/docs/reference/syscalls/vmar_unmap.md), and close the
[VMO](/docs/reference/kernel_objects/vm_object.md),
with [**zx_handle_close()**](/docs/reference/syscalls/handle_close.md).
As our final act, we release the device context block.
At this point, the device is finished.

### Block Operations

The **ramdisk_query()** function is called by the block protocol in order to get
information about the device.
There's a data structure (the `block_info_t`) that's filled out by the driver:

```c
// from .../system/public/zircon/device/block.h:
typedef struct {
    uint64_t    block_count;        // The number of blocks in this block device
    uint32_t    block_size;         // The size of a single block
    uint32_t    max_transfer_size;  // Max size in bytes per transfer.
                                    // May be BLOCK_MAX_TRANSFER_UNBOUNDED if there
                                    // is no restriction.
    uint32_t    flags;
    uint32_t    reserved;
} block_info_t;

// our helper function
static void ramdisk_get_info(void* ctx, block_info_t* info) {
    ramdisk_device_t* ramdev = ctx;
    memset(info, 0, sizeof(*info));
    info->block_size = BLOCK_SIZE;
    info->block_count = BLOCK_COUNT;
    // Arbitrarily set, but matches the SATA driver for testing
    info->max_transfer_size = BLOCK_MAX_TRANSFER_UNBOUNDED;
    info->flags = ramdev->flags;
}
```

In this simplified driver, the `block_size`, `block_count`, and `max_transfer_size`
fields are hardcoded numbers.

The `flags` member is used to identify if the device is read-only (`BLOCK_FLAG_READONLY`,
otherwise it's read/write), removable (`BLOCK_FLAG_REMOVABLE`, otherwise it's not
removable) or has a bootable partition (`BLOCK_FLAG_BOOTPART`, otherwise it doesn't).

The final value that **ramdisk_query()** returns is the "block operation size" value
through the pointer to `bopsz`.
This is a host-maintained block that's big enough to contain the `block_op_t` *plus*
any additional data the driver wants (appended to the `block_op_t`), like an
extended context block.

### Reading and writing

Finally, it's time to discuss the actual "block" data transfers; that is, how does
data get read from / written to the device?

The second block protocol handler, **ramdisk_queue()**, performs that function.

As you might suspect from the name, it's intended that this hook starts whatever
transfer operation (a read or a write) is requested, but doesn't require that
the operation completes before the hook returns.
This is a little like what we saw in earlier chapters
in the **read()** and **write()** handlers
for devices like `/dev/misc/demo-fifo` — there, we could either return
data immediately, or put the client to sleep, waking it up later when data (or room
for data) became available.

With **ramdisk_queue()** we get passed a block operations structure that indicates
the expected operation: `BLOCK_OP_READ`, `BLOCK_OP_WRITE`, or `BLOCK_OP_FLUSH`.
The structure also contains additional fields telling us the offset and size of
the transfer (from `//src/lib/ddk/include/ddk/protocol/block.h`):

```c
// simplified from original
struct block_op {
    struct {
        uint32_t    command;    // command and flags
        uint32_t    extra;      // available for temporary use
        zx_handle_t vmo;        // vmo of data to read or write
        uint32_t    length;     // transfer length in blocks (0 is invalid)
        uint64_t    offset_dev; // device offset in blocks
        uint64_t    offset_vmo; // vmo offset in blocks
        uint64_t*   pages;      // optional physical page list
    } rw;

    void (*completion_cb)(block_op_t* block, zx_status_t status);
};
```

The transfer takes place to or from the `vmo` in the structure — in the case of
a read, we transfer data to the [VMO](/docs/reference/kernel_objects/vm_object.md),
and vice versa for a write.
The `length` indicates the number of *blocks* (not bytes) to transfer, and the
two offset fields, `offset_dev` and `offset_vmo`, indicate the relative offsets (again,
in blocks not bytes) into the device and the [VMO](/docs/reference/kernel_objects/vm_object.md)
of where the transfer should take place.

The implementation is straightforward:

```c
static void ramdisk_queue(void* ctx, block_op_t* bop) {
    ramdisk_device_t* ramdev = ctx;

    // (1) see if we should still be handling requests
    if (ramdev->dead) {
        bop->completion_cb(bop, ZX_ERR_IO_NOT_PRESENT);
        return;
    }

    // (2) what operation are we performing?
    switch ((bop->command &= BLOCK_OP_MASK)) {
    case BLOCK_OP_READ:
    case BLOCK_OP_WRITE: {
        // (3) perform validation common for both
        if ((bop->rw.offset_dev >= BLOCK_COUNT)
            || ((BLOCK_COUNT - bop->rw.offset_dev) < bop->rw.length)
            || bop->rw.length * BLOCK_SIZE > MAX_TRANSFER_BYTES) {
            bop->completion_cb(bop, ZX_ERR_OUT_OF_RANGE);
            return;
        }

        // (4) compute address
        void* addr = (void*) ramdev->mapped_addr + bop->rw.offset_dev * BLOCK_SIZE;
        zx_status_t status;

        // (5) now perform actions specific to each
        if (bop->command == BLOCK_OP_READ) {
            status = zx_vmo_write(bop->rw.vmo, addr, bop->rw.offset_vmo * BLOCK_SIZE,
                                  bop->rw.length * BLOCK_SIZE);
        } else {
            status = zx_vmo_read(bop->rw.vmo, addr, bop->rw.offset_vmo * BLOCK_SIZE,
                                 bop->rw.length * BLOCK_SIZE);
        }

        // (6) indicate completion
        bop->completion_cb(bop, status);
        break;
        }

    case BLOCK_OP_FLUSH:
        bop->completion_cb(bop, ZX_OK);
        break;

    default:
        bop->completion_cb(bop, ZX_ERR_NOT_SUPPORTED);
        break;
    }
}
```

As usual, we establish a context block at the top by casting the `ctx` argument.
The `bop` argument is the "block operation" structure we saw above.
The `command` field indicates what the **ramdisk_queue()** function should do.

In step (1), we check to see if we've set the `dead` flag (**ramdisk_unbind()**
sets it when required).
If so, it means that our device is no longer accepting new requests, so we return
`ZX_ERR_IO_NOT_PRESENT` in order to encourage clients to close the device.

In step (3), we handle some common validation for both read and write —
neither should allow offsets that exceed the size of the device, nor transfer
more than the maximum transfer size.

Similarly, in step (4) we compute the device address (that is, we establish a
pointer to our [VMAR](/docs/reference/kernel_objects/vm_address_region.md)
that's offset by the appropriate number of blocks as per the request).

In step (5) we perform either a [**zx_vmo_read()**](/docs/reference/syscalls/vmo_read.md)
or a [**zx_vmo_write()**](/docs/reference/syscalls/vmo_write.md), depending
on the command.
This is what transfers data between a pointer within our
[VMAR](/docs/reference/kernel_objects/vm_address_region.md) (`addr`)
and the client's [VMO](/docs/reference/kernel_objects/vm_object.md) (`bop->rw.vmo`).
Notice that in the read case, we *write* to the [VMO](/docs/reference/kernel_objects/vm_object.md),
and in the write case, we *read* from the [VMO](/docs/reference/kernel_objects/vm_object.md).

Finally, in step (6) (and the other two cases), we signal completion with the
`completion` callback in the block ops structure.

The interesting thing about completion is that:

*   it doesn't have to happen right away — we could have queued this
    operation and signalled completion some time later,
*   it is allowed to be called before this function returns (like we did).

The last point simply means that we are not *forced* to defer completion until
after the queuing function returns.
This allows us to complete the operation directly in the function.
For our trivial RAM-disk example, this makes sense — we have the ability to
do the data transfer to or from media instantly; no need to defer.

## How is the real one more complicated?

The RAM-disk presented above is somewhat simplified from the "real" RAM-disk
device (present at `//src/devices/block/drivers/ramdisk/ramdisk.c`).

The real one adds the following functionality:

*   dynamic device creation through a new VMO
*   ability to use an existing VMO
*   background thread
*   sleep mode

<!--- how much, if anything, do we want to say about this one? I found the dynamic -->
<!--- device creation of interest, for example... -->

repository: gnoliyil/fuchsia

repo_name: gnoliyil/fuchsia


text:
### [CVE-2020-24753](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-24753)
![](https://img.shields.io/static/v1?label=Product&message=n%2Fa&color=blue)
![](https://img.shields.io/static/v1?label=Version&message=n%2Fa&color=blue)
![](https://img.shields.io/static/v1?label=Vulnerability&message=n%2Fa&color=brighgreen)

### Description

A memory corruption vulnerability in Objective Open CBOR Run-time (oocborrt) in versions before 2020-08-12 could allow an attacker to execute code via crafted Concise Binary Object Representation (CBOR) input to the cbor2json decoder. An uncaught error while decoding CBOR Major Type 3 text strings leads to the use of an attacker-controllable uninitialized stack value. This can be used to modify memory, causing a crash or potentially exploitable heap corruption.

### POC

#### Reference
- https://warcollar.com/cve-2020-24753.html

#### Github
- https://github.com/ARPSyndicate/cvemon
- https://github.com/ep-infosec/50_google_honggfuzz
- https://github.com/google/honggfuzz
- https://github.com/lllnx/lllnx

repository: trickest/cve

repo_name: trickest/cve


text:
# Overview

This file contains a convenience wrapper for logging to the Apple System Log
facility from Objective-C code.  It provides a series of macros for writing
to the log at various levels. It also provides a single function, `ASLog()`,
which takes a log level, an `NSString` format string, and a sequence of zero
or more arguments that are inserted into the format string if necessary.

Prior to importing this header file, it is possible to define `ASLOG_LEVEL` to
any of the `ASL_LEVEL` constants. This can be used to filter the log messages
to only those at an equal or higher level than `ASLOG_LEVEL`.

The log levels, from highest to lowest, are:

* `ASL_LEVEL_EMERG`
* `ASL_LEVEL_ALERT`
* `ASL_LEVEL_CRIT`
* `ASL_LEVEL_ERR`
* `ASL_LEVEL_WARNING`
* `ASL_LEVEL_NOTICE`
* `ASL_LEVEL_INFO`
* `ASL_LEVEL_DEBUG`

If this file is included in the PCH file, one log level shall be enabled for
the entire application. If this file is included by each file using logging,
it is possible to set the log level on a per-file basis.

To make log messages appear in Xcode's debug output area, the following must
be called (for example, in `main.m`):

    ASLogToStderr();

By default, log messages are prefixed with `__PRETTY_FUNCTION__` which expands
to the class name and method name at the location of the `ASLogXXX()` macro
call. If you define `ASLOG_EXCLUDE_PRETTY_FUNCTION` before importing this
file, log messages will not be prefixed with `__PRETTY_FUNCTION__`.


text:
# LearnWords2
A vocabulary learning component for flash cards using the Leitner system

## Aim

The aim of the project is twofold
1. Provide a JavaScript library which keeps a collection of vocabulary questions in a database.
2. Provide data conversion and report functions for lists of words to learn.

It does not provide a GUI though it has a demo how a GUI could be constructed.

A vocabulary question may be picked at random and then depending on the answer a new date is noted for the question to be asked again. The time interval increases if the question has been answered correctly. To do this a step (or level) value is maintained for each question to determine the time period after which the question might be asked again.

This is known as [spaced repetition](https://en.wikipedia.org/wiki/Spaced_repetition).

HTML and LibreOffice Draw reports are provided for flash cards to be printed (experimental feature).

![Leitner system](docs/230px-Leitner_system_alternative.svg.png)

![learnwords2-layers](docs/LW2-context.png)

## Setup

Assuming ``grunt-cli``
````
npm install -g grunt-cli
````
is installed globally, do

````
git clone https://github.com/hhzl/LearnWords2.git
cd LearnWords2
npm install
grunt
````

The ``grunt`` command generates the files in the ``dist`` directory. These JavaScript files are the debug and minified versions of the LearnWords2 library (``dist/LW.js`` and ``dist/LW-debug.js``). They are copied to the ``public/js`` directory so the demo and Jasmine tests are using the "latest version" (i.e. derived from the JavaScript in ``src``) of the library. The directory `public/data/json` contains the output data to be used by the library.

Note that maybe you have to do

````
sudo npm install -g grunt-cli
sudo npm install
````

To use the library:

```
<script src="<path>/LW.js"></script>
```
The file ``public/demo.html`` illustrates this.

## API

![learnwords2-layers](docs/learnwords2-layers.png)

- LWdb is the data access layer.
- LWBoxOfQuestions contains the application logic for a Leitner box

Code:

````JavaScript
"use strict";
var lw = function(){

	var db = LW.LWdb('learnWords');

	var box = LW.BoxOfQuestions(db);

        // configuration

        .......

        box.importFrom(wordlist);

	return box;
}();

````

The construction above (IIFE) returns the BoxOfQuestions object ready for use.

If there is no configuration of the db you could as well just do

````JavaScript
var lw = LW.BoxOfQuestions(LW.LWdb('learnWords'));
    lw.importFrom(wordlist);

````

Then access to everything goes through  ``lw.method()``.

````JavaScript

lw.importFrom(....);

lw.question();

lw.answer();
lw.answerOptions();

lw.moveQuestionBackwards();

lw.moveQuestionForward();

````

Note: If `lw.moveQuestionBackwards()` and `lw.moveQuestionForward()` are called repeatedly on the same question it has only an effect the first time. The next time `lw.question()` is called a new question is provided.

TODO: Add more usage examples.

## Input data

Input data are CSV files with words and pictures in the directory `data/csv`.

## Export data

Export data in as HTML report and Anki database are generated and put into `public/data`.
The data to be used by the library is in `public/data/json`.

## Specification and tests with Jasmine

### In the browser

- on the command line do

     ```
     grunt
     ```

- then a browser window will launch and bring you to a Jasmine test harness page
  which is usually [http://localhost:8000](http://localhost:8000)

### On the command line with node

     npm test

## Credits

Anatol Marezhanyi, for the https://github.com/e1r0nd/LearnWords project
which implements data storage, Leitner box mechanism and a GUI.

The project here is based on ideas and originally on code from LearnWords but
1. implements the non-gui part only to allow other projects to use a library
2. is rewritten to a large extent
3. adds data conversion functions for word list preparation and production of printed flash cards.

## Status

* Jasmine specs are set up and work both in the browser and on the command line.
* The LWdb (database layer) specification is set up and implemented with an MVP set of specs.
* [release 0.2.2-beta](https://github.com/hhzl/LearnWords2/releases/tag/v0.2.2-beta) has a MVP implementation for the Leitner box and may be used for development work.
* release 0.3beta2 will come soon.