text:
# pytorch-checkpoint

[![PyPI version](https://badge.fury.io/py/pytorchcheckpoint.svg)](https://badge.fury.io/py/pytorchcheckpoint)

This package supports saving and loading PyTorch training checkpoints. It is useful when trying the resume model training from a previous step, and can become handy when working with spot instances or when trying to reproduce results.

A model is saved not only with its weights, as one might do for later inference, but the entire state of the model, including the optimizer state and parameters.

In addition, it allows saving metrics and other values generated while training, such as accuracy and loss values. This makes it possible to recreate the learning curves from past values and continue to update them as training proceed.

See accompanying blog post here: [Where did I put my loss values?](https://medium.com/p/where-did-i-put-my-loss-values-acf6264ea347?source=email-486b68bc632a--writer.postDistributed&sk=39b0e303a259edd240b0068e41351c34)

------------------

## Prerequisites
Developed with **Python 3.7.3**, but should be compatible with previous Python version.
```
pip install torch==1.1.0 torchvision==0.3.0
```

## Installation
```pip install pytorchcheckpoint```

## Usage
```python
from pytorchcheckpoint.checkpoint import CheckpointHandler
checkpoint_handler = CheckpointHandler()
```

#### Storing a general value
```python
checkpoint_handler.store_var(var_name='num_of_classes', value=1000)
```
#### Reading a general value
```python
num_of_classes = checkpoint_handler.get_var(var_name='num_of_classes')
```

#### Storing values and metrics for each epoch/iteration. For example, the loss value:
```python
checkpoint_handler.store_running_var(var_name='loss', iteration=0, value=1.0)
checkpoint_handler.store_running_var(var_name='loss', iteration=1, value=0.9)
checkpoint_handler.store_running_var(var_name='loss', iteration=2, value=0.8)
```

#### Reading stored values for epoch/iteration
```python
loss = checkpoint_handler.get_running_var(var_name='loss', iteration=0)
```

#### Storing values and metrics per set: train/valid/test for each epoch/iteration. For example, the top1 value of the train and valid sets:
```python
checkpoint_handler.store_running_var_with_header(header='train', var_name='top1', iteration=0, value=80)
checkpoint_handler.store_running_var_with_header(header='train', var_name='top1', iteration=1, value=85)
checkpoint_handler.store_running_var_with_header(header='train', var_name='top1', iteration=2, value=90)
checkpoint_handler.store_running_var_with_header(header='train', var_name='top1', iteration=3, value=91)

checkpoint_handler.store_running_var_with_header(header='valid', var_name='top1', iteration=0, value=70)
checkpoint_handler.store_running_var_with_header(header='valid', var_name='top1', iteration=1, value=75)
checkpoint_handler.store_running_var_with_header(header='valid', var_name='top1', iteration=2, value=80)
checkpoint_handler.store_running_var_with_header(header='valid', var_name='top1', iteration=3, value=85)
```

#### Reading stored values per set: train/valid/test for epoch/iteration
```python
loss = checkpoint_handler.get_running_var_with_header(header='train', var_name='loss', iteration=0)
```

#### Save checkpoint:
```python
import torchvision.models as models
from torch import optim
checkpoint_handler.store_running_var(var_name='loss', iteration=0, value=1.0)
model = models.resnet18()
optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
checkpoint_handler.optimizer = optimizer
path2save = '/tmp'
checkpoint_path = checkpoint_handler.generate_checkpoint_path(path2save=path2save)
checkpoint_handler.save_checkpoint(checkpoint_path=checkpoint_path, iteration=25, model=model)
```

#### Load checkpoint:
```python
checkpoint_path = '<checkpoint_path>'
checkpoint_handler = checkpoint_handler.load_checkpoint(checkpoint_path)
```

repository: pgsrv/pytorch-checkpoint

repo_name: pgsrv/pytorch-checkpoint


text:
# ember-layout-components

Generic & reusable layout components for your Ember app.

This addon provides easy to use and flexible layout components.
By using these components, your layout will automatically be flexible and consistent.
Instead of defining exactly how something should look like at a given viewport, let the intrinsic size of
content and CSS calcualte the best possible layout, based on the basic rules these layout components define.

Based on [every-layout.dev](https://every-layout.dev/).

![Test & Lint](https://github.com/fabscale/ember-layout-components/workflows/Test%20&%20Lint/badge.svg)

## Compatibility

* Ember.js v3.24 or above
* Ember CLI v3.24 or above
* Node.js v14 or above

This addon makes use of some modern CSS features, which means that browser support might be limited:

- It makes heavy use of runtime CSS Variables, which means it will not work in IE11
- The Cluster uses [flex gap](https://caniuse.com/flexbox-gap), which is supported in all modern browsers (Safari 14.1+). There is an included fallback behavior for browsers that do not support it.

## Installation

```
ember install ember-layout-components
```

## Usage

```hbs
<Layout::Wrapper>
  <Layout::Stack @gap='large' as |Section|>
    <Section>
      First section goes here.
    </Section>

    <Section>
      Second section goes here.
    </Section>
  </Layout::Stack>
</Layout::Wrapper>
```

For more details, see the [documentation](https://fabscale.github.io/ember-layout-components/).

## Available layout components

- `<Layout::Wrapper>`
- `<Layout::Center>`
- `<Layout::Stack>`
- `<Layout::Cluster>`
- `<Layout::Grid>`

## Contributing

See the [Contributing](CONTRIBUTING.md) guide for details.

## License

This project is licensed under the [MIT License](LICENSE.md).

license: MIT

repository: fabscale/ember-layout-components

branch_name: refs/heads/main

extension: md

gha_language: JavaScript

gha_license_id: MIT

license_type: permissive

repo_name: fabscale/ember-layout-components


text:
# Python Tutorial

This tutorial is firstly designed for the study group called '**fMRI with Deep Learning**'. This tutorial is not going to deal with how to use deep learning frameworks, such as Tensorflow, PyTorch, Caffe2, and so on. It will cover only Python prerequisites for learning deep learning frameworks.

To start to learn any deep learning library, you should know the Python primitives: the reserved words, built-in functions, and data types in Python. At the same time, you should know how to write, run, and import a Python file, and how to use the interactive Python environment.

After the basics, you should know widely used open source libraries and enviroments that are equipped with scientific computing. The libraries are, for example, Numpy, Matplotlib, Scipy, and Pandas. Numpy and Matplotlib are undisputablely essential tools for deep learning in terms of data representation and visualization. There are good software environments to help effectively, effieciently develop experiments using those libraires. For instance, there are Jupyter notebook or lab, Anaconda, Spyder, PyCharm, and so on. We will explore how to use those environments.

## Goals

### Python basics

#### Python primitives
* To understand built-in data types: `int`, `float`, `bool`, `NoneType`, `list`, `dict`, `tuple`, `set`
  * `type()`
  * `len()`
  * To make an empty `Collections` instance
  * Add, delete, and modify elements in a `Collections` instance
* To assign a value to a variable
  * Call-by-value, call-by-reference
* To define a function
  * To understand the scope of variables
* To write `for` loops
   * Index access
   * Element-copy access
* To write `while` loops
* To use `break` and `continue` with loop statements
* To write commments

#### Python environments
* To execute a Python file
* To import functions from external files
  * To import a function from an external files
    * The scope of imported lines
  * To import all function from an external files

### Scientific computing tools for Python

#### Numpy
* Base N-dimensional array package
* Matrix operations

#### Matplotlib
* Comprehensive 2D Plotting

#### Scipy
* Fundamental library for scientific computing

#### Pandas
* Data structures & analysis

#### Etc.
* The Scipy ecosystem [[link](https://www.scipy.org/about.html#the-scipy-ecosystem)]

repository: sungjae-cho/python-tutorial

branch_name: refs/heads/master

repo_name: sungjae-cho/python-tutorial


text:
# ls-puck

A script to show which puck is which (I'm always forgetting)

![](screenshot.png)

```bash
npm install -g ls-puck

ls-puck
```

* Select a puck with `up`/`down`
* Press `enter` to highlight the puck with a flash of LED1 (red)

### todo

* implement battery levels
* write stuff better
* link to ide?

repository: benfoxall/ls-puck

repo_name: benfoxall/ls-puck


text:
---
indicator: 2.3.1
layout: indicator
permalink: /2-3-1/
sdg_goal: '2'
data_non_statistical: false
goal_meta_link: 'https://unstats.un.org/sdgs/files/metadata-compilation/Metadata-Goal-2.pdf'
goal_meta_link_text: United Nations Sustainable Development Goals Metadata (PDF 4.0 MB)
graph_type: line
indicator_name: >-
  Volume of production per labour unit by classes of farming/pastoral/forestry
  enterprise size
indicator_sort_order: 02-03-01
published: true
reporting_status: complete
target: >-
  By 2030, double the agricultural productivity and incomes of small-scale food
  producers, in particular women, indigenous peoples, family farmers,
  pastoralists and fishers, including through secure and equal access to land,
  other productive resources and inputs, knowledge, financial services, markets
  and opportunities for value addition and non-farm employment
target_id: '2.3'
un_custodian_agency: Food and Agricultural Organization (FAO)
un_designated_tier: '3'
data_show_map: false
source_active_1: true
source_url_text_1: Link to source
source_active_2: false
source_url_text_2: Link to Source
source_active_3: false
source_url_3: Link to source
source_active_4: false
source_url_text_4: Link to source
source_active_5: false
source_url_text_5: Link to source
source_active_6: false
source_url_text_6: Link to source
title: Untitled
national_indicator_available: Labour productivity in agriculture
computation_calculations: >-
  Volume of agricultural production of small producers of vegetal, animal and
  fish food products, and forestry activity as related to the number of working
  days in one year.

computation_definitions: >-
  Volume of agricultural production per labour unit in
  agricultural/livestock/forestry enterprises
  FAO suggest defining the
  small producers of food supply as producers that:
  1) operate with a size
  of land plots that fits the first two quintiles (under 40%) from the cumulated
  distribution of the size of land plots at the national level (measured in
  hectares); and
  2) operate with a number of animals which fits the first
  two quintiles (under 40%) of the cumulated distribution of the number of
  animals per production unit at the national level (measured in livestock
  units); and
  3) obtain an annual economic revenue from agricultural
  activities which fits the first two quintiles (under 40%) out of the cumulated
  distribution of economic revenues from agricultural activities per production
  unit at the national level (measured in purchasing power parity in USD), which
  does not exceed the amount of 34,387 USD at PPP .
  As the indicator refers
  to a set of production units - the small scale ones - the denominator should
  include the information about the entire production undertaken in every unit.
  Hence it is necessary for the production volumes to be reported in a common
  number, taking into account that it is impossible to refer to physical units.
  The most convenient number for aggregating the products from the denominator
  is a vector of constant prices. When measured at different moments in time, as
  it is requested for monitoring the SDG indicators, the change of constant
  values represents change in aggregated volume.
computation_units: USD
national_geographical_coverage: Republic of Moldova
graph_title: '2.3.1 Labour productivity in agriculture '
source_data_source_1: >-
  1) Agricultural surveys;
  2) Ad-hoc module, complementary in HBS;
  3)
  Agricultural census;
  4) Administrative data

source_data_supplier_1: National Bureau of Statistics
source_organisation_1: National Bureau of Statistics
source_responsible_monitoring_1: >-
  1.Ministry of Agriculture, Regional Development and Environment
  2.Food
  and Agriculture Organization (FAO)
source_periodicity_1: Annual
previous: 2-2-2-b
next: 2-3-1-1
---

repository: statisticamd/open-sdg-data-starter

branch_name: refs/heads/develop

gha_language: Shell

repo_name: statisticamd/open-sdg-data-starter


text:
# ClojureFam - Week 2 - Day 4

I mentioned yesterday that I was unsure what direction to continue my ClojureFam journey in. After talking about it with my cohort members, I decided to continue on with learning Clojure for the Brave and True (henceforth shall be known as CFTBAT) while also trying to get practical knowledge of the Athen's Codebase. To this end, some of my cohort members and I have decided on a particular [issue](https://github.com/athensresearch/athens/issues/209) that we would like to focus our attention on.

Luckily this issue turned out to inform exactly what my next steps should be.

## Athens Issue

Let's take a quick look at what the issue says -

> prevent `indent` when already a leaf block, prevent `unindent` when already a top-level child

From my understanding of how Roam Research works, I guessed that when it says "prevent indent of leaf nodes", a node that doesn't have any children cannot further indented to the right. Also, if we are at the root node, we cannot unindent it. This also makes intuitive sense since the root node is already at the left most point, and nodes can only be further indented fromt this point.

The relevant code for this issue is found in this code snippet at `athens/events.cljs` -

```clojure
;; TODO: no-op when indenting as the right-most child
(reg-event-fx
  :indent
  (fn [_ [_ uid]]
    (let [block (get-block [:block/uid uid])
          parent (get-parent [:block/uid uid])
          older-sib (->> parent
                      :block/children
                      (filter #(= (dec (:block/order block)) (:block/order %)))
                      first
                      :db/id
                      get-block)
          new-block {:db/id (:db/id block) :block/order (count (:block/children older-sib))}
          reindex-blocks (->> (d/q '[:find ?ch ?new-o
                                     :in $ % ?p ?at
                                     :where (dec-after ?p ?at ?ch ?new-o)]
                                @db/dsdb rules (:db/id parent) (:block/order block))
                           (map (fn [[id order]] {:db/id id :block/order order})))]
      {:transact [[:db/retract (:db/id parent) :block/children (:db/id block)]
                  {:db/id (:db/id older-sib) :block/children [new-block]} ;; becomes child of older sibling block — same parent but order-1
                  {:db/id (:db/id parent) :block/children reindex-blocks}]}))) ;; reindex parent
```

A cursory reading of this seems somewhat understandable given what I know about Clojure. But it largely relies on knowing the `re-frame` library and how state management and events are handled in this library.

So I set out to get myself familiarized with `re-frame` and also with Datalog, a declarative query language used in the project. I'm currently doing the exercises on [LearnDatalogToday](http://www.learndatalogtoday.org/) and re-reading the introduction to [Re-frame](https://purelyfunctional.tv/guide/re-frame-a-visual-explanation/). I feel like I conceptually get the motivations behind `re-frame` but there's quite a lot of implementation details to understand.

## Other things I worked on Today

I also spent some today doing a few other Clojure related things. Having gotten through Clojure from the Ground Up, I started reading Chapter 3 of CFTBAT again. Chapters 1 and 2 are somewhat irrelevant since they cover setting up Emacs as an environment which I'd already done.

Speaking of environments, I decided to take a peek at Calva and VS Code today and I'm somewhat enjoying the experience. The keybinds for Calva are just dissimilar enough from CIDER that it keeps tripping me up, but from a functional standpoint, Calva seems perfectly adequate for my workflow which mostly just involves evaluating the current s-exp repeatedly.

One productivity benefit I've noticed is that using Git has been a lot easier since I like the GitLens extension quite a lot more than I like Magit. Probably because I never got around to seriously learning Magit. It just seems comfortable doing simple things like accessing the terminal, interacting with the file system etc. However, I miss Emacs's buffer and window system.

While reading CFTBAT, I came across the line - "Strings can only be concatenated. Clojure does not have string interpolation". I immediately took this as a challenge and decided to attempt implementing a Clojure macro that allows for String Interpolation.

I imagined the syntax for string interpolation to be something similar to Python's f-strings -

```python
f"Hello my name is {name} and I'm {age} years old"
```

I got as far as writing some code to perform a regex search and capture the variables between the `{}`. This looks like -

```clojure
(re-seq #"\{([a-zA-Z]+)\}" "name is {name} and lang is {lang}")
```

Unfortunately, that's about as far as I got. I'll have to leave the rest of this macro for another day.

## Takeaways

I did some quick and dirty dabbling into `re-frame` and Datalog. `re-frame` is quite interesting and I definitely need to spend a lot more time getting the mental model for Re-frame into my head. The comparison of `re-frame` to Redux isn't exactly a helpful one since I'm not that familiar with State Management in React either. (Just making pretty components).

Anyway, I think this was a very broad learning day rather than one with deep focus. Which in my opinion is ok since it offeres a lot of interesting things to digest. What is crucial is that I follow up on the knowledge from today and keep building on it.

repository: itsrainingmani/learn-clojure-in-public

repo_name: itsrainingmani/learn-clojure-in-public


text:
---
type: quick-start
hide_in_page_nav: true
---
# Fork a collection

<LoggedIn id='postman_credentials'>
  Now that you are logged in we can fork the **Box Postman Collection** into the
  **Postman App** that we installed previously. When we fork the Postman
  Collection we will also automatically load your API credentials as a Postman
  environment.

  ## Forking a collection and environment

  By clicking the button below you will fork the **Box Postman
  Collection** into your Postman application. In the same click it will also
  load your **Access Token**, **Refresh Token**, **Client ID** and **Client
  Secret** into a Postman environment.

  <Trigger option='postman_collection_downloaded' value='true'>
    <Postman env='postman_credentials' />
  </Trigger>

  We recommend to fork the Box Postman Collection - you will be asked if you
  want to update the collection any time Box makes changes to it. You can also
  copy the collection, but you might miss important updates.
</LoggedIn>

<Choice option='postman_collection_downloaded' value='true' color='none'>

## Exploring the collection

When you clicked the button above it would have asked you to fork the
collection into the Postman application. Once imported, the collection should
appear within the app in the left-hand sidebar.

<ImageFrame border center shadow width='600'>
  ![Box collection in Postman](./collection-in-postman.png)
</ImageFrame>

You can click on the collection to open it up and explore our over 170 API
endpoints.

## Summary

* You forked the Postman collection into Postman
* You additionally loaded your Box Postman environment into Postman

</Choice>

<Choice option='postman.app_type' unset color='none'>
  <LoggedIn reverse>
    <Message danger>
      # Incomplete previous step

      Complete the previous steps to select and log in to a **Box App**.
    </Message>
  </LoggedIn>
</Choice>

<Choice option='postman.app_type' value='create_new,use_existing' color='none'>
  <LoggedIn id='postman_credentials' reverse>
    <Message danger>
      # Incomplete previous step

      Complete the previous steps to select and log in to a **Box App**.
    </Message>
  </LoggedIn>
</Choice>

<Observe option='postman_collection_downloaded' value='true'>
  <Next>I have forked the collection</Next>
</Observe>


text:
# Experiment Algorithm Deep Dive

This guide's purpose is to provide familiarity of the inner workings of a Triage Experiment to people with some
experience in data science and Python. A Triage Experiment is a highly structured way of defining the experimentation
phase of a data science project. To those wondering whether this Experiment structure is flexible enough to fit their
needs, this should help.

## 1. Temporal Validation Setup

First, the given `temporal_config` section in the experiment definition is transformed into train and test splits,
including `as_of_times` for each matrix.

We create these splits by figuring out the latest reasonable split time from the inputs, and moving backwards in time
at the rate of the given `model_update_frequency`, until we get to the earliest reasonable split time.

For each split, we create `as_of_times` by moving either backwards from the split time towards the
`max_training_history` (for train matrices) or forwards from the split time towards the `test_duration` (for test
matrices) at the provided `data_frequency`.

Many of these configured values may be lists, in which case we generate the cross-product of all the possible values
and generate more splits.

For a more detailed look at the temporal validation logic, see [Temporal Validation Deep Dive](temporal-validation.md).

The train and test splits themselves are not used until the [Building Matrices](#3-building-matrices) section, but a
flat list of all computed `as_of_times` for all matrices needed in the experiment is used in the next section,
[Transforming Data](#2-transforming-data).

## 2. Transforming Data

With all of the `as_of_times` for this Experiment now computed, it's now possible to transform the input data into
features and labels as of all the required times.

### Labels

The Experiment populates a 'labels' table using the following input:
1. A query, provided by the user in the configuration file, that generates entity_ids and outcomes for a given
as_of_date and label_timespan.

2. Each as_of_date and label_timespan defined in temporal config

For instance, an inspections-style query (for the given timespan, return the entity and outcome of any matching
inspections) would look like:

```
	select
	events.entity_id,
	bool_or(outcome::bool)::integer as outcome
	from events
	where '{as_of_date}' <= outcome_date
		and outcome_date < '{as_of_date}'::timestamp + interval '{label_timespan}'
		group by entity_id
```

This binary labels table is scoped to the entire Experiment, so all `as_of_time` (computed in step 1) and
`label_timespan` (taken straight from `temporal_config`) combinations are present. Additionally, the 'label_name' and
'label_type' are also recorded with each row in the table.

The name of the labels table is based on both the name of the label and a hash of the label query (e.g `labels_failedviolation_a0b1c2d3`), so any prior experiments that shared both the name and query will be able to reuse the labels table.  If the 'replace' flag was sent, for each `as_of_time` and `label_timespan`, the labels table is queried to check if any rows exist that match. If any such rows exist, the labels query for that date and timespan is not run.

At this point, the 'labels' table may not have entries for all entities and dates that need to be in a given matrix.
How these rows have their labels represented is up to the configured `include_missing_labels_in_train_as` value in the
experiment. This value is not processed when we generate the labels table, but later on when the matrix is built (see
'Retrieving Data and Saving Completed Matrix')

### Cohort Table

The Experiment keeps track of the which entities are in the cohort on any given date. Similarly to the labels table, the experiment populates a cohort table. using one of two options:

1. A query, provided by the user in the configuration file, that generates entity_ids for a given as_of_date. This is run for each as_of_date as generated from the temporal config.

2. All distinct entity ids and as_of_dates in the labels table, if no query is provided by the user in the configuration file.

This cohort table is scoped to the entire Experiment, so all `as_of_times` (computed in step 1) are present.

The name of the cohort table is based on both the name of the cohort and a hash of the cohort query (e.g `cohort_permitted_a0b1c2d3`), so any prior experiments that shared both the name and query will be able to reuse the cohort table.  If the 'replace' flag was sent, for each `as_of_time`, the cohort table is queried to check if any rows exist that match. If any such rows exist, the cohort query for that date is not run.

### Features

Each provided `feature_aggregation` configures the creation and population of several feature tables in the 'features'
schema: one for each of the groups specified in the config, one that merges the groups together into one table, and one
that fills in null values from the merged table with imputed values based on imputation config.

#### Generating Aggregation SQL
To generate the SQL that creates the pre-imputation table, the Experiment assembles building blocks from the feature
aggregation config, as well as the experiment's list of `as_of_times`:

* `from_obj` represents, well, the object of the FROM clause in the SQL query. Often this is just a table, but can be
configured to be a subquery. This holds all the data that we want to aggregate into features
* Each `as_of_time` in the experiment and `interval` in the `feature_aggregation` is combined with the
`knowledge_date_column` to create a WHERE clause representing a valid window of events to aggregate in the `from_obj`:
e.g (`where {knowledge_date_column} >= {as_of_time} - interval {interval}`)
* Each `aggregate`, `categorical`, or `array_categorical` represents a SELECT clause. For aggregates, the `quantity` is
a column or SQL expression representing a numeric quantity present in the `from_obj`, and the `metrics` are any number
of aggregate functions we want to use. The aggregate function is applied to the quantity.
* By default the query is joined with the cohort table to remove unnecessary rows. If `features_ignore_cohort` is passed to the Experiment this is not done.

So a simplified version of a typical query would look like:
```
SELECT {group}, {metric}({quantity})
FROM {from_obj}
JOIN {cohort_table} ON (
    {cohort_table.entity_id} = {from_obj.entity_id}
    AND {cohort_table.date} = {as_of_time}
)
WHERE {knowledge_date_column} >= {as_of_time} - interval {interval}
GROUP BY entity_id
```

#### Writing Group-wide Feature Tables
For each `as_of_time`, the results from the generated query are written to a table whose name is prefixed with the
`prefix`, and suffixed with the `group` (always `entity_id` in `triage`). The table is keyed on this grouping column plus the as_of_date.

#### Merging into Aggregation-wide Feature Tables
Each generated group table is combined into one representing the whole aggregation with a left join to ensure all valid entity/date pairs are included (allowing for identification of nulls requiring imputation). This aggregation-level table represents all of the features
in the aggregation, pre-imputation. Its output location is generally `{prefix}_aggregation`

#### Imputing Values
A table that looks similar, but with imputed values is created. The cohort table from above is passed into collate as
the comprehensive set of entities and dates for which output should be generated, regardless if they exist in the
`from_obj`. Each feature column has an imputation rule, inherited from some level of the feature definition. The
imputation rules that are based on data (e.g. `mean`) use the rows from the `as_of_time` to produce the imputed value.
In addition, each column that needs imputation has an imputation flag column created, which contains a boolean flagging which rows were imputed or not. Since the values of these columns are redundant for most aggregate functions that look at a given timespan's worth of data (they will be imputed only if zero events in their timespan are seen), only one imputation flag column per timespan is created. An exception to this are some statistical functions that require not one, but two values, like standard deviation and variance. These boolean imputation flags are *not* merged in with the others.
Its output location is generally `{prefix}_aggregation_imputed`

### Recap

At this point, we have at least three tables that are used to populate matrices:

- `labels_{labelname}_{labelqueryhash}` with computed labels for each date
- `cohort_{cohortname}_{cohortqueryhash}` with the cohort for each date
- A `features.{prefix}_aggregation_imputed` table for each feature aggregation present in the experiment config.

## 3. Building Matrices

At this point, we have to build actual train and test matrices that can be processed by machine learning algorithms, save at the user's specified path, either on the local filesystem or s3 depending on the scheme portion of the path (e.g. `s3://bucket-name/project_directory`)

First we have to figure out exactly what matrices we have to build. The split definitions from step 1 are a
good start -- they are our train and test splits -- but sometimes we also want to test different subsets of the data,
like feature groups (e.g. 'how does using group of features A perform against using all features?'). So there's a layer
of iteration we introduce for each split, that may produce many more matrices.

What do we iterate over?
* Feature List - All subsets of features that the user wants to cycle through. This is the end result of the feature
group generation and mixing process, which is described more below.
* Cohorts - In theory we can take in different cohorts and iterate in the same experiment.  This is not fully implemented, so in reality we just use the one cohort that is passed in the `cohort_config`
* Label names - In theory we can take in different labels (e.g. complaints, sustained complaints) in the same
experiment. Right now there is no support for multiple label names, but the label name used is configurable through the
optional 'label_config'->'name' config value
* Label types - In theory we can take in different label types (e.g. binary) in the same experiment. *Right now this
isn't done, there is one label type and it is hardcoded as 'binary'.*

### Feature Lists
How do we arrive at the feature lists? There are two pieces of config that are used: `feature group_definition` and
`feature_group_strategies`. Feature group definitions are just ways to define logical blocks of features, most often
features that come from the same source, or describing a particular type of event. These groups within the experiment
as a list of feature names, representing some subset of all potential features for the experiment. Feature group
strategies are ways to take feature groups and mix them together in various ways. The feature group strategies take
these subsets of features and convert them into another list of subsets of features, which is the final list iterated
over to create different matrices.

#### Feature Group Definition
Feature groups, at present, can be defined as either a `prefix` (the prefix of the feature name), a `table` (the
feature table that the feature resides in), or `all` (all features).  Each argument is passed as a list, and each entry
in the list is interpreted as a group. So, a feature group config of `{'table': ['complaints_aggregate_imputed',
'incidents_aggregate_imputed']}` would result in two feature groups: one with all the features in
`complaints_aggregate_imputed`, and one with all the features in `incidents_aggregate_imputed`. Note that this requires
a bit of knowledge on the user's part of how the feature table names will be constructed.

`prefix` works on the prefix of the feature name as it exists in the database. So this also requires some knowledge of
how these get created. The general format is: `{aggregation_prefix}_{group}_{timeperiod}_{quantity}`, so with some
knowledge the user can create groups with the aggregation's configured prefix (common), or the aggregations configured
prefix + group (in case they want to compare, for instance, zip-code level features versus entity level features).

`all`, with a single value of `True`, will include a feature group with all defined features. If no feature group
definition is sent, this is the default.

Either way, at the end of this process the experiment will be aware of some list of feature groups, even if the list is
just length 1 with all features as one group.

#### Feature Group Mixing
A few basic feature group mixing strategies are implemented: `leave-one-in`, `leave-one-out`, and `all`. These are sent
in the experiment definition as a list, so different strategies can be tried in the same experiment. Each included
strategy will be applied to the list of feature groups from the previous step, to convert them into

For instance, 'leave-one-in' will cycle through each feature group, and for each one create a list of features that
just represents that feature group, so for some matrices we would only use features from that particular group.
`leave-one-out` does the opposite, for each feature group creating a list of features that includes all other feature
groups but that one. `all` just creates a list of features that represents all feature groups together.

### Iteration and Matrix Creation

At this point, matrices are created by looping through all train/test splits and data subsets (e.g. feature groups,
state definitions), grabbing the data corresponding to each from the database, and assembling that data into a design
matrix that is saved along with the metadata that defines it.

As an example, if the experiment defines 3 train/test splits (one test per train in this example, for simplicity), 3
feature groups that are mixed using the 'leave-one-out' and 'all' strategies, and 1 state definition, we'll expect 18
matrices to be saved: 9 splits after multiplying the time splits by the feature groups, and each one creating a train
and test matrix.

### Associating Matrices with Experiment

After all matrices for the Experiment are defined but before any are built, the Experiment is associated with each Matrix in the database through the `triage_metadata.experiment_matrices` table. This means that whether or not the Experiment has to end up building a matrix, after the fact a user can query the database to see if it used said matrix.

#### Retrieving Data and Saving Completed Matrix

Each matrix that has to be built (i.e. has not been built by some prior experiment) is built by retrieving its data out of the database.

How do we get the data for an individual matrix out of the database?

1. Create an entity-date table for this specific matrix. There is some logic applied to decide what rows show up. There
are two possible sets of rows that could show up.

- `all valid entity dates`. These dates come from the entity-date-state table for the experiment (populated using the
rules defined in the 'cohort_config'), filtered down to the entity-date pairs that match both *the state filter and the
list of as-of-dates for this matrix*.

- `all labeled entity dates`. These dates consist of all the valid entity dates from above, that also have an entry in
the labels table.

If the matrix is a test matrix, all valid entity dates will be present.

If the matrix is a train matrix, whether or not valid but unlabeled examples show up is decided by the
`include_missing_labels_in_train_as` configuration value. If it is present in any form, these labels will be in the
matrix. Otherwise, they will be filtered out.

2. Write features data from tables to disk in CSV format using a COPY command, table by table. Each table is joined
with the matrix-specific entity-date table to only include the desired rows.

3. Write labels data to disk in CSV format using a COPY command. These labels will consist of the rows in the
matrix-specific entity-date table left joined to the labels table. Rows not present in the labels table will have their
label filled in (either True or False) based on the value of the `include_missing_labels_in_train_as` configuration key.

4. Merge the features and labels CSV files horizontally, in pandas. They are expected to be of the same shape, which is
enforced by the entity-date table. The resulting matrix is indexed on `entity_id` and `as_of_date`, and then saved to
disk (in CSV format, more formats to come) along with its metadata: time, feature, label, index, and state information.
along with any user metadata the experiment config specified. The filename is decided by a hash of this metadata, and
the metadata is saved in a YAML file with the same hash and directory. The metadata is additionally added to a database table 'matrices'.

Matrix metadata reference:
- [Train matrix temporal info](https://github.com/dssg/timechop/blob/master/timechop/timechop.py#L433-L440)
- [Test matrix temporal info](https://github.com/dssg/timechop/blob/master/timechop/timechop.py#L514-L523)
- [Feature, label, index, cohort, user
metadata](https://github.com/dssg/triage/blob/master/src/triage/component/architect/planner.py#L89-L112)

### Recap

At this point, all finished matrices and metadata will be saved under the `project_path` supplied by the user to the
Experiment constructor, in the subdirectory `matrices`.

## 4. Running Models

The last phase of an Experiment run uses the completed design matrices to train, test, and evaluate classifiers. This procedure writes a lot of metadata to the 3 schemas: 'triage_metadata', 'train_results', and 'test_results'.

### Associating Models with Experiment

Every combination of training matrix + classifier + hyperparameter is considered a Model. Before any Models are trained, the Experiment is associated with each Model in the database through the `triage_metadata.experiment_models` table. This means that whether or not the Experiment has to end up training a model, after the fact a user can query the database to see if it used said model.

### Train

Each matrix marked for training is sent through the configured grid in the experiment's `grid_config`. This works much
like the scikit-learn `ParameterGrid` (and in fact uses it on the backend). It cycles through all of the classifiers
and hyperparameter combinations contained herein, and calls `.fit()` with that train matrix. Any classifier that
adheres to the scikit-learn `.fit/.transform` interface and is available in the Python environment will work here,
whether it is a standard scikit-learn classifier, a third-party library like XGBoost, or a custom-built one in the
calling repository (for instance, one that implements the problem domain's baseline heuristic algorithm for
comparison).  Metadata about the trained classifier is written to the `triage_metadata.models` Postgres table. The trained model is saved to a filename with the model hash (see Model Hash section below).

#### Model Groups

Each model is assigned a 'model group'. A model group represents a number of trained classifiers that we want to treat
as equivalent by some criteria. By default, this is aimed at defining models which are equivalent across time splits,
to make analyzing model stability easier. This default is accomplished with a set of 'model group keys' that includes
data about the classifier (module, hyperparameters), temporal intervals used to create the train matrix (label
timespan, training history, as-of-date frequency), and metadata describing the data in the train matrix (features and
feature groups, label name, cohort name). The user can override this set of `model_group_keys` in the experiment
definition, with all of the default information plus other matrix metadata at their disposal (See end of 'Retrieving
Data and Saving Completed Matrix' section for more about matrix metadata). This data is stored in the `triage_metadata.model_groups` table, along with a `model_group_id` that is used as a foreign key in the `triage_metadata.models` table.

#### Model Hash
Each trained model is assigned a hash, for the purpose of uniquely defining and caching the model. This hash is based
on the training matrix metadata, classifier path, hyperparameters (except those which concern execution and do not
affect results of the classifier, such as `n_jobs`), and the given project path for the Experiment. This hash can be
found in each row of the `triage_metadata.models` table. It is enforced as a unique key in the table.

#### Global Feature Importance
The training phase also writes global feature importances to the database, in the `train_results.feature_importances` table.
A few methods are queried to attempt to compute feature importances:
* The bulk of these are computed using the trained model's `.feature_importances_` attribute, if it exists.
* For sklearn's `SVC` models with a linear kernel, the model's `.coef_.squeeze()` is used.
* For sklearn's LogisticRegression models, `np.exp(model.coef_).squeeze()` is used.
* Otherwise, no feature importances are written.

### Test Matrix
For each test matrix, predictions, individual importances, and the user-specified testing evaluation metrics are written to the 'test_results' schema. For each train matrix, predictions and the user-specified training evaluation metrics are written to the 'train_results' schema.

#### Predictions
The trained model's prediction probabilities (`predict_proba()`) are computed both for the matrix it was trained on and any testing matrices. The predictions for the training matrix are saved in `train_results.predictions` and those for the testing matrices are saved in the `test_results.predictions`. More specifically, `predict_proba` returns the probabilities for each label (false and true), but in this case only the probabilities for the true label are saved in the `{train or test}_predictions` table. The `entity_id` and `as_of_date` are retrieved from the matrix's index, and stored in the database table along with the probability score, label value (if it has one), as well as other metadata.

### Individual Feature Importance
Feature importances (of a configurable number of top features, defaulting to 5) for each prediction are computed and written to the `test_results.individual_importances` table. Right now, there are no sophisticated calculation methods integrated into the experiment; simply the top 5 global feature importances for the model are copied to the `individual_importances` table.

### Metrics
Triage allows for the computation of both testing set and training set evaluation metrics. Evaluation metrics, such as precision and recall at various thresholds, are written to either the `train_results.evaluations` table or the `test_results.evaluations`. Triage defines a number of [Evaluation Metrics](https://github.com/dssg/triage/blob/master/src/triage/component/catwalk/evaluation.py#L45-L58) metrics that can be addressed by name in the experiment definition, along with a list of thresholds and/or other parameters (such as the 'beta' value for fbeta) to iterate through.

Thresholding is done either via absolute value (top k) or percentile by sorting the predictions and labels by the row's predicted probability score, with ties broken in some way (see next paragraph), and assigning the predicted value as True for those above the threshold. Note that the percentile thresholds are in terms of the population percentage, not a cutoff threshold for the predicted probability.

A few different versions of tiebreaking are implemented to deal with the nuances of thresholding, and each result is written to the evaluations table for each metric score, along with some related statistics:

* `worst_value` - Ordering by the label ascending. This has the effect of as many predicted negatives making it above thresholds as possible, thus producing the worst possible score.
* `best_value` - Ordering by the label descending. This has the effect of as many predicted positives making it above thresholds as possible, thus producing the best possible score.
* `stochastic_value` - If the `worst_value` and `best_value` are not the same (as defined by the floating point tolerance at catwalk.evaluation.RELATIVE_TOLERANCE), the sorting/thresholding/evaluation will be redone many times, and the mean of all these trials is written to this column. Otherwise, the `worst_value` is written here
* `num_sort_trials` - If trials are needed to produce the `stochastic_value`, the number of trials taken is written here. Otherwise this will be 0
* `standard_deviation` - If trials are needed to produce the `stochastic_value`, the standard deviation of these trials is written here. Otherwise this will be 0

Sometimes test matrices may not have labels for every row, so it's worth mentioning here how that is handled and interacts with thresholding. Rows with missing labels are not considered in the metric calculations, and if some of these rows are in the top k of the test matrix, no more rows are taken from the rest of the list for consideration. So if the experiment is calculating precision at the top 100 rows, and 40 of the top 100 rows are missing a label, the precision will actually be calculated on the 60 of the top 100 rows that do have a label. To make the results of this more transparent for users, a few extra pieces of metadata are written to the evaluations table for each metric score.

* `num_labeled_examples` - The number of rows in the test matrix that have labels
* `num_labeled_above_threshold` - The number of rows above the configured threshold for this metric score that have
labels
* `num_positive_labels` - The number of positive labels in the test matrix

Triage supports performing a bias audit using the Aequitas library, if a `bias_audit_config` is passed in configuration. This is handled first through creating a 'protected groups'table which retrieves the configured protected group information for each member of the cohort, and the time that this protected group information was first known. This table is named using a hash of the bias audit configuration, so data can be reused across experiments as long as the bias configuration does not change.

A bias audit is performed alongside metric calculation time for each model that is built, on both the train and test matrices, and each subset. This is very similar to the evaluations table schema, in that for each slice of data that has evaluation metrics generated for it, also receives a bias audit. The change is that thresholds are not borrowed from the evaluation configuration, as aequitas audits are computationally expensive and large threshold grids are common in Triage experiments; the bias audit has its evaluation thresholds configured in the `bias_audit_config`. All data from the bias audit is saved to either the `train_results.aequitas` or `test_results.aequitas` tables.

Triage also supports evaluating a model on a subset of the predictions made.
This is done by passing a subset query in the prediction config. The model
evaluator will then subset the predictions on valid entity-date pairs for the
given model and will calculate metrics for the subset, re-applying thresholds
as necessary to the predictions in the subset. Subset definitions are stored in
the `triage_metadata.subsets` table, and the evaluations are stored in the
`evaluations` tables. A hash of the subset configuration identifies subset
evaluations and links the `subsets` table.

### Recap
At this point, the 'triage_metadata', 'train_results', and 'test_results' database schemas are fully populated with data about models, model groups, predictions, feature importances, and evaluation metrics for the researcher to query. In addition, the trained model pickle files are saved in the configured project path. The experiment is considered finished.

repository: dssg/triage

gha_language: Jupyter Notebook

gha_license_id: NOASSERTION

repo_name: dssg/triage


text:
# Oracle Sharding on Docker
Oracle Sharding is a scalability and availability feature for custom-designed OLTP applications that enables distribution and replication of data across a pool of Oracle databases that do not share hardware or software. The pool of databases is presented to the application as a single logical database.

Sample Docker build files to facilitate installation, configuration, and environment setup for DevOps users. For more information about Oracle Database please see the [Oracle Sharded Database Management Documentation](http://docs.oracle.com/en/database/).

## How to build and run
This project offers sample Dockerfiles for:
  * Oracle Database 19c Client (19.3) for Linux x86-64

To assist in building the images, you can use the [buildDockerImage.sh](dockerfiles/buildDockerImage.sh) script.See section **Create Oracle Global Service Manager Image** for instructions and usage.

**IMPORTANT:** Oracle Global Service Manager container is useful when you want to configure Global Data Service Framework. The Global Data Services framework consists of at least one global service manager, a Global Data Services catalog, and the GDS configuration databases.

For complete Oracle Sharding Database setup, please go through following steps and execute them as per your environment:

### Create Oracle Global Service Manager Image
**IMPORTANT:** You will have to provide the installation binaries of Oracle Global Service Manager Oracle Database 19c  (19.3) for Linux x86-64 and put them into the `dockerfiles/<version>` folder. You  only need to provide the binaries for the edition you are going to install. The binaries can be downloaded from the [Oracle Technology Network](http://www.oracle.com/technetwork/database/enterprise-edition/downloads/index.html). You also have to make sure to have internet connectivity for yum. Note that you must not uncompress the binaries.

The `buildDockerImage.sh` script is just a utility shell script that performs MD5 checks and is an easy way for beginners to get started. Expert users are welcome to directly call `docker build` with their preferred set of parameters.Before you build the image make sure that you have provided the installation binaries and put them into the right folder. Go into the **dockerfiles** folder and run the **buildDockerImage.sh** script as root or with sudo privileges:

```
./buildDockerImage.sh -v (Software Version)
./buildDockerImage.sh -v 19.3.0
```
For detailed usage of command, please execute following command:
```
./buildDockerImage.sh -h
```
### Create Oracle Database Image
To download and build Oracle 19.3 Database image, please refer [README.MD](https://github.com/oracle/docker-images/blob/master/OracleDatabase/SingleInstance/README.md) of Oracle Single Database available on Oracle GitHub repository.

**Note**: You just need to create the image. You will create the container as per the steps given in this document.

### Create Network Bridge
Before creating container, create the macvlan bridge. If you are using same bridge name with same network subnet then you can use same IPs mentioned in **Create Containers** section.

```
# docker network create -d macvlan --subnet=<IP_ADDRESS>/24 --gateway=<IP_ADDRESS> -o parent=eth0 shard_pub1_nw
```

If you are planing to create a test env within a single machine, you can use docker bridge but these IPs will not be reachable on user network.

```
# docker network create --driver=bridge --subnet=<IP_ADDRESS>/24 shard_pub1_nw
```

**Note:** You can change subnet according to your environment.

### Create Containers.
Before creating the GSM container, you need to build the catalog and shard containers. Execute following steps to create containers:

#### Setup Hostfile
All containers will share a host file for name resolution.  The shared hostfile must be available to all container. Create the shared host file (if it doesn't exist) at `/opt/containers/shard_host_file`:

For example:

```
# mkdir /opt/containers
# touch /opt/containers/shard_host_file
```

Add following host entries in /opt/containers/shard_host_file as Oracle database containers do not have root access to modify the /etc/hosts file. This file must be pre-populated. You can change these entries based on your environment and network setup.

```
<IP_ADDRESS>       localhost.localdomain   localhost
<IP_ADDRESS>     oshard-gsm1.example.com  oshard-gsm1
<IP_ADDRESS>     oshard-catalog-0.example.com  oshard-catalog-0
<IP_ADDRESS>     oshard1-0.example.com   oshard1-0
<IP_ADDRESS>     oshard2-0.example.com   oshard2-0
<IP_ADDRESS>     oshard3-0.example.com   oshard3-0
```

#### Copy User Scripts to setup Env
From the cloned Oracle Sharding repository, you need to copy Sharding/dockerfile/<version/setupOshardEnv.sh to some other directory to expose to DB containers. In our example, we created /tmp/oshard and copied the file under that location.

#### Password Setup
Specify the secret volume for resetting database users password during catalog and shard setup. It can be shared volume among all the containers

```
mkdir /opt/.secrets/
openssl rand -hex 64 -out /opt/.secrets/pwd.key
```

Edit the `/opt/.secrets/common_os_pwdfile` and seed the password for grid/oracle and database. It will be a common password for all the database users. Execute following command:

```
openssl enc -aes-256-cbc -salt -in /opt/.secrets/common_os_pwdfile -out /opt/.secrets/common_os_pwdfile.enc -pass file:/opt/.secrets/pwd.key
rm -f /opt/.secrets/common_os_pwdfile
```

#### Deploying Catalog Container
The shard catalog is a special-purpose Oracle Database that is a persistent store for SDB configuration data and plays a key role in automated deployment and centralized management of a sharded database.  Also, change the datafile volume location. In the above example, it is set to /docker_volumes/oradata.

To check the catalog container/services creation logs , please tail docker logs. It will take 20 minutes to create the catalog container service.

```
docker logs catalog
```

#### Deploying Shard Containers
A database shard is a horizontal partition of data in a database or search engine. Each individual partition is referred to as a shard or database shard.

 Also, change the datafile volume location. In the above example, it is set to /docker_volumes/oradata.

To check the shard1 container/services creation logs , please tail docker logs.  Also, change the datafile volume location. In the above example, it is set to /docker_volumes/oradata.

**Note**: You can add more shards based on your requirement.

To check the shard2 container/services creation logs , please tail docker logs.

To check the gsm1 container/services creation logs , please tail docker logs. It will take 2 minutes to create the gsm container service.

```
docker logs gsm1
```

You should see following when cman container setup is done:

```
###################################
GSM IS READY TO USE!
###################################
```


text:
# List View

A List View displays data in rows. Each row displays the same set of attributes, although not necessarily displayed in columns, and the attributes may wrap. The List View omits column headers and other visual noise, making it a cleaner visualization than a [Table View](http://www.patternfly.org/pattern-library/content-views/table-view/).

Jump to [Standard List View](#standard-list-view), [List View with Expanding Rows](#list-view-with-expanding-rows) or [Row Variation Examples](#row-variation-examples)

## Standard List View

![List view](img/standard-list-view.png)

[View full page example](https://www.patternfly.org/pattern-library/content-views/list-view/list-view.html#_)

## List View with Expanding Rows

Use a List View with expanding rows when you want to display details in place without requiring drill-down to a new page. This is useful when you want to allow users to view details of an object without reloading the page, or when the detail information is small and does not require its own page. Simple expansion and compound expansion are two variants of this pattern.

### Simple Expansion

Simple expansion supports a single expansion state to expose more detail for a selected row.

![List view with an expanded row](img/list-view-simple-expansion.png)

### Compound Expansion

Compound expansion supports multiple expansion panels to examine multiple attributes of the selected row.

![List view with compound row expansion](img/list-view-compound-expansion.png)

## List View with Draggable Rows

Provide customizable sorting capabilities to users by applying the [Drag and Drop](https://www.patternfly.org/pattern-library/forms-and-controls/drag-and-drop/#/api) pattern to a standard List View. Drag and Drop allows users to dynamically reorder items in the list.

![List view with draggable rows](img/list-view-drag-and-drop.png)

## Row Variation Examples

The following row variations provide some examples of how you might visualize content within a row. You are not limited to these layouts. You may need to adjust the visualizations to best fit your application’s needs. Each row within a List Vsiew should have a consistent layout. Do not mix and match.

![List view row variations](img/list-view-variations.png)

repository: jsdelivrbot/pf-dev

repo_name: jsdelivrbot/pf-dev


text:
# Angular 1.6 Practice set
Repository content the practice set of code related to different topics like.

Core Example:
1. Expressions
2. Modules
3. Directives
4. Model
5. Data Binding
6. Controllers
7. Scopes
8. Filters
9. Services
10. Http
11. Table
12. SQL
13. DOM
14. Events
15. Form
16. Validation
17. API
18. Includes
19. Animations
20. Routing

Express API
1. Intro
2. API Routing
3. HTTP Method
4. URL Building
5. Middleware
6. Templating
7. Form Data
8. Database
9. Cookies
10. Session
11. OAuth
12. Scaffolding
13. Error handling
14. Debugging
15. Best Practice

Practice set is available for all, can be downloaded and do the practice in your end.

Step to Run the server use http-server npm module.
Run in the terminal and get the global http server.
npm install -g http-server

Thanks,
Manoj


text:
# PHP Lightweight Framework Packages
A collection of packages for [PHP Lightweight Framework](https://github.com/LeviMooreDev/PHP-Lightweight-Framework)


text:
# RainbowAdminPortal.SystemsGroupsApi

All URIs are relative to *http://localhost*

Method | HTTP request | Description
------------- | ------------- | -------------
[**deleteSystemsFromGroupId**](SystemsGroupsApi.md#deleteSystemsFromGroupId) | **DELETE** /api/rainbow/admin/v1.0/systems/{systemId}/groups/{groupId} | Remove a system from a group
[**deleteSystemsGroupId**](SystemsGroupsApi.md#deleteSystemsGroupId) | **DELETE** /api/rainbow/admin/v1.0/systems/groups/{groupId} | Delete a systems group
[**getSystemsGroupId**](SystemsGroupsApi.md#getSystemsGroupId) | **GET** /api/rainbow/admin/v1.0/systems/groups/{groupId} | Get systems group data
[**getSystemsGroups**](SystemsGroupsApi.md#getSystemsGroups) | **GET** /api/rainbow/admin/v1.0/systems/groups | Get all systems groups
[**postSystemsGroups**](SystemsGroupsApi.md#postSystemsGroups) | **POST** /api/rainbow/admin/v1.0/systems/groups | Create a systems group
[**putSystemsGroupId**](SystemsGroupsApi.md#putSystemsGroupId) | **PUT** /api/rainbow/admin/v1.0/systems/{systemId}/groups/{groupId} | Add a new system to a group
[**putSystemsGroupNameId**](SystemsGroupsApi.md#putSystemsGroupNameId) | **PUT** /api/rainbow/admin/v1.0/systems/groups/{groupId} | Rename a systems group

## deleteSystemsFromGroupId

> DeleteSystemsFromGroupIdSuccess deleteSystemsFromGroupId(systemId, groupId, accept, contentType)

Remove a system from a group

Remove system to a group. If the system doesn't exist in the group, an error is raised (404 Not found)    A systems group is a logical set of systems (pabx), sharing the same dial plan and linked through a private network.    Systems belonging to this group may either belonging to the same Site, or belonging from several sites under the same company.    Waiting for a real use case, it is possible to federate inside the same group, systems belonging to companies under the same organization.

### Example

```javascript
var RainbowAdminPortal = require('rainbow_admin_portal');
var defaultClient = RainbowAdminPortal.ApiClient.instance;
// Configure API key authorization: Bearer
var Bearer = defaultClient.authentications['Bearer'];
Bearer.apiKey = 'YOUR API KEY';
// Uncomment the following line to set a prefix for the API key, e.g. "Token" (defaults to null)
//Bearer.apiKeyPrefix = 'Token';

var apiInstance = new RainbowAdminPortal.SystemsGroupsApi();
var systemId = "systemId_example"; // String | System unique identifier
var groupId = "groupId_example"; // String | SystemsGroup unique identifier
var accept = "accept_example"; // String | application/json
var contentType = "contentType_example"; // String | application/json
apiInstance.deleteSystemsFromGroupId(systemId, groupId, accept, contentType).then(function(data) {
  console.log('API called successfully. Returned data: ' + data);
}, function(error) {
  console.error(error);
});

```

### Parameters

Name | Type | Description  | Notes
------------- | ------------- | ------------- | -------------
 **systemId** | **String**| System unique identifier |
 **groupId** | **String**| SystemsGroup unique identifier |
 **accept** | **String**| application/json |
 **contentType** | **String**| application/json |

### Return type

[**DeleteSystemsFromGroupIdSuccess**](DeleteSystemsFromGroupIdSuccess.md)

### Authorization

[Bearer](../README.md#Bearer)

### HTTP request headers

- **Content-Type**: Not defined
- **Accept**: application/json

## deleteSystemsGroupId

> DeleteSystemsGroupIdSuccess deleteSystemsGroupId(groupId, accept)

Delete a systems group

Delete a systems groups.

### Example

```javascript
var RainbowAdminPortal = require('rainbow_admin_portal');
var defaultClient = RainbowAdminPortal.ApiClient.instance;
// Configure API key authorization: Bearer
var Bearer = defaultClient.authentications['Bearer'];
Bearer.apiKey = 'YOUR API KEY';
// Uncomment the following line to set a prefix for the API key, e.g. "Token" (defaults to null)
//Bearer.apiKeyPrefix = 'Token';

var apiInstance = new RainbowAdminPortal.SystemsGroupsApi();
var groupId = "groupId_example"; // String | SystemsGroup unique identifier
var accept = "accept_example"; // String | application/json
apiInstance.deleteSystemsGroupId(groupId, accept).then(function(data) {
  console.log('API called successfully. Returned data: ' + data);
}, function(error) {
  console.error(error);
});

```

### Parameters

Name | Type | Description  | Notes
------------- | ------------- | ------------- | -------------
 **groupId** | **String**| SystemsGroup unique identifier |
 **accept** | **String**| application/json |

### Return type

[**DeleteSystemsGroupIdSuccess**](DeleteSystemsGroupIdSuccess.md)

### Authorization

[Bearer](../README.md#Bearer)

### HTTP request headers

- **Content-Type**: Not defined
- **Accept**: application/json

## getSystemsGroupId

> GetSystemsGroupIdSuccess getSystemsGroupId(groupId, accept)

Get systems group data

A systems group is a logical set of systems (pabx), sharing the same dial plan and linked through a private network.    Systems belonging to this group may either belonging to the same Site, or belonging from several sites under the same company.       According with api permission, the response is not the same.    - Superadmin, Support and organization_admin must see all Systems inside the group.    - bp_admin, company_admin may see Systems hosted by sites under his company.    - site_admin only see systems of his site, inside the group>.

### Example

```javascript
var RainbowAdminPortal = require('rainbow_admin_portal');
var defaultClient = RainbowAdminPortal.ApiClient.instance;
// Configure API key authorization: Bearer
var Bearer = defaultClient.authentications['Bearer'];
Bearer.apiKey = 'YOUR API KEY';
// Uncomment the following line to set a prefix for the API key, e.g. "Token" (defaults to null)
//Bearer.apiKeyPrefix = 'Token';

var apiInstance = new RainbowAdminPortal.SystemsGroupsApi();
var groupId = "groupId_example"; // String | SystemsGroup unique identifier
var accept = "accept_example"; // String | application/json
apiInstance.getSystemsGroupId(groupId, accept).then(function(data) {
  console.log('API called successfully. Returned data: ' + data);
}, function(error) {
  console.error(error);
});

```

### Parameters

Name | Type | Description  | Notes
------------- | ------------- | ------------- | -------------
 **groupId** | **String**| SystemsGroup unique identifier |
 **accept** | **String**| application/json |

### Return type

[**GetSystemsGroupIdSuccess**](GetSystemsGroupIdSuccess.md)

### Authorization

[Bearer](../README.md#Bearer)

### HTTP request headers

- **Content-Type**: Not defined
- **Accept**: application/json

## getSystemsGroups

> GetSystemsGroupsSuccess getSystemsGroups(accept, opts)

Get all systems groups

Get all systems groups.    A systems group is a logical set of systems (pabx), sharing the same dial plan and linked through a private network.    Systems belonging to this group may either belonging to the same Site, or belonging from several sites under the same company.    According with api permission, the response is not the same.    - Superadmin and Support see all SystemsGroups    - bp_admin, organization_admin and company_admin may see several SystemsGroups, **but some systems could be hidden for company_admin**.    - site_admin only see SystemsGroups hosting some systems of his site, **but some systems could be hidden as they are hosted by another site**.

### Example

```javascript
var RainbowAdminPortal = require('rainbow_admin_portal');
var defaultClient = RainbowAdminPortal.ApiClient.instance;
// Configure API key authorization: Bearer
var Bearer = defaultClient.authentications['Bearer'];
Bearer.apiKey = 'YOUR API KEY';
// Uncomment the following line to set a prefix for the API key, e.g. "Token" (defaults to null)
//Bearer.apiKeyPrefix = 'Token';

var apiInstance = new RainbowAdminPortal.SystemsGroupsApi();
var accept = "accept_example"; // String | application/json
var opts = {
  'name': "name_example", // String | Allows to filter groups list on the given keyword(s) on field name. The filtering is **not case insensitive** and on partial name match: all groups containing the provided name value will be returned (whatever the position of the match). Ex: if filtering is done on comp, group with the following names are match the filter 'My Group', 'Group', 'A Group 1', 'Group of', ...
  'format': "format_example", // String | Should allow to retrieve more or less group details in response. But in fact whatever the choice:    - `small`: _id, name, companies, systems (all SystemsGroup fields)    - `medium`: all SystemsGroup fields    - `full`: all SystemsGroup fields
  'limit': 3.4, // Number | Allow to specify the number of companies to retrieve.
  'offset': 3.4, // Number | Allow to specify the position of first company to retrieve (first company if not specified). Warning: if offset > total, no results are returned.
  'sortField': "sortField_example", // String | Sort company list based on the given field.
  'sortOrder': 3.4 // Number | Specify order when sorting company list.
};
apiInstance.getSystemsGroups(accept, opts).then(function(data) {
  console.log('API called successfully. Returned data: ' + data);
}, function(error) {
  console.error(error);
});

```

### Parameters

Name | Type | Description  | Notes
------------- | ------------- | ------------- | -------------
 **accept** | **String**| application/json |
 **name** | **String**| Allows to filter groups list on the given keyword(s) on field name. The filtering is **not case insensitive** and on partial name match: all groups containing the provided name value will be returned (whatever the position of the match). Ex: if filtering is done on comp, group with the following names are match the filter 'My Group', 'Group', 'A Group 1', 'Group of', ... | [optional]
 **format** | **String**| Should allow to retrieve more or less group details in response. But in fact whatever the choice:    - `small`: _id, name, companies, systems (all SystemsGroup fields)    - `medium`: all SystemsGroup fields    - `full`: all SystemsGroup fields | [optional]
 **limit** | **Number**| Allow to specify the number of companies to retrieve. | [optional]
 **offset** | **Number**| Allow to specify the position of first company to retrieve (first company if not specified). Warning: if offset > total, no results are returned. | [optional]
 **sortField** | **String**| Sort company list based on the given field. | [optional]
 **sortOrder** | **Number**| Specify order when sorting company list. | [optional]

### Return type

[**GetSystemsGroupsSuccess**](GetSystemsGroupsSuccess.md)

### Authorization

[Bearer](../README.md#Bearer)

### HTTP request headers

- **Content-Type**: Not defined
- **Accept**: application/json

## postSystemsGroups

> PostSystemsGroupsSuccess postSystemsGroups(body)

Create a systems group

A systems group is a logical set of systems (pabx), sharing the same dial plan and linked through a private network.    Systems belonging to this group may either belonging to the same Site, or belonging from several sites under the same company.    Waiting for a real use case, it is possible to federate inside the same group, systems belonging to companies under the same organization.    It's not possible to include multi-tenant systems (isCentrex = true) inside a systems group.       **Specific feature:** Sharing a system between several companies    Since 1.47.0 release, configuring companies sharing a multi-tenant system is possible.   An OXE can be multi-company.    For us an entity is a company with the flag isCentrex=true.   A multi-tenant system can't be set inside a systems group. That does not make sense.

### Example

```javascript
var RainbowAdminPortal = require('rainbow_admin_portal');
var defaultClient = RainbowAdminPortal.ApiClient.instance;
// Configure API key authorization: Bearer
var Bearer = defaultClient.authentications['Bearer'];
Bearer.apiKey = 'YOUR API KEY';
// Uncomment the following line to set a prefix for the API key, e.g. "Token" (defaults to null)
//Bearer.apiKeyPrefix = 'Token';

var apiInstance = new RainbowAdminPortal.SystemsGroupsApi();
var body = new RainbowAdminPortal.PostSystemsGroups(); // PostSystemsGroups |
apiInstance.postSystemsGroups(body).then(function(data) {
  console.log('API called successfully. Returned data: ' + data);
}, function(error) {
  console.error(error);
});

```

### Parameters

Name | Type | Description  | Notes
------------- | ------------- | ------------- | -------------
 **body** | [**PostSystemsGroups**](PostSystemsGroups.md)|  |

### Return type

[**PostSystemsGroupsSuccess**](PostSystemsGroupsSuccess.md)

### Authorization

[Bearer](../README.md#Bearer)

### HTTP request headers

- **Content-Type**: application/json
- **Accept**: application/json

## putSystemsGroupId

> PutSystemsGroupIdSuccess putSystemsGroupId(systemId, groupId)

Add a new system to a group

Add a new system to a group. If the system already exists in the group or inside another one, an error is raised (409 Conflict)    A systems group is a logical set of systems (pabx), sharing the same dial plan and linked through a private network.    Systems belonging to this group may either belonging to the same Site, or belonging from several sites under the same company.    Waiting for a real use case, it is possible to federate inside the same group, systems belonging to companies under the same organization.       **Specific feature:** Sharing a system between several companies    Since 1.47.0 release, configuring companies sharing a multi-tenant system is possible.   An OXE can be multi-company.    For us an entity is a company with the flag isCentrex=true.   A multi-tenant system can't be set inside a systems group. That does not make sense.

### Example

```javascript
var RainbowAdminPortal = require('rainbow_admin_portal');
var defaultClient = RainbowAdminPortal.ApiClient.instance;
// Configure API key authorization: Bearer
var Bearer = defaultClient.authentications['Bearer'];
Bearer.apiKey = 'YOUR API KEY';
// Uncomment the following line to set a prefix for the API key, e.g. "Token" (defaults to null)
//Bearer.apiKeyPrefix = 'Token';

var apiInstance = new RainbowAdminPortal.SystemsGroupsApi();
var systemId = "systemId_example"; // String | System unique identifier
var groupId = "groupId_example"; // String | SystemsGroup unique identifier
apiInstance.putSystemsGroupId(systemId, groupId).then(function(data) {
  console.log('API called successfully. Returned data: ' + data);
}, function(error) {
  console.error(error);
});

```

### Parameters

Name | Type | Description  | Notes
------------- | ------------- | ------------- | -------------
 **systemId** | **String**| System unique identifier |
 **groupId** | **String**| SystemsGroup unique identifier |

### Return type

[**PutSystemsGroupIdSuccess**](PutSystemsGroupIdSuccess.md)

### Authorization

[Bearer](../README.md#Bearer)

### HTTP request headers

- **Content-Type**: Not defined
- **Accept**: application/json

## putSystemsGroupNameId

> PutSystemsGroupNameIdSuccess putSystemsGroupNameId(groupId, body)

Rename a systems group

Only allow a systems group renaming.

### Example

```javascript
var RainbowAdminPortal = require('rainbow_admin_portal');
var defaultClient = RainbowAdminPortal.ApiClient.instance;
// Configure API key authorization: Bearer
var Bearer = defaultClient.authentications['Bearer'];
Bearer.apiKey = 'YOUR API KEY';
// Uncomment the following line to set a prefix for the API key, e.g. "Token" (defaults to null)
//Bearer.apiKeyPrefix = 'Token';

var apiInstance = new RainbowAdminPortal.SystemsGroupsApi();
var groupId = "groupId_example"; // String | SystemsGroup unique identifier
var body = new RainbowAdminPortal.PutSystemsGroupNameId(); // PutSystemsGroupNameId |
apiInstance.putSystemsGroupNameId(groupId, body).then(function(data) {
  console.log('API called successfully. Returned data: ' + data);
}, function(error) {
  console.error(error);
});

```

### Parameters

Name | Type | Description  | Notes
------------- | ------------- | ------------- | -------------
 **groupId** | **String**| SystemsGroup unique identifier |
 **body** | [**PutSystemsGroupNameId**](PutSystemsGroupNameId.md)|  |

### Return type

[**PutSystemsGroupNameIdSuccess**](PutSystemsGroupNameIdSuccess.md)

### Authorization

[Bearer](../README.md#Bearer)

### HTTP request headers

- **Content-Type**: application/json
- **Accept**: application/json

gha_language: Java


text:
# fluent-agent-hydra

A Fluentd log agent.

[![Build status](https://api.travis-ci.org/fujiwara/fluent-agent-hydra.svg?branch=master)](https://travis-ci.org/fujiwara/fluent-agent-hydra)

This agent is inspired by [fluent-agent-lite](https://github.com/tagomoris/fluent-agent-lite).

## Features

- Tailing log files (like in_tail)
  - enable to handle multiple files in a single process.
  - parse JSON or LTSV format.
- Forwarding messages to external fluentd (like out_forward)
  - multiple fluentd server can be used. When primary server is down, messages will sent to secondary server.
  - if config.ServerRoundRobin = true, select one server from all servers by round robin.
- Receiving a fluentd's forward protocol messages via TCP (like in_forward)
  - includes simplified on-memory queue.
- Stats monitor httpd server
  - serve an agent stats by JSON format.

## Installation

[Binary releases](https://github.com/fujiwara/fluent-agent-hydra/releases)

or

```
go get github.com/fujiwara/fluent-agent-hydra/cmd/fluent-agent-hydra/
```

## Usage

### Using command line arguments

```
fluent-agent-hydra [options] TAG TARGET_FILE PRIMARY_SERVER SECONDARY_SERVER
```

Options

* -f="message": fieldname of fluentd log message attribute (DEFAULT: message)
* -monitor="<IP_ADDRESS>:24223": monitor httpd daemon address (DEFAULT: -)

Usage example

```
fluent-agent-hydra -f msg tagname /path/to/foo.log fluentd.example.com:24224 <IP_ADDRESS>:24224
```

* Field name: "msg"
* Filename: "/path/to/foo.log"
* Primary server: "fluentd.example.com:24224"
* Secondary server: "<IP_ADDRESS>:24224"

### Using configuration file

```
fluent-agent-hydra -c /path/to/config.toml
```

A example of config.toml

```toml
# global settings
TagPrefix = "nginx"       # "nginx.access", "nginx.error"
FieldName = "message"     # default "message"
ReadBufferSize = 1048576  # default 64KB.
ServerRoundRobin = true   # default false

# tailing log file (in_tail)
[[Logs]]
File = "/var/log/nginx/access.log"
Tag = "access"
# parse as ltsv format. (see http://ltsv.org/)
# Format = "None"(default) | "LTSV" | "JSON"
Format = "LTSV"

# convert column data type
# 'column1_name:type,column2_name:type'
# type = "interger" | "float" | "bool" | otherwise as string
Types = "reqtime:float,size:integer,apptime:float,status:integer"

# parse a time string in log lines, and set it as record's timestamp
TimeParse = true      # default false
TimeKey = "timestamp" # default "time"

# TimeFormat is passed to Golang's time.Parse().
# http://golang.org/pkg/time/#Parse
# default time.RFC3339 == "2006-01-02T15:04:05Z07:00"
TimeFormat = "02/Jan/2006:15:04:05 Z0700"

[[Logs]]
File = "/var/log/nginx/error.log"
Tag = "error"

# forwarding fluentd server (out_forward)
[[Servers]]
Host = "fluentd.example.com"
Port = 24224

[[Servers]]
Host = "fluentd-backup.example.com"
Port = 24224

# receive fluentd forward protocol daemon (in_forward)
[Receiver]
Port = 24224

# stats monitor http daemon
[Monitor]
Host = "localhost"
Port = 24223
```

### About special conversion behavior for numerical value

When the `Format` is JSON, fluent-agent-hydra treats a numerical value as float64 even if its type is integer.
For treating a numerical value as integer, set a column data type integer with the `Types`.

```toml
Types = "column_name:integer"
```

Its type is converted to int64.

## Stats monitor

For enabling stats monitor, specify command line option `-m host:port` or `[Monitor]` section in config file.

### Hydra application stats

`curl -s [Monitor.Host]:[Monitor.Port]/ | jq .`

An example response.

```json
{
  "receiver": {
    "buffered": 0,
    "disposed": 0,
    "messages": 123,
    "max_buffer_messages": 1048576,
    "current_connections": 1,
    "total_connections": 10,
    "address": "[::]:24224"
  },
  "servers": [
    {
      "error": "",
      "alive": true,
      "address": "fluentd.example.com:24224"
    },
    {
      "error": "[2014-08-18 18:25:28.965066394 +0900 JST] dial tcp <IP_ADDRESS>:24224: connection refused",
      "alive": false,
      "address": "fluentd-backup.example.com:24224"
    }
  ],
  "files": {
    "/var/log/nginx/error.log": {
      "error": "",
      "position": 95039,
      "tag": "nginx.error"
    },
    "/var/log/nginx/access.log": {
      "error": "",
      "position": 112093,
      "tag": "nginx.access"
    }
  },
  "sent": {
    "nginx.error": {
      "bytes": 2578,
      "messages": 8
    },
    "nginx.access": {
      "bytes": 44996,
      "messages": 109
    }
  }
}
```

### system stats

`curl -s [Monitor.Host]:[Monitor.Port]/system | jq .`

An example response.

```json
{
  "time":<PHONE_NUMBER>556699400,
  "go_version": "go1.3",
  "go_os": "darwin",
  "go_arch": "amd64",
  "cpu_num": 4,
  "goroutine_num": 17,
  "gomaxprocs": 1,
  "cgo_call_num": 48,
  "memory_alloc": 551840,
  "memory_total_alloc": 17886960,
  "memory_sys": 5310712,
  "memory_lookups": 321,
  "memory_mallocs": 4645,
  "memory_frees": 3622,
  "memory_stack": 131072,
  "heap_alloc": 551840,
  "heap_sys": 2097152,
  "heap_idle": 1253376,
  "heap_inuse": 843776,
  "heap_released": 0,
  "heap_objects": 1023,
  "gc_next": 1083088,
  "gc_last":<PHONE_NUMBER>454501600,
  "gc_num": 34,
  "gc_per_second": 0.966939666110773,
  "gc_pause_per_second": 0.641048,
  "gc_pause": [
    0.2991,
    0.341948
  ]
}
```

## Benchmark

* AWS EC2 c3.2xlarge
  * Amazon Linux AMI release 2014.03
  * Linux 3.10.48-55.140.amzn1.x86\_64 #1 SMP Wed Jul 9 23:32:19 UTC 2014 x86\_64 x86\_64 x86\_64 GNU/Linux
* fluentd 0.10.52
  * ruby 2.1.2p95 (2014-05-08 revision 45877) [x86_64-linux]
* fluent-agent-lite 1.0
  * perl This is perl 5, version 16, subversion 3 (v5.16.3) built for x86_64-linux-thread-multi
* fluent-agent-hydra v0.0.5
  * go version go1.3.1 linux/amd64

Benchmark set [fluentd-benchmark/one_forward](https://github.com/fluent/fluentd-benchmark/tree/master/one_forward)

| lines/sec  | fluentd CPU |         RSS | lite CPU |      RSS | hydra(1) CPU |    RSS | hydra(2) CPU | RSS  |
|-----------:|------------:|------------:|---------:|---------:|----------:|----------:|-------------:|-----:|
|        100 |        0.65 |       40152 |     0.18 |     8424 |      0.30 |      5400 |         0.78 |  5244 |
|       1000 |        1.7  |       51836 |     0.34 |     8420 |      0.68 |      5872 |          1.2 |  6212 |
|       5000 |        5.5  |       87316 |      1.0 |     9280 |       1.8 |      7728 |          2.8 |  9804 |
|      10000 |         11  |       85468 |      2.0 |     9812 |       3.4 |      9796 |          5.1 | 10560 |
|      50000 |         48  |      132496 |      8.9 |    15216 |        15 |     10840 |           24 | 11728 |
|     100000 |        107  |      636892 |       18 |    20948 |        31 |     11092 |           48 | 13284 |
|     250000 |          -  |           - |       45 |    21048 |        71 |     15748 |          115 | 20076 |
|     500000 |          -  |           - |       87 |    21092 |     87(a) |  65788(a) |          173 | 30344 |
|     700000 |          -  |           - |       -  |       -  |         - |         - |       165(b) | 81756(b) |

* fluent-agent-lite max:  580,000/sec
* fluent-agent-hydra(GOMAXPROCS=1) max: 460,000/sec
* fluent-agent-hydra(GOMAXPROCS=1, ReadBufferSize=1,000,000) max: 550,000/sec (a)
* fluent-agetn-hydra(GOMAXPROCS=2, ReadBufferSize=1,000,000) max: 700,000/sec (b)

## Thanks to

* `fluent/fluent.go, utils.go` imported and modified from [github.com/t-k/fluent-logger-golang](https://github.com/t-k/fluent-logger-golang).
* `fluent/server.go` imported and modified from [github.com/moriyoshi/ik](https://github.com/moriyoshi/ik/).

## Author

Fujiwara Shunichiro<EMAIL_ADDRESS>
## Licence

Copyright 2014 Fujiwara Shunichiro.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

repository: grubern/fluent-agent-hydra

repo_name: grubern/fluent-agent-hydra


text:
# ricoh_camera

ROS package for the [Ricoh Theta S omnidirectional camera](https://theta360.com/en/about/theta/s.html).

Tested with [ROS Kinetic](http://wiki.ros.org/kinetic).

The [user guide](https://support.theta360.com/en/manual/s/index.html) provides more detailed explanations
on how to use this camera.

## Calibration

See the [wiki](https://github.com/RobInLabUJI/ricoh_camera/wiki/Calibration).

## Testing

Connect the camera to the computer in [Live Streaming](https://support.theta360.com/en/manual/s/content/streaming/streaming_01.html) mode and run:
```
roslaunch ricoh_camera capture.launch
```

A window will display the images of both fish-eye cameras (front and back).

Default configuration:
* Input device: `/dev/video0`
* Resolution: 1280 x 720
* Frequency: 30 Hz

You can also [test the package with Docker](https://github.com/RobInLabUJI/ricoh_camera/wiki/Use-with-Docker).

Notes:
* With USB connection the frequency is limited to 15 Hz
* With HDMI to USB3 connection the resolution can be set to 1920 x 1080 at 30 Hz

### Visualization with rviz

This package can be used for visualization on a skysphere with rviz: [https://github.com/RobInLabUJI/rviz_textured_sphere](https://github.com/RobInLabUJI/rviz_textured_sphere)

Clone and build the package in your workspace:
```
cd <YOUR_WORKSPACE>/src
git clone https://github.com/RobInLabUJI/rviz_textured_sphere.git
cd ..
catkin_make
```

Launch the camera streaming and visualization with:
```
roslaunch ricoh_camera capture.launch
roslaunch rviz_textured_sphere demo.launch
```

Note: the images are projected without any calibration.

## Equirectangular projection
Test with live images:
```
roslaunch ricoh_camera test_eqrect_python_live.launch
```

Test with a bag file (set the path in the launch file):
```
roslaunch ricoh_camera test_eqrect_python_bag.launch
```

Calibration files in:
* `config/front_camera_calib.yaml`
* `config/back_camera_calib.yaml`

### Example notebook
Requisites:
* [`jupyter notebook`](https://jupyter.org), install with:
```
python -m pip install --upgrade pip
python -m pip install jupyter
```

* `matplotlib>2.0.0`, install with:
```
python -m pip install --upgrade --user matplotlib
```

Run the notebook with:
```
jupyter notebook `rospack find ricoh_camera`/notebooks/Equirectangular_Projection.ipynb
```


text:
# Matrix Distribution

The matrix (also denoted global matrix) is the matrix which is distributed using a 2D block cyclic distribution scheme.
Therefore any of the rank stores in its own memory only a subset of the elements which compose the local matrix.
The matrix is divided in submatrices of the same size (exceptions are the submatrices in the boundaries which can be smaller) called tiles.

## Global to local mapping

The global matrix of size `size` (denoted by blue lines in Fig. 1) is divided in blocks of size `block_size` (red lines).
The blocks are then distributed cyclically among the processes which are mapped in a 2D grid `grid_size`. In Fig. 1 there are 6 processes on a (3x2) grid.

![Fig. 1 Distribution of a matrix. Each color represents a process.](figures/distribution.png)
*Fig. 1 Distribution of a matrix. Each color represents a process.*

An index is used to indicate the source rank `source_rank_index`, i.e. the rank which has the top left block of the matrix.
For example, Fig. 2 illustrates how the first (4x4) blocks are distributed for different source rank.

![Fig. 2. Mapping of blocks and processes with different source rank. On the left the source rank is {0, 0}, in the center is {2, 1} and on the right is {1, 0}.](figures/src_rank.png)
*Fig. 2. Mapping of blocks and processes with different source rank. On the left the source rank is {0, 0}, in the center is {2, 1} and on the right is {1, 0}.*

Fig. 3 shows the local matrix of size `local_size` (green lines in Fig. 3) of each of the 6 processes.

![Fig. 3. Local matrix of each process for the matrix depicted in Fig 1.](figures/local.png)
*Fig. 3. Local matrix of each process for the matrix depicted in Fig 1.*

## Distribution constraints

The following constraints hold:
- Each tile is stored in the memory of a single rank.
- A tile is entirely stored in the memory of a single rank (it cannot be shared/distributed among multiple ranks).
- Each block of the 2D block cyclic distribution is a tile (`tile_size == block_size` for tiles not in the boundaries, `tile_size.rows() <= block_size.rows() && tile_size.cols() <= block_size.cols()` otherwise).
- Each tile is stored using column major layout.

## Naming conventions

Strong type 2D indices and sizes are used to reduce the possibility of index mixing errors.
The following naming convention is used for the types of indices and sizes:
- `GlobalElementIndex`: index of an element in a global matrix.
- `GlobalTileIndex`: index of a tile in a global matrix.
- `LocalTileIndex`: index of a tile in a local matrix.
- `TileElementIndex`: index of in element in a tile.
- `GlobalElementSize`: the size of a global matrix in elements.
- `LocalElementSize`: the size of a local matrix in elements.
- `GlobalTileSize`: the size of a global matrix in tiles.
- `LocalTileSize`: the size of a local matrix in tiles.
- `TileElementSize`: the size of a tile in elements.

Warning: 1D indices and sizes have all the same type (`SizeType`).

## Index manipulation

Utilities exist to transform global indices to local indices and viceversa.
The "Next" versions of conversions from global to local indices return the corresponding local index of the global index if the global index is the given process,
otherwise they return the local index corresponding to the next global index in the given process.

The following table shows how a component of GlobalElementIndex transforms for `block_size = 3`, `grid_size = 3`, `source_rank_index = 1`.

| GlobalElementIndex        |  0 |  1 |  2 |  3 |  4 |  5 |  6 |  7 |  8 |  9 | 10 | 11 | 12 | 13 | 14 | 15 |
|---------------------------|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| GlobalTileIndex           |  0 |  0 |  0 |  1 |  1 |  1 |  2 |  2 |  2 |  3 |  3 |  3 |  4 |  4 |  4 |  5 |
| CommunicatorRank          |  1 |  1 |  1 |  2 |  2 |  2 |  0 |  0 |  0 |  1 |  1 |  1 |  2 |  2 |  2 |  0 |
| LocalElementIndex Rank 0  |    |    |    |    |    |    |  0 |  1 |  2 |    |    |    |    |    |    |  3 |
| LocalElementIndex Rank 1  |  0 |  1 |  2 |    |    |    |    |    |    |  3 |  4 |  5 |    |    |    |    |
| LocalElementIndex Rank 2  |    |    |    |  0 |  1 |  2 |    |    |    |    |    |    |  3 |  4 |  5 |    |
| LocalTileIndex Rank 0     |    |    |    |    |    |    |  0 |  0 |  0 |    |    |    |    |    |    |  1 |
| LocalTileIndex Rank 1     |  0 |  0 |  0 |    |    |    |    |    |    |  1 |  1 |  1 |    |    |    |    |
| LocalTileIndex Rank 2     |    |    |    |  0 |  0 |  0 |    |    |    |    |    |    |  1 |  1 |  1 |    |
| NextLocalTileIndex Rank 0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  1 |  1 |  1 |  1 |  1 |  1 |  1 |
| NextLocalTileIndex Rank 1 |  0 |  0 |  0 |  1 |  1 |  1 |  1 |  1 |  1 |  1 |  1 |  1 |  2 |  2 |  2 |  2 |
| NextLocalTileIndex Rank 2 |  0 |  0 |  0 |  0 |  0 |  0 |  1 |  1 |  1 |  1 |  1 |  1 |  1 |  1 |  1 |  2 |
| TileElementIndex          |  0 |  1 |  2 |  0 |  1 |  2 |  0 |  1 |  2 |  0 |  1 |  2 |  0 |  1 |  2 |  0 |

`include/dlaf/matrix/util_distribution.h` contains the functions to transform the indices for any distribution (distribution details are arguments of the functions),
while `include/dlaf/matrix/distribution.h` provides index conversion member functions for the distribution object.

# Local Matrix Layout

The layout of the matrix represents the map which gives the position in the memory of each element of the local matrix.
It is used for compatibility with other libraries and existing applications.
The matrix layout can describe both a column-major matrix or a matrix stored in tile layout.

![Fig. 4. Left: Layout of a column major matrix. Right: Mapping of a tile layout matrix.](figures/layout1.png)
*Fig. 4. Left: Layout of a column major matrix. Right: Mapping of a tile layout matrix.*

![Fig. 5. Left: Layout of a column major matrix. Right: Mapping of a tile layout matrix.](figures/layout2.png)
*Fig. 5. Left: Layout of a column major matrix. Right: Mapping of a tile layout matrix.*

Fig. 4 and 5 show some examples of layouts. The colored and black dots represent each an element of the matrix.
The grey dots are positions in the memory which are never accessed.

The layout is described by 3 parameters which are indicated in Fig 4:
- `tile_leading_dimension`: the distance in the memory of tile element `(i,j)` and `(i,j+1)` (red line),
- `tile_row_offset`: the distance between the first element of the (i,j) tile and the first element of the tile in the next row, i.e. tile (i+1,j) (blue line),
- `tile_col_offset`: the distance between the first element of the (i,j) tile and the first element of the tile in the next column, i.e. tile (i,j+1) (green line),

repository: eth-cscs/DLA-Future

repo_name: eth-cscs/DLA-Future


text:
# StatusBar

Report messages of minor importance to the user. Maintain a **stack of messages**.The message at the top of the stack is the one that will be displayed.

repository: jsuarezruiz/FormsGtkToolkit

gha_language: C#

repo_name: jsuarezruiz/FormsGtkToolkit


text:
# Space Age Calculator

This is a project created for the Epicodus code challenge for week 1 of Javascript.

## Prerequisites

Prior to cloning, please ensure that you have Node.js installed on your machine.

### Installing

Navigate to the project repository on Github: https://github.com/rpgandhi/JavaScriptAgeCalculator.git

Select the green Clone button and copy the link

Open Terminal or your shell of choice

Navigate to your desktop and run the following command:
```
$ git clone
```
Navigate into the project directory and run the following commands:
```
$ npm install
```
```
$ bower install
```
```
$ gulp build
```

You shoud now have a copy fully installed on your machine.

Then inn order to laungh the project in your browser, run the following command which will launch the server then in your browser naviagate to localhost:3000
```
$ gulp serve
```
When you are ready to shut down your server press CTL + C

## Running the tests

In order to run the tests and lauch the test runner (Karma), run the following command in your terminal:
```
$ npm test
```
A Karma test runner window should pop up.
Click the "debug" button on the right hand side to see the specs running.

## Built With

* JavaScript
* HTML
* Node.js

## Authors

* **Rakhee Gandhi**

## License

This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details

repository: rpgandhi/JavaScriptAgeCalculator

repo_name: rpgandhi/JavaScriptAgeCalculator


text:
# sigsel.js
[![npm](https://img.shields.io/npm/v/sigsel.svg)](https://www.npmjs.org/package/sigsel)
[![size](https://img.shields.io/bundlephobia/minzip/sigsel.svg)](https://bundlephobia.com)
[![deps](https://david-dm.org/rliang/sigsel/status.svg)](https://david-dm.org/rliang/sigsel)

Selectable signals.

Inspired by Go's channels and Clojure's `core.async`,
this library aims to make concurrent programming easier.

Since JS is single-threaded,
there can be no data races across synchronous calls.
This means we do not need channels and can just share variables.
However, we still want a mechanism for synchronizing asynchronous operations.
For that purpose, we use a `signal` object.

## Installation

```sh
npm i sigsel
```

## Usage

### Counter

```jsx
import { signal, select } from "sigsel";

(async () => {
  let counter = 0;
  const click = signal();
  const timeout = signal();
  for (;;) {
    const timer = setTimeout(timeout, 1000);
    render(<button onclick={click}>{counter}</button>);
    switch (await select(click, timeout)) {
      case click:
        counter++;
        break;
      case timeout:
        counter--;
        break;
    }
    clearTimeout(timer);
  }
})();
```


text:
# Basic setup for all firmware examples

All firmware examples in this documentation are based on [Mbed OS](https://os.mbed.com).

!!! info "Tools"
    These steps assume that `mbed-cli` and [GNU Arm Embedded Toolchain](https://developer.arm.com/open-source/gnu-toolchain/gnu-rm/downloads) are already installed. For instructions on tooling setup visit [Tooling setup](./tools.md).

## Using template repository

The easiest way to start is to clone [SensorIO template](https://github.com) repository:

```bash
git clone https://github.com/slashdevteam/sensorio-template.git sensorio-example
```

and initialize submodules:

```bash
cd sensorio-example
git submodule update --init --recursive
```

add any necessary drivers, for example for light sensor:

```bash
git clone https://github.com/slashdevteam/ltr303-mbed.git
```

edit `main.cpp`, [compile](./compilation.md) and [flash](./flashing.md).

## Manually

Steps needed to compile Mbed OS based firmware for SensorIO:

1. Create a new directory for your project, for example:

    ```bash
    mkdir sensorio-example
    ```

2. Enter this new directory, for example:

    ```bash
    cd sensorio-example
    ```

3. Get Mbed OS source:

    ```bash
    git clone https://github.com/slashdevteam/mbed-os.git`
    git checkout -t origin/sensorio-dev-target
    ```

    !!! warning "Mbed OS repository"
        SensorIO is currently in the process of Mbed Enabled certification, hence the need to clone a fork of Mbed OS.

4. Get appropriate drivers, for example:

    ```bash
    git clone https://github.com/slashdevteam/ltr303-mbed.git
    ```

5. Download or copy-paste example `main.cpp` into `sensorio-example` directory.

6. Download compilation profile: [release](./assets/release.json) or [debug](./assets/debug.json).

7. Initialize Mbed OS project:

    ```bash
    mbed config root .
    ```

8. Select SensorIO as build target:

    ```bash
    mbed target SENSORIO
    ```

9. Select GNU GCC Toolchain as default:

    ```bash
    mbed toolchain GCC_ARM
    ```

10. Compile firmware example:

    ```bash
    mbed compile --profile debug.json
    ```

11. [Flash binary](./flashing.md) and enjoy!

repository: makerfactoryio/docs

gha_language: HTML

repo_name: makerfactoryio/docs