text:
# SQL Jobber

![test workflow](https://github.com/knadh/sql-jobber/actions/workflows/test.yml/badge.svg)
![release workflow](https://github.com/knadh/sql-jobber/actions/workflows/release.yml/badge.svg)

sql-jobber is a light weight SQL "job server" that maintains a distributed, asynchronous job queue of SQL read jobs against one or more large source databases. The results are written to one or more separate "cache" databases, where each result set is a newly created table, from where the results can be fetched as many times much faster than querying the source databases.

This is useful for queuing and offloading report generation on applications without clogging source databases, especially in the case of end user applications.

## Parts

- Supports MySQL and PostgreSQL as source databases.
- Supports MySQL and PostgreSQL as result / cache databases for job responses.
- Standalone server that exposes HTTP APIs for managing jobs and groups of jobs (list, post, status check, cancel jobs).
- Reads SQL queries from .sql files and registers them as jobs ready to be queued.
- Written in Go and built on top of [Machinery](https://github.com/RichardKnop/machinery). Supports multi-process, multi-threaded, asynchronous distributed job queueing via a common broker backend (Redis, AMQP etc.)

## Usecase

An application that has a very large SQL database, when there are several thousand concurrent users requesting reports from the database simultaneously, every second of IO delay in query execution locks up the application's threads, snowballing and overloading the application.

Instead, defer every single report request into a job queue, there by immediately freeing up the front end application. The reports are presented to users as they're executed (frontend polls the job's status and prevents the user from sending any more queries). Fixed SQL Jobber servers and worker threads also act as traffic control and prevent the primary database from being indundated with requests.

Once the reports are generated, it's natural for users to further transform the results by slicing, sorting, and filtering, generating additional queries to the primary database. To offset this load, these subsequent queries can be sent to the smaller, much faster results cache database. These results are of course ephemeral and can be thrown away or expired.

![sql-job-server png](https://user-images.githubusercontent.com/547147/44912100-d3f27b80-ad46-11e8-9938-2b6c0f974488.png)

## Concepts
#### Task
A task is a named SQL job is loaded into the server on startup. Tasks are defined in .sql files in the simple [goyesql](https://github.com/knadh/goyesql) format. Such queries are self-contained and produce the desired final output with neatly named columns. They can take arbitrary positional arguments for execution. A task can be attached to one or more specific databases defined in the configuration using the `-- db:` tag. In case of multiple databases, the query will be executed against a random one from the list, unless a specific database is specified in the API request (`db`). A `-- queue:` tag to always route the task to a particular queue, unless it's overriden by the `queue` param when making a job request. A `-- results:` tag specifies the results backend to which the results of a task will be written. If there are multiple result backends specified, the results are written a random one.

Example:
```sql
-- queries.sql

-- name: get_profit_summary
SELECT SUM(amount) AS total, entry_date FROM entries GROUP BY entry_date WHERE user_id = ?;

-- name: get_profit_entries
-- db: db0, other_db
-- queue: myqueue
-- results: my_res_db
SELECT * FROM entries WHERE user_id = ?;

-- name: get_profit_entries_by_date
SELECT * FROM entries WHERE user_id = ? AND timestamp > ? and timestamp < ?;

-- name: get_profit_entries_by_date
-- raw: 1
-- This query will not be prepared (raw=1)
SELECT * FROM entries WHERE user_id = ? AND timestamp > ? and timestamp < ?;
```

Here, when the server starts, the queries `get_profit_summary` and `get_profit_entries` are registered automatically as tasks. Internally, the server validates and prepares these SQL statements (unless `raw: 1`). `?` are MySQL value placholders. For Postgres, the placeholders are `$1, $2 ...`

#### Job
A job is an instance of a named task that has been queued to run. Each job has an ID that can be used to track its status. If an ID is not passed explicitly, it is generated internally and returned. These IDs need not be unique, but only a single job with a certain ID can run at any given point. For the next job with the same ID to be scheduled, the previous job has to finish execution. Using non-unique IDs like this is useful in cases where users can be prevented from sending multiple requests for the same reports, like in our usecases.

An application polls with the job ID to check if results are ready for consumption.

#### Results
The results from an SQL query job are written to a result backend (MySQL or Postgres) from where they can be further read or queried. This is configured in the configuration file. The results from a job are written to a new table named after that job, where schema of the results table is automatically generated from the results of the original SQL query. All fields are transformed into one of these types `BIGINT, DECIMAL, TIMESTAMP, DATE, BOOLEAN, TEXT`.

## Installation
A pre-compiled binary can be downloaded from the [releases](https://github.com/knadh/sql-jobber/releases) page.

### 2) Configure
Copy the `config.toml.sample` file as `config.toml` somewhere and edit the configuration values.

### 3) Setup tasks
Write your SQL query tasks in `.sql` files in the `goyesql` format (as shown in the examples earlier) and put them in a directory somewhere.

### 4) Start the server
```shell
sql-jobber --config /path/to/config.toml --sql-directory /path/to/your/sql/queries

# Run 'sql-jobber --help' to see all supported arguments
```

Starting the server runs a set of workers listening on a default job queue. It also starts an HTTP service on `http://<IP_ADDRESS>:6060` which is the control interface. It's possible to run the server without the HTTP interface by passing the `--worker-only` flag.

### Usage
| Method | URI                    |                                                 |
|--------|------------------------|-------------------------------------------------|
| GET    | /tasks                 | Returns the list of registered SQL tasks        |
| POST   | /tasks/{taskName}/jobs | Schedules a job for a given task                |
| GET    | /jobs/{jobID}          | Returns the status of a given job               |
| GET    | /jobs/queue/{queue}    | Returns the list of all pending jobs in a queue |
| POST   | /groups                | Schedule a group of jobs                        |
| GET    | /groups/{groupID}      | Get the status of a job group and its jobs      |
| DELETE | /jobs/{jobID}          | Deletes a pending job from the queue and immediately cancels its execution and frees the thread. Send a query param "purge=true" to delete completed jobs. Only the Go PostgreSQL driver cancels queries mid execution. MySQL server will keep continuing to execute the query. For MySQL, it's important to set `max_execution_time`.   |
| DELETE | /groups/{groupID}          | Deletes a pending job from the queue and immediately cancels its execution and frees the thread. Send a query param "purge=true" to delete completed jobs. Only the Go PostgreSQL driver cancels queries mid execution. MySQL server will keep continuing to execute the query. For MySQL, it's important to set `max_execution_time`  |

POST requests accept raw JSON bodies. The JSON params are listed below.

| Job param     |                                                                                |   |
|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------|---|
| `job_id` *string*      | (Optional) Alphanumeric ID for the job. Can be non-unique. If this is not passed, the server generates and returns one                            |   |
| `queue` *string*     | (Optional) Queue to send the job to. Only workers listening on this queue will receive the jobs.                                                                                |   |
| `eta` *string*         | (Optional) Timestamp (`yyyy-mm-dd hh:mm:ss`) at which the job should start. If this is not provided, the job is queued immediately.               |   |
| `retries` *int*     | (Optional) The number of times a failed job should be retried. Default is 0                                                                      |   |
| ~~`ttl`~~ *int*     | (Optional) TTL for the results in the results backend for this particular job. This is NOT supported by the default SQL DB result backend |   |
| `args[]` *[]string*         | (Optional) The positional argument to pass to the SQL query in the task being executed. This can be passed multiple times, one for each argument |   |

| Group param     |                                                                                |   |
|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------|---|
| `group_id` *string*      | (Optional) Alphanumeric ID for the group of jobs. Can be non-unique. If this is not passed, the server generates and returns one                            |   |
| `concurrency` *int*     | (Optional) Number of jobs to run concurrently in the group

##### Schedule a job
```shell
$ curl localhost:6060/tasks/get_profit_entries_by_date/jobs -H "Content-Type: application/json" -X POST --data '{"job_id": "myjob", "args": ["USER1", "2017-12-01", "2017-01-01"]}'

{"status":"success","data":{"job_id":"myjob","task_name":"get_profit_entries_by_date","queue":"sqljob_queue","eta":null,"retries":0}}
```

##### Schedule a group of jobs
Sometimes, it's necessary to schedule a group of jobs and perform an action once they're all complete. Group jobs here run concurrently and independent of each other. The group state can be polled to figure out if all the jobs in it have finished executing.

```shell
$ curl localhost:6060/groups -H "Content-Type: application/json" -X POST --data '{"group_id": "mygroup", "concurrency": 3, "jobs": [{"job_id": "myjob", "task": "get_profit_entries_by_date", "args": ["USER1", "2017-12-01", "2017-01-01"]}, {"job_id": "myjob2", "task": "get_profit_entries_by_date", "args": ["USER1", "2017-12-01", "2017-01-01"]}]'

{"status":"success","data":{"group_id":"mygroup","jobs":[{"job_id":"myjob","task":"test1","queue":"sqljob_queue","eta":null,"retries":0},{"job_id":"myjob2","task":"test2","queue":"sqljob_queue","eta":null,"retries":0}]}}
```

##### Check a job's status
```shell
$ curl localhost:6060/jobs/myjob
{"status":"success","data":{"job_id":"myjob","status":"SUCCESS","results":[{"Type":"int64","Value":2}],"error":""}}~

# `Results` indicates the number of rows generated by the query.
```

## Advanced usage
### Multiple queues, workers, and job distribution
It's possible to run multiple workers on one or more machines that run different jobs with different concurrency levels independently of each other using different queues. Not all of these instances need to expose the HTTP service and can run as `--worker-only`. This doesn't really make a difference as long as all instances connect to the same broker backend. A job posted to any instance will be routed correctly to the right instances based on the `queue` parameter.

Often times, different queries have different priorities of execution. Some may need to return results faster than others. The below example shows two SQL Jobber servers being run, one with 30 workers and one with just 5 to process jobs of different priorities.

```shell
# Run the primary worker + HTTP control interface
sql-jobber --config /path/to/config.toml --sql-directory /path/to/sql/dir \
	--queue "high_priority" \
    --worker-name "high_priority_worker" \
    --worker-concurrency 30

# Run another worker on a different queue to handle low priority jobs
sql-jobber --config /path/to/config.toml --sql-directory /path/to/sql/dir \
	--queue "low_priority" \
    --worker-name "low_priority_worker" \
    --worker-concurrency 5 \
    --worker-only

# Send a job to the high priority queue.
$ curl localhost:6060/tasks/get_profit_entries_by_date/jobs -H "Content-Type: application/json" --data '{"job_id": "myjob", "queue": "high_priority", "args": ["USER1", "2017-12-01", "2017-01-01"]}'

# Send another job to the low priority queue.
$ curl localhost:6060/tasks/get_profit_entries_by_date/jobs -H "Content-Type: application/json" --data '{"job_id": "myjob", "queue": "low_priority"}'
```

## API client
`github.com/knadh/sql-jobber/client` package can be used as a Go HTTP API client for sql-jobber.

## License
Licensed under the MIT License.

repository: knadh/sql-jobber

gha_language: Go

repo_name: knadh/sql-jobber


text:
# Development Guide

<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
**Table of Contents**  *generated with [DocToc](https://github.com/thlorenz/doctoc)*

- [Development Guide](#development-guide)
    - [Required tools/Binaries](#required-tools/binaries)
    - [Launch Dev mode](#launch-dev-mode)
    - [Build a local image](#build-a-local-image)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

## Required tools/Binaries

Multicloud-Operators projects are built with following tools. Use the links provided to install them on your environment:

 - Find an environment, or set up Kubernetes version 1.13 and later: [kubernetes.io](https://kubernetes.io/docs/setup/#learning-environment)

 - Build with [Go verson 1.13 and later](https://golang.org/dl/)

 - Enforce style and formatting with Lint:

    - [hadolint](https://github.com/hadolint/hadolint#install)
    - [shellcheck](https://github.com/koalaman/shellcheck#installing)
    - [yamllint](https://github.com/adrienverge/yamllint#installation)
    - [helm client](https://helm.sh/docs/using_helm/#install-helm)
    - [golangci-lint](https://github.com/golangci/golangci-lint#install)
    - [autopep8](https://github.com/hhatto/autopep8#installation)
    - [mdl](https://github.com/markdownlint/markdownlint#installation)
    - [awesome_bot](https://github.com/dkhamsing/awesome_bot#installation)
    - [sass-lint](https://github.com/sds/scss-lint#installation)
    - [tslint](https://github.com/palantir/tslint#installation--usage)
    - [prototool](https://github.com/uber/prototool/blob/dev/docs/install.md)
    - goimports -  Run `go get -v golang.org/x/tools/cmd/goimports` to install

 - Test with [kubebuilder v1.0.8](https://github.com/kubernetes-sigs/kubebuilder/releases/tag/v1.0.8)

## Launch Dev mode

Run the following command to launch developer mode:

```shell
git clone git@github.com:open-cluster-management/multicloud-operators-deployable.git
cd multicloud-operators-deployable
export GITHUB_USER=<github_user>
export GITHUB_TOKEN=<github_token>
make
make build
./build/_output/bin/multicluster-operators-deployable
```

## Build a local image

Build a local image by running the following command:

```shell
git clone git@github.com:open-cluster-management/multicloud-operators-deployable.git
cd multicloud-operators-deployable
export GITHUB_USER=<github_user>
export GITHUB_TOKEN=<github_token>
make
make build-images
```

branch_name: refs/heads/main

extension: md

license_type: permissive


text:
##  Installation and testing the camera

The camera is an optional feature and allows you to scan LNURL addresses. The 5MP Camera OV5647 is available from different manufacturers and with slightly different designs.
To test the function of the camera you can take a picture and download it to you computer.

### Installation

The ribbon cable is a bit special as the connectors on the camera and the Raspberry Pi are slightly different sizes. So the cable tapers.
Before sliding the cable into the slot, the black latch must be released by gently pulling upwards.
The cable can then simply be pushed in. The silver contacts must point away from the detent. See image!
When the cable is centered, you can lock the latches in place by sliding them back slightly.

connection

### Activate camera

- Login to the Raspberry Pi

      $ ssh<EMAIL_ADDRESS>
- Directory displayed: `pi@raspberrypi:~ $`

- Call up the Raspi-Config and activate the camera

      $ sudo raspi-config

- Choose: `Interfacing Options  Configure connections to peripherals`
- Choose: `Camera      Enable/Disable connection to the Raspberry Pi Camera`
- Confirm: `Would you like the camera interface to be enabled?` -> `\<YES>`
- Confirm: `The camera interface is enabled` -> `\<OK>`
- Go To: `\<Finish>`
- When ask for: `Would you like to reboot now?` -> `\<YES>`
- If you have not already restarted from the menu, then restart manually to activate the changes

      $ sudo reboot

raspi-config

![](../pictures/camera_raspi-config_1.png)
![](../pictures/camera_raspi-config_2.png)

  Your config menu can look slightly different, depends of you hard- and software

### Take a picture

- Take a picture after logging in to the Raspberry Pi

      $ raspistill -v -o test.jpg

  Note: The -v is only optional and will show more display data. The -o is necessary to write the file.

The recording lasts 5 seconds (time delay 5000 (ms)). You can shorten the recording time, but you should not make the time too short, otherwise the picture quality will suffer. One second (= 1000 ms) is definitely possible. The command then looks like:

   $ raspistill -v -o test.jpg -t 1000

  Further functions for a recording can be found in the help `$ raspistill --help`

- Check if the image has been saved

      $ ls -l

  The file `test.jpg` should now appear in the list.

- Check the directory structure

      $ pwd

  The directory should read: `/home/pi/`

 raspistill command

 ### Transfer the image to the computer

 - Open a *second terminal window* on the computer
 - Switch to a folder of your choice (example `C:\temp`)
 - The command to copy for a Windows system is

       $ scp pi@192.168.x.x:/home/pi/test.jpg .

   -> The password from the Raspberry Pi must be entered for comfirmation

 - If everything went well, the image was transferred to the computer and can now be vieweden

 - Note: The command is slightly different on a Mac or Linux system

       $ scp 'pi@192.168.x.x:/home/pi/test.jpg' ./

scp command

Example of a recording

The structure            |  The picture
:-------------------------:|:-------------------------:
![](../pictures/camera_example_structure.jpg)  |  ![](../pictures/camera_example_image.jpg)

Note: This camera was pretty cheap and has a variable lens that was pretty loose and difficult to adjust. You might want to use a better one for your project.

### Aditional hints

- If you want to take several pictures, the pictures must be numbered consecutively. E.g. test1.jpg, test2.jpg etc.
- Then you can transfer all images at once

      $ scp pi@192.168.x.x:/home/pi/*.jpg .   or    scp 'pi@192.168.x.x:/home/pi/*.jpg' ./

- You can do also videos. The command is `raspivid`

---

#### [option: button](/docs/guide/button.md)  ᐊ  previous | next  ᐅ  [option: lockout relay](/docs/guide/relay.md)

license: MIT

branch_name: refs/heads/master

gha_language: Python

gha_license_id: MIT


text:
# Missing Migrants Visualization

[Click here](https://dbeach24.github.io/missing-migrants-vis/) for a live demo of the visualization.

## Overview

This project is a visualization of a dataset I originally discovered on [Kaggle](https://www.kaggle.com/), called [Missing Migrants](https://www.kaggle.com/jmataya/missingmigrants).  The dataset describes events where migrants have been discovered dead or missing, and aggregated from a combination of media, medical, and governmental reports from around the world.    This project is attempt to clean this data and, hopefully, to shine a light on certain aspects of this problem in a way that communicates something of interest to the casual reader.

In particular, this project attempts to aid in answering the following questions:

* How does the number of missing / dead vary over time and by region?
* Are there certain locations that exhibit many small events which accumulate over time?
* What are the different causes of migrant death?
* How does cause of death vary by region?

## Using the visualization

### Stream Graph

The stream graph on the left hand size provides an overview of the missing migrant activity.  This graph is computed using the per-month totals of worldwide migrants deaths per cause.  This view is static, but brushing provides a linked brush-and-zoom interaction with the main bubble chart view.

### Bubble Chart

The bubble chart view shows individual migrant death/missing events.  Each event is marked by a circle where:

  * x-axis position = longitude of the event,
  * y-axis position = time of event,
  * area = number of migrants affected (dead + missing), and
  * color = indicates the dominant cause for this event.

You can use the mouse wheel to zoom in around a particular time, and use clicking and dragging to pan through time.  This will update the brushed selection region shown in the stream graph on the left.

From the bubble chart data, you can see that the events cluster into major vertical bands.  From left-to-right, these correspond to:

  * Americas
  * Europe, Middle East, and Africa, and
  * Asia Pacific.

### Cause Legend

The cause legend uses custom icon markers and colors to indicate six major causal categories.  Each icon's color corresponds to the color used in the bubble chart.  Hovering over a category will perform a highlight-by-category interaction, for example:

In the above view, the "Drowning" cause has been selected.  This emphasizes the marks related to drowning in both the bubble chart and the stream graph, and deemphasizes all marks corresponding to other causes.  This should provide a "pop-out" effect for the selected data.

### Size Legend

The size legend is a static element which aids in interpreting the size of circles in the bubble chart.

### Details

Hovering over any circle in the bubble chart will cause a details pop-up to appear.  The details pop-up gives specific information, including:

  * event date,
  * number of dead and missing migrants,
  * all causal categories matched by this event (as icons),
  * the text location of the event,
  * a text description of the cause,
  * the nationalities of the migrants (when known),
  * the region of origin of the migrants (may be different from nationality),
  * the reporting source, and
  * whether the event is `Verified`, `Partially Verified`, or `Unverified`.

## Some Remarks

The visualization is intended to aid in answer several questions.  Here, I provide my own interpretation of the answers based on developing and using the visualization tool:

* How does the number of missing / dead vary over time and by region?

We can see three primary regions of activity.  In order of significance, these are:

1. Europe, Middle East and Africa,
2. Americas, and
3. Asia Pacific.

* Are there certain locations that exhibit many small events which accumulate over time?

Yes, there are many small events.  However these events tend to occur in the same locations as larger events.  In certain parts of the Americas, it appears that there are years where small events were omitted from the data in lieu of an end-of-year summary event.

* What are the different causes of migrant death?

Drowning (especially in the Mediterranean) is the \#1 cause of migrant deaths.  All other causes tend to be much smaller.

* How does cause of death vary by region?

In Europe and the Middle East, most deaths are due to drowning.  This is also largely true for the Asia Pacific, but with some noteworthy exceptions.  The data from the Americas tends not to have the cause of death identified, however deaths from all causes can be seen if one looks closely enough.

## About the Data

This dataset is from the [Missing Migrants](https://www.kaggle.com/jmataya/missingmigrants) data uploaded to Kaggle, and contains information about people who have gone missing while traveling along migration routes.  This data originates from the [Missing Migrants Project](https://missingmigrants.iom.int/).

The original data describe events where migrants have gone missing, and generally answer the following questions:

* Who – Number of people, Nationality, Region of Origin
* What – Dead or Missing
* Where – latitude, longitude, Incident Region
* When – Date
* How – Cause of Death

The data sets has 12 dimensions, and includes the following fields:

| Field            | Kind          | Notes |
| ---------------- | ------------- | ----- |
| ID               | Categorical   | Unique Event Identifier |
| Cause of Death   | Categorical   | Free-form text requires classification into categories to be useful |
| Region of Origin | Categorical   | Inconsistent encoding; needs cleaning |
| Nationality      | Categorical   | Origin nation names with mixed text; needs cleaning |
| Missing          | Quantitative  | Number of missing |
| Dead             | Quantitative  | Number of dead |
| Incident Region  | Categorical   | Mostly clean, but can also be computed from lat/lon |
| Date             | Quantitative  | European-style date in DD/MM/YYYY format |
| Source           | Categorical   | Fairly high cardinality (603), may be hard to reduce |
| Reliability      | Categorical   | Mostly clean: Verified, Partially Verified, Unverified |
| Latitude         | Quantitative  | Degrees latitude |
| Longitude        | Quantitative  | Degrees longitude |

## Enrichment and Feature Extraction

A script (included as a Jupyter/Python notebook is used to extract additional features from the data:

| Field            | Kind             | Notes |
| ---------------- | ---------------- | ----- |
| Region Num       | Categorical, Int | Region, encoded as a number, based on longitude.

0 = Americas (longitude < -50)

1 = EMEA (-50 <= longitude < 75)

2 = Southeast Asia (longitude > 75)

 |
| Cause            | Categorical, Set | Text pattern extraction from “Cause of Death” field, into categories:

Drowning/Asphyxiation

Exposure / Dehydration / Starvation

Vehicular / Mechanical

Homicide / Violence

Illness / Medical

Unknown

| Nationalities    | Categorical, Set | Extracted country names from “Nationality” text field.  Also needs to recognize region names (such as North Africa, Central America, or MENA, and handle appropriately). |

## Earlier work

### First Prototype

An [earlier example of this work](http://blockbuilder.org/dbeach24/599725c960ce2881b1dcd46c3b20e268) can be found on [bl.ocks.org](https://bl.ocks.org).

### Design Sketches

## Attribution

This `D3.js` web project is based on [this template project](https://github.com/curran/dataviz-project-template) from [curran](https://github.com/curran).

Elements of the map projection were taken from [Mike Bostock's](https://bl.ocks.org/mbostock) [Equirectangular (Plate Carrée) Block](https://bl.ocks.org/mbostock/3757119).

gha_language: Jupyter Notebook


text:
# Trello Birds-eye View

> Chrome Extension for minimalistic board/card styling giving you a birds-eye view in Trello

![Before & after](images/screenshot-comparison.gif)

**Why?**
Trello is great, but if you have a lot of cards on your boards, the interface becomes a mess. Too much bulky chrome, borders, margins and paddings. Wouldn't you sometimes wish you could just chip away all the unnecessary junk?

**How?**
This small Chrome Extension adds an extra button to the Trello toolbar. Click it and you get more than *50% more actual content on your screen*.

## Install

[**Add the extension to Chrome** from the webstore](https://chrome.google.com/webstore/detail/trello-birdseye/jddlcajmgfdpfnekdppnapoldfhabkeg)

That's it. Look for the "Birds Eye" button in the boards toolbar in Trello :+1:

repository: Ishamyyl/trello-birdseye

repo_name: Ishamyyl/trello-birdseye


text:
+++
Title = "Propose"
Type = "event"
Description = "Propose a talk for devopsdays Amsterdam 2018"
+++
  {{< cfp_dates >}}

  ### Equal Opportunity

  devopsdays Amsterdam is committed to creating a diverse environment and is proud to invite people of all backgrounds and profiles to submit presentation proposals. All potential speakers will receive consideration without regard to race, colour, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, ability or age.

  For more information, please see our published {{< event_link page="conduct" text="Code of Conduct" >}}.

  ### Proposals

  There are three ways to propose a session:

  1. __A proposal for a talk/panel__ during the conference part : these are 30 minute slots that will have the full attention of all attendees, as everybody will be in that one room.
  2. __An Ignite talk__ that will be presented during the Ignite sessions. These are 5 minutes slots with slides changing every 15 seconds (20 slides total) which are also presented to all attendees in one room
  3. __A Workshop Session__ that will be given during the Workshop day on June 27 presented as a hands-on sessions in a classroom setting on a particular topic, generally on an opensource technology. These are 2.5 hour slots with approximately 20 to 50 participants.

  Choosing talks is part art, part science; here are some factors we consider when trying to assemble the best possible program for our local audience:

  - __Broad appeal__: How will your talk play out in a room of people with a variety of backgrounds? Technical deep dives need more levels to provide value for the whole room, some of whom might not use your specific tool.
  - __New local presenters__: You are the only one who can tell your story. We are very interested in the challenges and successes being experienced in our local area. We are happy to provide guidance/coaching for new speakers upon request.
  - __Under-represented voices__: We want to hear all voices, including those that may speak less frequently at similar events. Whether you're in a field not typically thought of as a technology field, you're in a large, traditional organization, or you're the only person at your organization with your background, we are interested in your unique experience.
  - __Original content__: We prefer talks not already presented at another event, especially another one in the local area or one available in recorded form.
  - __No third-party submissions__: If a PR firm or your marketing department is proposing the talk, you've already shown that as a speaker you're distant from the process. This is a small community-driven event, and speakers need to be directly engaged with the organizers and attendees.
  - __No vendor pitches__: As much as we value vendors and sponsors, we are not going to accept a talk that appears to be a pitch for your product.

  ### Submit

  To submit a talk visit our Papercall site.

repository: devopskorea/devopsdays-web

repo_name: devopskorea/devopsdays-web


text:
[![Published on webcomponents.org](https://img.shields.io/badge/webcomponents.org-published-blue.svg)](https://www.webcomponents.org/element/LostInBrittany/granite-prismjs)

## granite-prismjs

*granite-prismjs* is a wrapping of [Prism.js](http://prismjs.com/) CSS as [Polymer](https://www.polymer-project.org/) [shared styles modules](https://www.polymer-project.org/1.0/docs/devguide/styling.html#style-modules) (i.e. inside `<dom-module>` tags).

> Hybrid Polymer element, 1.x-2.x ready

## Doc & demo

[https://lostinbrittany.github.io/granite-prismjs](https://lostinbrittany.github.io/granite-prismjs)

### Using *granite-prismjs* modules

Using  polymer [shared styles](https://www.polymer-project.org/1.0/docs/devguide/styling.html#style-modules) modules is a two-step process: you need to use a `<link>` tag to import the module, and a `<style>` tag to include the styles in the correct place.

To use *granite-prismjs* in an element:

#### 1. Add the dependency

Add the dependency to the `bower.json` of your application:

```
   "dependencies": {
     [...]
     "granite-prismjs": "LostInBrittany/granite-prismjs#^1.8.3"
   }
```

And then recover them via `bower install`.

#### 2. Import the *granite-prismjs* module you want to use

Usually you will simply want to import `granite-prismjs.html` (wrap around `prismjs.css`) or any other of the prism themes, each one in  it's own HTML file as a style module..

Supossing you're using the standard folder locations for your components:

```
<link rel="import" href="../granite-prismjs/granite-prismjs.html">
```

#### 3. Inside your component, use *granite-prismjs* as shared style

In your element's template you add the include for the *granite-prismjs* module:

```
<style include="granite-prismjs"></style>
```

#### A complete example

```
<!-- import the module  -->
<link rel="import" href="../granite-prismjs/granite-prismjs.html">
<dom-module id="x-foo">
  <template>
    <!-- include the style module by name -->
    <style include="granite-prismjs"></style>
    <style>:host { display: block; }</style>
    Hi
  </template>
  <script>Polymer({is: 'x-foo'});</script>
</dom-module>
```

### Generating the style modules

To generate the style modules we use the [granite-css-modularizer](https://github.com/LostInBrittany/granite-css-modularizer) node script:

#### 1. Clone the repository and recover the dependencies of `granite-css-modularizer`

Clone the [granite-css-modularizer](https://github.com/LostInBrittany/granite-css-modularizer) repository and recover the dependencies using `yarn` (or `npm`) :

```
$ yarn install
yarn install v1.2.1
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
Done in 0.83s.
```

#### 2. Recover Prism.js

Recover Prism.js themesribution using `yarn` (or `npm`):

```
$ yarn add<EMAIL_ADDRESS>yarn add v1.2.1
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
success Saved 1 new dependency.
└─<EMAIL_ADDRESS>Done in 0.55s.
```

Currently *granite-prismjs* uses Prism.js version 1.8.3, if you need another version you can change simply install it.

#### 3. Generate the components

Using NodeJS and the `granite-css-modularizer.js` to transform Prism.js CSS files into polymer elements.

```
$ node ../granite-css-modularizer.js ./node_modules/prismjs/themes/css ./css_modules/granite-prismjs
```

After executing it, a series of HTML files is generated in the `./css_modules/granite-prismjs` folder, each one corresponding to a Prism.js CSS file.

```
$ ls ./css_modules/granite-prismjs/*.html
granite-prism-coy.html   granite-prism-funky.html  granite-prism-okaidia.html         granite-prism-tomorrow.html
granite-prism-dark.html  granite-prism.html        granite-prism-solarizedlight.html  granite-prism-twilight.html```

## Contributing

1. Fork it!
2. Create your feature branch: `git checkout -b my-new-feature`
3.  Push to the branch: `git push origin my-new-feature`
5. Submit a pull request :D

## Note on semver versioning

I'm aligning the versions of this element with Prism.js version, in order to make easier to choose the right version

## License

[Apache 2.0](http://www.apache.org/licenses/LICENSE-2.0)

repository: LostInBrittany/granite-prismjs

gha_language: HTML

repo_name: LostInBrittany/granite-prismjs


text:
<img src="images/logo.png" width="100" align="right" vspace="20" />

# Babelfish Web

This chart deploys a [web client](https://github.com/bblfsh/web) for [Babelfish](https://doc.bblf.sh/), and optionally a Babelfish server, if you don't have one running in your environment.

## Official Documentation

Official project documentation found [here](https://doc.bblf.sh/)

## Prerequisites

- Kubernetes 1.4+ with Beta APIs enabled

## Installing the Chart

To install the chart with the release name `my-release`:

```bash
$ helm install --name my-release bblfsh-web
```

## Configuration

See `values.yaml` for the available configuration parameters. Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`.

For example, to use an existing bblfsh server we set the server address:

```sh
$ helm install --name my-release --set settings.serverAddr=<IP_ADDRESS>:9432 bblfsh-web
```

repository: meyskens/charts-srcd

gha_language: Smarty

repo_name: meyskens/charts-srcd


text:
---
title: Virtual Environments in Python
author: Ratul Maharaj
date: 2020-07-16
hero: ./images/hero.jpg
excerpt: Managing project dependencies with venv
---

# Why use virtual environments?

It can be quite tempting to just install all your dependencies globally, but you'll soon find yourself running into issues.

By default, every Python project on your system uses the same directories to store and retrieve third party libraries (site-packages) and given that Python isn't able to differentiate between various versions of packages in the `site-packages` directory, a real problem arises.

A virtual environment is a tool that helps to keep dependencies required by different projects separate by creating isolated Python environments for them.

# Creating virtual environments

The easiest way to create a virtual environment in Python (version 3.3 and later) is to use the `venv` package that is built into the standard library.

A virtual environment can be created in your current working directory with the following command:

```python
python -m venv venv
```

The `python -m` command tells Python to run a library module as a script. The first `venv` specifies which module should be run i.e. the `venv` module. The second `venv` is the name of the virtual environment. I like naming my virtual environments `venv`, but you can really name it anything you like.

You should now be able to see a new folder named `venv` in your current working directory.  The structure of this folder should look similar to the following:

```
main.py
venv
├── Include
├── Lib
│   └── site-packages
├── Scripts
│   ├── activate
│   ├── activate.bat
│   ├── activate.ps1
│   ├── deactivate.bat
│   ├── pip.exe
│   └── python.exe
└── pyvenv.cfg
```

This is the virtual environment. As you'll be able to see, a `python` interpreter is placed in the scripts folder as well as a version of `pip`.

So now when you `pip install` packages into your virtual environment, they will get placed in the `site-packages` folder above. The `scripts` folder also contains scripts that can be used to activate the virtual environment. Again, `venv` is just a Python package and the version of Python that exists inside the virtual environment will depend on the version of Python used to create the environment.

Note that any Python scripts you create e.g. `main.py` should sit  outside of the `venv` folder.

# Activating the virtual environment

In order to `pip install` packages into the virtual environment you need to first activate it. This step is quite important as it allows you to specify that all project dependencies must come from the virtual environment and not the global environment. As seen in the `Scripts` folder above, creating a new virtual environment will create various scripts that can be used to activate and deactivate the virtual environment.

From the root of your project folder, you can activate the `venv` with one of the following:

```python
# bash
. venv/scripts/activate

# cmd
venv\scripts\activate.bat

# powershell
venv\scripts\activate.ps1
```

Once the virtual environment has been activated you can them `pip install` a package e.g. `pip install pip-install-test` and it will be installed to the `site-packages` folder in the virtual environment. You can then proceed to the `import` packages as you usually would in your Python scripts.

# Requirements.txt

To quickly and easily replicate environments elsewhere or even on other machines, you can create a `requirements.txt` file using the commands:

```python
pip freeze > requirements.txt
```

This will create a file which specifies all the dependencies of a project along with the specific version of each required package. The `requirements.txt` file sits outside of the `venv` folder and looks something like this:

```python
pip-install-test==0.5
```

You can copy this file elsewhere, create another virtual environment and install all the listed packages into your new virtual environment using:

```python
pip install -r requirements.txt
```

Note that the `requirements.txt` file does not specify the version of Python that is required.

# Deactivating the virtual environment

The easiest way to get out of a virtual environment is to use the `deactivate` command in the terminal.

# Other alternatives for creating virtual environments

There are many ways to create virtual environments in Python. In this post, I demonstrated how to use `venv` simply because it is built into the standard library and doesn't require any additional packages. However there are many other feature rich packages out there worth exploring.

The notable ones are [virtualenv](https://pypi.org/project/virtualenv/) and [pipenv](https://pypi.org/project/pipenv/).

repository: RatulMaharaj/Blog

repo_name: RatulMaharaj/Blog


text:
# ... Your Name

## Basic Info
**Nickname:** ...
**Age:** ...
**Ethnicity:** ...
**Nationality:** ...
**Email Adress:** ...
**GitHub ID:** ...
**Portfolio:** ...

## Education Info
**Highest Degree Earned:** ...
**University/College:** ...
**Year of Completion:** ...
**Percentage/Grade:** ...

## Past Training/Certification
**Name of the Trainning:** ...
**Institute/Organization:** ...
**Date/Year of Certification:** ...
**Remarks:** ...

## Past Work/Internship
**Company/Organization:** ...
**Position:** ...
**Duration:** ...
**Location:** ...
**Description:** ...

## Course Specific Info
- I get to know about the course from:
  - Author
  - Friends/Collegues
  - Developer Group _(mention name)_
  - Internet

- Domain of the course is:
  - Very much familiar
  - Mostly Familiar
  - Somewhat Familiar
  - Somewhat Unfamiliar
  - Little bit Familiar
  - Completely Unknown

- Have you done any such course in past, if yes mention your experience:
  - yes
  - no

- Why do you want to do the course?
  - ...


text:
# Basilicata Enviromental procedures

Viabas is a simple tool performing data mining on environmental procedures before Basilicata Region.

Information are stored in json format, allowing further processing of the information retrieved. E.g.

``` json
{
  "VIARegionaleConclusi": [
    {
      "id": "123456",
      "link": "[...]",
      "text": "Progetto di una centrale idroelettrica [...]",
      "isNew": true,
      "subtitle": "[...]",
      "description": "[...]",
      "attachments": [
        {
          "text": "[...]",
          "link": "[...]"
        }
      ]
    },
    [...]
  ]
}
```

## Installation

1) Download and install Node.js;
2) Download or clone this repository and execute `npm run build` from the package directory;
3) (Optional) execute `npm install -g` to install viabas globally.

## Usage

### Warning: This is a proof-of-concept. You are kindly advised to use the tool responsibly and to avoid generating unnecessary traffic on the Region EIA portal.

``` bash
$ viabas

Usage: viabas [options] [command]

VIA/PAUR Basilicata

Options:
  -V, --version      output the version number
  -h, --help         display help for command

Commands:
  update [options]
  process [options]
  help [command]     display help for command
```

``` bash
$ viabas help update

Usage: viabas update [options]

Options:
  -o, --outputdir [outputdir]  Output dir (default: "output")
  -f, --filename [filename]    Output filename (default: "output.json")
  -h, --help                   display help for command

```

### Data processing

The tool can provide the retrieved data as input to any templates file in a specified folder, to generate CSV, HTML, MD or other text formats.
Simply execute `viabas process -f [path of the generated json file] -o [output folder] -t [path of the templates folder]`.

Each template must be in a format compatible with [handlebarsjs](https://handlebarsjs.com/).
The example template included in the package's "templates" folder would convert the retrieved data in a csv file (e.g. for further processing in an excel spreadsheet):

``` csv
"Procedure","ID","Link","Title","Subtitle","Description","Started","Ended","Status"
{{#each data.items}}
"{{procedure}}","{{id}}","{{{link}}}","{{text}}","{{subtitle}}","{{description}}","{{started}}","{{ended}}","{{status}}"
{{/each}}
```

More complex data processing is left to the users.

## License

Copyright (c) 2021, Gianpaolo Terranova. (MIT License)

repository: gterranova/viabas

repo_name: gterranova/viabas


text:
#US2MapperKit ChangeLog

###Version 0.2.0

#####Important Upgrade Notes

Please note the following API/project changes have been made:

* New method introduced to allow the developer to update instance values with a new a dictionary. A new method is generated as part of the internal model object `func updateWithDictionary(dictionary: Dictionary<String, AnyObject>)`. You can now update values on an existing instance.
	* Fixes [Issue #21](https://github.com/ustwo/US2MapperKit/issues/21)
* The `US2TransformerProtocol` has been updated with support for returning objects of `Any` type, which includes support for _Struct_, _Enums_, _Closures_, and _Tuples_.
	* Fixes [Issue #23](https://github.com/ustwo/US2MapperKit/issues/23)

#####Important Upgrade Notes

Be sure to update your custom transformers accordingly for continued support of custom transformers.

```
Version 0.1.0 US2TransformerProtocol Definition

public protocol US2TransformerProtocol {
    func transformValues(inputValues : Dictionary<String, AnyObject>?) -> AnyObject?
}

Version 0.2.0 US2TransformerProtocol Definition

public protocol US2TransformerProtocol {
    func transformValues(inputValues : Dictionary<String, Any>?) -> Any?
}

```

###Version 0.1.0

Initial Release

gha_language: Swift

gha_license_id: NOASSERTION


text:
# TMRL

[![PyPI version](https://badge.fury.io/py/tmrl.svg)](https://badge.fury.io/py/tmrl)
[![PyPI - License](https://img.shields.io/pypi/l/tmrl?color=blue)](https://github.com/trackmania-rl/tmrl/blob/master/LICENSE)
[![DOI](https://zenodo.org/badge/277973609.svg)](https://zenodo.org/badge/latestdoi/277973609)

| **`API reference`**                                                                                |
|---------------------------------------------------------------------------------------------------------------------------------------------|
| [![Documentation Status](https://readthedocs.org/projects/tmrl/badge/?version=latest)](https://tmrl.readthedocs.io/en/latest/?badge=latest) |

`tmrl` is a fully-fledged distributed RL framework for robotics, designed to help you train Deep Reinforcement Learning AIs in real-time applications.

`tmrl` comes with a readily implemented pipeline for the TrackMania 2020 video game.

![example](https://github.com/trackmania-rl/tmrl/releases/download/v0.2.0/video_lidar.gif)

 **TL;DR:**

- :red_car: **AI and TM enthusiasts:**\
`tmrl` enables you to train AIs in TrackMania with minimal effort. Tutorial for you guys [here](readme/get_started.md), video of a pre-trained AI [here](https://www.youtube.com/watch?v=hQkltOX0TYw), and beginner introduction to the SAC algorithm [here](https://www.youtube.com/watch?v=LN29DDlHp1U).
- :rocket: **ML developers / roboticists:**\
`tmrl` is a python library designed to facilitate the implementation of deep RL applications in real-time settings such as robots and video games. Full tutorial [here](readme/tuto_library.md) and documentation [here](https://tmrl.readthedocs.io/en/latest/).

- :ok_hand: **ML developers who are TM enthusiasts with no interest in learning this huge thing:**\
`tmrl` provides a Gymnasium environment for TrackMania that is easy to use. Fast-track for you guys [here](#trackmania-gymnasium-environment).

- :earth_americas: **Everyone:**\
`tmrl` hosts the [TrackMania Roborace League](readme/competition.md), a vision-based AI competition where participants design real-time self-racing AIs in the TrackMania video game.

## Quick links
- [The TMRL Project](#the-tmrl-project)
  - [Introduction](#introduction)
    - [User features](#user-features-trackmania)
    - [Developer features](#developer-features-real-time-applications-in-python)
    - [TMRL in the media](#tmrl-in-the-media)
  - [Installation](readme/Install.md)
  - [Getting started](readme/get_started.md)
  - [TMRL python library for robot RL](readme/tuto_library.md)
    - [API reference](https://tmrl.readthedocs.io/en/latest/)
  - [Security (important)](#security)
- [TrackMania applications](#autonomous-driving-in-trackmania)
  - [TrackMania Roborace League](readme/competition.md)
  - [TrackMania Gymnasium environment](#trackmania-gymnasium-environment)
    - [LIDAR environment](#lidar-environment)
    - [Full environment](#full-environment)
  - [TrackMania training details](#trackmania-training-details)
    - [RL basics](#reinforcement-learning-basics)
    - [SAC](#soft-actor-critic)
    - [REDQ](#randomized-ensembled-double-q-learning)
    - [A clever reward](#a-clever-reward)
    - [Available action spaces](#available-action-spaces)
    - [Available observation spaces](#available-observation-spaces)
    - [Results](#results)
- [Framework details](#framework-details)
    - [Real-time Gym framework](#real-time-gym-framework)
      - [rtgym repo](https://github.com/yannbouteiller/rtgym)
  - [Remote training architecture](#remote-training-architecture)
- [Contribute](#authors)
- [Sponsors](#sponsors)

# The TMRL project

## Introduction

`tmrl` is a python framework designed to help you train Artificial Intelligences (AIs) through deep Reinforcement Learning (RL), for your own robots or real-time video games.

_Note: In the context of RL, an AI is called a policy._

### User features (TrackMania):
* **Training algorithms:**
`tmrl` lets you easily train policies in TrackMania with state-of-the-art Deep Reinforcement Learning algorithms such as [Soft Actor-Critic](https://www.youtube.com/watch?v=LN29DDlHp1U) (SAC) and [Randomized Ensembled Double Q-Learning](https://arxiv.org/abs/2101.05982) (REDQ).
These algorithms store collected samples in a large dataset, called a replay memory.
In parallel, this dataset is used to train an artificial neural network (policy) that maps observations (images, speed...) to relevant actions (gas, break, steering angle...).

* **Analog control:**
`tmrl` controls the game using a virtual gamepad, which enables analog input.

* **Different types of observation:**
The AI can either use raw unprocessed snapshots, or a LIDAR (Light Detection and Ranging) computed from the snapshots in order to perceive its environment.

* **Models:**
To process LIDAR measurements, `tmrl` uses a Multi-Layer Perceptron (MLP).
To process raw camera images (snapshots), it uses a Convolutional Neural Network (CNN).
These models learn the physics from histories or observations equally spaced in time.

### Developer features (real-time applications in Python):
* **Python library:**
`tmrl` is a complete framework designed to help you successfully implement deep RL in your [real-time applications](#real-time-gym-framework) (e.g., robots...).
A complete tutorial toward doing this is provided [here](readme/tuto_library.md).

* **TrackMania Gymnasium environment:**
`tmrl` comes with a real-time Gymnasium environment for the TrackMania2020 video game, based on [rtgym](https://pypi.org/project/rtgym/). Once `tmrl` is installed, it is easy to use this environment in your own training framework. More information [here](#trackmania-gymnasium-environment).

* **Distributed training:**
`tmrl` is based on a single-server / multiple-clients architecture.
It enables collecting samples locally on one or several computers and training remotely on a High Performance Computing cluster.
Find out more [here](#remote-training-architecture).

* **External libraries:**
This project gave birth to a few sub-projects of more general interest that were cut out and packaged as standalone python libraries.
In particular, [rtgym](https://github.com/yannbouteiller/rtgym) enables implementing Gymnasium environments in real-time applications,
[vgamepad](https://github.com/yannbouteiller/vgamepad) enables emulating virtual game controllers,
and [tlspyo](https://github.com/MISTLab/tls-python-object) enables transferring python object over the Internet in a secure fashion.

### TMRL in the media:
- In the french show [Underscore_ (2022-06-08)](https://www.youtube.com/watch?v=c1xq7iJ3f9E), we used a vision-based (LIDAR) policy to play against the TrackMania world champions. Spoiler: our policy lost by far (expectedly :smile:); the superhuman target was set to about 32s on the `tmrl-test` track, while the trained policy had a mean performance of about 45.5s. The Gymnasium environment that we used for the show is available [here](#lidar-with-track-progress).

## Installation

Detailed instructions for installation are provided at [this link](readme/Install.md).

## Getting started

Full guidance toward setting up an environment in TrackMania, testing pre-trained weights, as well as a beginner-friendly tutorial to train, test, and fine-tune your own models,
are provided at [this link](readme/get_started.md).

## TMRL python library

An advanced tutorial toward implementing your own ad-hoc optimized training pipelines for your own real-time tasks other than TrackMania (robots, other video games...) is provided [here](readme/tuto_library.md).

Security-wise, `tmrl` is based on [tlspyo](https://github.com/MISTLab/tls-python-object).

By default, `tmrl` transfers objects via non-encrypted TCP in order to work out-of-the-box.
This is fine as long as you use `tmrl` on your own private network.

To use `tmrl` on a public network (for instance, on the Internet), we recommend that you enable Transport Layer Security (TLS).
To do so, follow these instructions on all your machines:

- Open `config.json`;
- Set the `"TLS"` entry to `true`;
- Replace the `"PASSWORD"` entry with a strong password of your own (the same on all your machines);
- On the machine hosting your `Server`, generate a TLS key and certificate (follow the [tlspyo instructions](https://github.com/MISTLab/tls-python-object#tls-setup));
- Copy your generated certificate on all other machines (either in the default tlspyo credentials directory or in a directory of your choice);
- If you used your own directory in the previous step, replace the `"TLS_CREDENTIALS_DIRECTORY"` entry with its path.

If for any reason you do not wish to use TLS (not recommended), you should still at least use a custom password in `config.json` when training over a public network.
HOWEVER, DO NOT USE A PASSWORD THAT YOU USE FOR OTHER APPLICATIONS.
This is because, without TLS encryption, this password will be readable in the packets sent by your machines over the network.

# Autonomous driving in TrackMania

## TrackMania Roborace League

We host the [TrackMania Roborace League](readme/competition.md), a fun way of benchmarking self-racing approaches in the TrackMania2020 video game.
Follow the link for information about the competition, including the current leaderboard and instructions to participate.

Regardless of whether they want to compete or not, ML developers will find the [competition tutorial script](https://github.com/trackmania-rl/tmrl/blob/master/tmrl/tuto/competition/custom_actor_module.py) useful for creating advanced training pipelines in TrackMania.

## TrackMania Gymnasium environment
In case you only wish to use the `tmrl` Real-Time Gym environment for TrackMania in your own training framework, this is made possible by the `get_environment()` method:

_(NB: the game needs to be set up as described in the [getting started](readme/get_started.md) instructions)_
```python
from tmrl import get_environment
from time import sleep
import numpy as np

# default LIDAR observations are of shape: ((1,), (4, 19), (3,), (3,))
# representing: (speed, 4 last LIDARs, 2 previous actions)
# actions are [gas, break, steer], analog between -1.0 and +1.0
def model(obs):
    """
    simplistic policy for LIDAR observations
    """
    deviation = obs[1].mean(0)
    deviation /= (deviation.sum() + 0.001)
    steer = 0
    for i in range(19):
        steer += (i - 9) * deviation[i]
    steer = - np.tanh(steer * 4)
    steer = min(max(steer, -1.0), 1.0)
    return np.array([1.0, 0.0, steer])

# Let us retrieve the TMRL Gymnasium environment.
# The environment you get from get_environment() depends on the content of config.json
env = get_environment()

sleep(1.0)  # just so we have time to focus the TM20 window after starting the script

obs, info = env.reset()  # reset environment
for _ in range(200):  # rtgym ensures this runs at 20Hz by default
    act = model(obs)  # compute action
    obs, rew, terminated, truncated, info = env.step(act)  # step (rtgym ensures healthy time-steps)
    if terminated or truncated:
        break
env.wait()  # rtgym-specific method to artificially 'pause' the environment when needed
```

The environment flavor can be chosen and customized by changing the content of the `ENV` entry in `TmrlData\config\config.json`:

_(NB: do not copy-paste the examples, comments are not supported in vanilla .json files)_

### Full environment:
This version of the environment features full screenshots to be processed with, e.g., a CNN.
In addition, this version features the speed, gear and RPM.
This works on any track, using any (sensible) camera configuration.

```json5
{
  "ENV": {
    "RTGYM_INTERFACE": "TM20FULL",  // TrackMania 2020 with full screenshots
    "WINDOW_WIDTH": 256,  // width of the game window (min: 256)
    "WINDOW_HEIGHT": 128,  // height of the game window (min: 128)
    "SLEEP_TIME_AT_RESET": 1.5,  // the environment sleeps for this amount of time after each reset
    "IMG_HIST_LEN": 4,  // length of the history of images in observations (set to 1 for RNNs)
    "IMG_WIDTH": 64,  // actual (resized) width of the images in observations
    "IMG_HEIGHT": 64,  // actual (resized) height of the images in observations
    "IMG_GRAYSCALE": true,  // true for grayscale images, false for color images
    "RTGYM_CONFIG": {
      "time_step_duration": 0.05,  // duration of a time step
      "start_obs_capture": 0.04,  // duration before an observation is captured
      "time_step_timeout_factor": 1.0,  // maximum elasticity of a time step
      "act_buf_len": 2,  // length of the history of actions in observations (set to 1 for RNNs)
      "benchmark": false,  // enables benchmarking your environment when true
      "wait_on_done": true
    }
  }
}
```
Note that human players can see or hear the features provided by this environment: we provide no "cheat" that would render the approach non-transferable to the real world.
In case you do wish to cheat, though, you can easily take inspiration from our [rtgym interfaces](https://github.com/trackmania-rl/tmrl/blob/master/tmrl/custom/custom_gym_interfaces.py) to build your own custom environment for TrackMania.

The `Full` environment is used in the official [TMRL competition](https://github.com/trackmania-rl/tmrl/blob/master/readme/competition.md), and custom environments are featured in the "off" competition :wink:

### LIDAR environment:
In this version of the environment, screenshots are reduced to 19-beam LIDARs to be processed with, e.g., an MLP.
In addition, this version features the speed (that human players can see).
This works only on plain road with black borders, using the front camera with car hidden.
```json5
{
  "ENV": {
    "RTGYM_INTERFACE": "TM20LIDAR",  // TrackMania 2020 with LIDAR observations
    "WINDOW_WIDTH": 958,  // width of the game window (min: 256)
    "WINDOW_HEIGHT": 488,  // height of the game window (min: 128)
    "SLEEP_TIME_AT_RESET": 1.5,  // the environment sleeps for this amount of time after each reset
    "IMG_HIST_LEN": 4,  // length of the history of LIDAR measurements in observations (set to 1 for RNNs)
    "RTGYM_CONFIG": {
      "time_step_duration": 0.05,  // duration of a time step
      "start_obs_capture": 0.04,  // duration before an observation is captured
      "time_step_timeout_factor": 1.0,  // maximum elasticity of a time step
      "act_buf_len": 2,  // length of the history of actions in observations (set to 1 for RNNs)
      "benchmark": false,  // enables benchmarking your environment when true
      "wait_on_done": true
    }
  }
}
```

### LIDAR with track progress

If you have watched the [2022-06-08 episode](https://www.youtube.com/watch?v=c1xq7iJ3f9E) of the Underscore_ talk show (french), note that the policy you have seen has been trained in a slightly augmented version of the LIDAR environment: on top of LIDAR and speed value, we have added a value representing the percentage of completion of the track, so that the model can know the turns in advance similarly to humans practicing a given track.
This environment will not be accepted in the competition, as it is de-facto less generalizable.
However, if you wish to use this environment, e.g., to beat our results, you can use the following `config.json`:

```json5
{
  "ENV": {
    "RTGYM_INTERFACE": "TM20LIDARPROGRESS",  // TrackMania 2020 with LIDAR and percentage of completion
    "WINDOW_WIDTH": 958,  // width of the game window (min: 256)
    "WINDOW_HEIGHT": 488,  // height of the game window (min: 128)
    "SLEEP_TIME_AT_RESET": 1.5,  // the environment sleeps for this amount of time after each reset
    "IMG_HIST_LEN": 4,  // length of the history of LIDAR measurements in observations (set to 1 for RNNs)
    "RTGYM_CONFIG": {
      "time_step_duration": 0.05,  // duration of a time step
      "start_obs_capture": 0.04,  // duration before an observation is captured
      "time_step_timeout_factor": 1.0,  // maximum elasticity of a time step
      "act_buf_len": 2,  // length of the history of actions in observations (set to 1 for RNNs)
      "benchmark": false,  // enables benchmarking your environment when true
      "wait_on_done": true
    }
  }
}
```

## TrackMania training details

In `tmrl`, model (AI) that knows absolutely nothing about driving or even about what a road is, is set at the starting point of a track.
Its goal is to learn how to complete the track as fast as possible by exploring its own capacities and environment.

The car feeds observations such as images to an artificial neural network, which must output the best possible controls from these observations.
This implies that the AI must understand its environment in some way.
To achieve this understanding, the car explores the world for a few hours (up to a few days), slowly gaining an understanding of how to act efficiently.
This is accomplished through Deep Reinforcement Learning (RL).

### Reinforcement Learning basics

Most RL algorithms are based on a mathematical description of the environment called Markov Decision Process (MDP).
A policy trained though RL interacts with an MDP as follows:

![reward](readme/img/mrp.png)

In this illustration, the policy is represented as the stickman, and time is represented as time-steps of fixed duration.
At each time-step, the policy applies an action (float values for gas, brake, and steering) computed from an observation.
The action is applied to the environment, which yields a new observation at the end of the transition.

For the purpose of training this policy, the environment also provides another signal, called the "reward".
Indeed, RL is derived from behaviorism, which relies on the fundamental idea that intelligence is the result of a history of positive and negative stimuli.
The reward received by the AI at each time-step is a measure of how well it performs.

In order to learn how to drive, the AI tries random actions in response to incoming observations, gets rewarded positively or negatively, and optimizes its policy so that its long-term reward is maximized.

### Soft Actor-Critic

([Introductory video](https://www.youtube.com/watch?v=LN29DDlHp1U))

([Full paper](https://arxiv.org/abs/1801.01290))

Soft Actor-Critic (SAC) is an algorithm that enables learning continuous stochastic controllers.

More specifically, SAC does this using two separate Artificial Neural Networks (NNs):

- The first one, called the "policy network" (or, in the literature, the "actor"), is the NN the user is ultimately interested in : the controller of the car.
  It takes observations as input and outputs actions.
- The second called the "value network" (or, in the literature, the "critic"), is used to train the policy network.
  It takes an observation ```x``` and an action ```a``` as input, to output a value.
  This value is an estimate of the expected sum of future rewards if the AI observes ```x```, selects ```a```, and then uses the policy network forever (there is also a discount factor so that this sum is not infinite).

Both networks are trained in parallel using each other.
The reward signal is used to train the value network, and the value network is used to train the policy network.

Advantages of SAC over other existing methods are the following:
- It is able to store transitions in a huge circular buffer called the "replay memory" and reuse these transitions several times during training.
  This is an important property for applications such as `tmrl` where only a relatively small number of transitions can be collected due to the Real-Time nature of the setting.
- It is able to output analog controls. We use this property with a virtual gamepad.
- It maximizes the entropy of the learned policy.
  This means that the policy will be as random as possible while maximizing the reward.
  This property helps explore the environment and is known to produce policies that are robust to external perturbations, which is of central importance e.g. in real-world self-driving scenarios.

### Randomized Ensembled Double Q-Learning

([Full paper](https://arxiv.org/abs/2101.05982))

REDQ is a more recent methodology that improves the performance of value-based algorithms such as SAC.

The improvement introduced by REDQ consists essentially of training an ensemble of parallel value networks from which a subset is randomly sampled to evaluate target values during training.
The authors show that this enables low-bias updates and a sample efficiency comparable to model-based algorithms, at a much lower computational cost.

By default, `tmrl` trains policies with vanilla SAC.
To use REDQ-SAC, edit `TmrlData\config\config.json` on the machine used for training, and replace the `"SAC"` value with `"REDQSAC"` in the `"ALGORITHM"` entry.
You also need values for the `"REDQ_N"`, `"REDQ_M"` and `"REDQ_Q_UPDATES_PER_POLICY_UPDATE"` entries, where `"REDQ_N"` is the number of parallel critics, `"REDQ_M"` is the size of the subset, and `"REDQ_Q_UPDATES_PER_POLICY_UPDATE"` is a number of critic updates happenning between each actor update.

For instance, a valid `"ALG"` entry using REDQ-SAC is:

```json
  "ALG": {
    "ALGORITHM": "REDQSAC",
    "LEARN_ENTROPY_COEF":false,
    "LR_ACTOR":0.0003,
    "LR_CRITIC":0.00005,
    "LR_ENTROPY":0.0003,
    "GAMMA":0.995,
    "POLYAK":0.995,
    "TARGET_ENTROPY":-7.0,
    "ALPHA":0.37,
    "REDQ_N":10,
    "REDQ_M":2,
    "REDQ_Q_UPDATES_PER_POLICY_UPDATE":20
  },
```

### A clever reward

As mentioned before, a reward function is needed to evaluate how well the policy performs.

There are multiple reward functions that could be used.
For instance, one could directly use the raw speed of the car as a reward.
This makes some sense because the car slows down when it crashes and goes fast when it is performing well.
We use this as a reward in TrackMania Nations Forever.

However, such approach is naive.
Indeed, the actual goal of racing is not to move as fast as possible.
Rather, one wants to complete the largest portion of the track in the smallest possible amount of time.
This is not equivalent as one should consider the optimal trajectory, which may imply slowing down on sharp turns in order to take the apex of each curve.

In TrackMania 2020, we use a more advanced and conceptually more interesting reward function:

![reward](readme/img/Reward.PNG)

For a given track, we record one single demonstration trajectory.
This does not have to be a good demonstration, but only to follow the track.
Once the demonstration trajectory is recorded, it is automatically divided into equally spaced points.

During training, at each time-step, the reward is then the number of such points that the car has passed since the previous time-step.
In a nutshell, whereas the previous reward function was measuring how fast the car was, this new reward function measures how good it is at covering a big portion of the track in a given amount of time.

### Available action spaces

In `tmrl`, the car can be controlled in two different ways:

- The policy can output simple (binary) arrow presses.
- On Windows, the policy controls the car with analog inputs by emulating an XBox360 controller thanks to the [vgamepad](https://pypi.org/project/vgamepad/) library.

### Available observation spaces

Different observation spaces are available in `tmrl`:

- A history of raw screenshots (typically 4).
- A history of LIDAR measurement computed from raw screenshots in tracks with black borders.

In addition, we provide the norm of the velocity as part of the observation space in all our experiments.

Example of `tmrl` environment in TrackMania Nations Forever with a single LIDAR measurement:

![reward](readme/img/lidar.png)

In TrackMania Nations Forever, we use to compute the raw speed from screenshots thanks to the 1-NN algorithm.

In TrackMania 2020, we now use the [OpenPlanet](https://openplanet.nl) API to retrieve the raw speed directly.

### Results

We train policies in Real-Time with several observation spaces.
We show that our AIs are able to take advantage of the more complex types of observations in order to learn complex dynamics, leading to more clever policies:

In the following experiment, on top of the raw speed, the blue car is using a single LIDAR measurement whereas the red car is using a history of 4 LIDAR measurements.
The positions of both cars are captured at constant time intervals in this animation:

![Turn](readme/img/turn_tm20.gif)

The blue car learned to drive at a constant speed, as it is the best it can do from its naive observation space.
Conversely, the red car is able to infer higher-order dynamics from the history of 4 LIDARs and successfully learned to break, take the apex of the curve, and accelerate again after this sharp turn, which is slightly better in this situation.

# Framework details

## Real-time Gym framework:
This project uses [Real-Time Gym](https://github.com/yannbouteiller/rtgym) (```rtgym```), a simple python framework that enables efficient real-time implementations of Delayed Markov Decision Processes in real-world applications.

```rtgym``` constrains the times at which actions are sent and observations are retrieved as follows:

![Real-Time Gym Framework](https://raw.githubusercontent.com/yannbouteiller/rtgym/main/figures/rt_gym_env.png "Real-Time Gym Framework")

Time-steps are being elastically constrained to their nominal duration. When this elastic constraint cannot be satisfied, the previous time-step times out and the new time-step starts from the current timestamp.

Custom `rtgym` interfaces for Trackmania used by `tmrl` are implemented in [custom_gym_interfaces.py](https://github.com/yannbouteiller/tmrl/blob/master/tmrl/custom/custom_gym_interfaces.py).

## Remote training architecture:

`tmrl` is built with [tlspyo](https://github.com/MISTLab/tls-python-object).
Its client-server architecture is similar to [Ray RLlib](https://docs.ray.io/en/latest/rllib.html).
`tmrl` is not meant to compete with Ray, but it is much simpler to adapt in order to implement ad-hoc pipelines, and works on both Windows and Linux.

`tmrl` collects training samples from several rollout workers (typically several computers and/or robots).
Each rollout worker stores its collected samples in a local buffer, and periodically sends this replay buffer to the central server.
Periodically, each rollout worker also receives new policy weights from the central server and updates its policy network.

The central server is located either on the localhost of one of the rollout worker computers, on another computer on the local network, or on another computer on the Internet.
It collects samples from all the connected rollout workers and stores these in a local buffer.
This buffer is periodically sent to the trainer interface.
Periodically, the central server receives updated policy weights from the trainer interface and broadcasts these to all connected rollout workers.

The trainer interface is typically located on a non-rollout worker computer of the local network, or on another computer on the Internet (like a GPU cluster).
Of course, it is also possible to locate everything on localhost when needed.
The trainer interface periodically receives the samples gathered by the central server and appends these to a replay memory.
Periodically, it sends the new policy weights to the central server.

These mechanics can be summarized as follows:

![Networking architecture](readme/img/network_interface.png "Networking Architecture")

# Development roadmap:
You are welcome to contribute to the `tmrl` project.
Please consider the following:
- Further profiling and code optimization.
- Find the cleanest way to support sequences in `Memory` for RNN training.

You can discuss contribution projects in the [discussions section](https://github.com/trackmania-rl/tmrl/discussions)

# Authors:

When contributing, please submit a PR with your name in the contributors list with a short caption.

## Maintainers:
- Yann Bouteiller
- Edouard Geze

## Contributors:
- Simon Ramstedt - initial code base
- AndrejGobeX - optimization of screen capture

# License

MIT, Bouteiller and Geze 2021-2022.

# Sponsors:

Many thanks to our sponsors for their support!

![mist](readme/img/mistlogo.png)
[MISTlab - Polytechnique Montreal](https://mistlab.ca)

repository: trackmania-rl/tmrl

repo_name: trackmania-rl/tmrl


text:
# Cistanthe monosperma (Greene) Hershk. SPECIES

#### Status
ACCEPTED

#### According to
The Catalogue of Life, 3rd January 2011

#### Published in
null

#### Original name
Calyptridium monospermum Greene

### Remarks
null

repository: mdoering/backbone

repo_name: mdoering/backbone


text:
# Using Minishift

The following sections describe different aspects of using Minishift and provide an
overview of different components and services.

- [Managing your Openshift instance](#managing-your-openshift-instance)
  - [Starting OpenShift](#starting-openshift)
  - [Stopping OpenShift](#stopping-openshift)
  - [Deleting OpenShift](#deleting-openshift)
- [Environment variables](#environment-variables)
- [Interacting with OpenShift](#interacting-with-openshift)
  - [OpenShift client binary \(oc\)](#openshift-client-binary-oc)
  - [Console](#console)
  - [Services](#services)
- [Mounted host folders](#mounted-host-folders)
- [Networking](#networking)
- [Persistent volumes](#persistent-volumes)
- [Private container registries](#private-container-registries)

## Managing your Openshift instance

This section contains information about basic virtual machine and OpenShift management operations.

### Starting OpenShift

The [minishift start](./docs/minishift_start.md) command can be used to start your OpenShift instance.
This command creates and configures a virtual machine that runs a single-node OpenShift instance.

### Stopping OpenShift

The [minishift stop](./docs/minishift_stop.md) command can be used to stop your OpenShift instance.
This command shuts down the Minishift virtual machine, but preserves all instance state and data.
Starting the instance again will restore it to it's previous state.

### Deleting OpenShift

The [minishift delete](./docs/minishift_delete.md) command can be used to delete the OpenShift instance.
This command shuts down and deletes the Minishift virtual machine. No data or state is preserved.

## Environment variables

Minishift allows you to specify command line flags you commonly use via environment variables.
To do so, apply the following rules to the flag you want to set via an environment variable.

* Apply the _MINISHIFT__ as a prefix to your environment variable, for example the _vm-driver_ flag
  of the [start](./docs/minishift_start.md) command becomes _MINISHIFT_vm-driver_.
* Uppercase the flag, _MINISHIFT_vm-driver_ becomes _MINISHIFT_VM-DRIVER_.
* Last but not least, replace _-_ with _\__, _MINISHIFT_VM-DRIVER_ becomes _MINISHIFT_VM_DRIVER_

Another common example might be the URL of the ISO to be used. Usually you specify it via
_iso-url_ of the [start](./docs/minishift_start.md) command. Applying the rules from above, you can
also specify this URL by setting the environment variable _MINISHIFT_ISO_URL_.

**Note:** There is also the _MINISHIFT_HOME_ environment variable. Per default Minishift places all
its runtime state into _~/.minishift_. Using _MINISHIFT_HOME_, you can choose a different directory
as Minishift's home directory. This is currently experimental and semantics might change in
future releases.

## Interacting with OpenShift

### OpenShift client binary (oc)

The `minishift start` command creates an OpenShift instance using the
[cluster up](https://github.com/openshift/origin/blob/master/docs/cluster_up_down.md) approach.
For this purpose it copies the _oc_ binary onto  your host. You find it under
_~/.minishift/cache/oc/\<OpenShift version\>/oc_. You can add this binary to your _PATH_ variable
in order to use _oc_, eg:

    $ export PATH=$PATH:~/.minishift/cache/oc/v1.3.1

In further versions we will provide a command which will assit in setting up the _PATH_. See
also Github issue [#142](https://github.com/minishift/minishift/issues/142).

### Console

To access the [OpenShift console](https://docs.openshift.org/latest/architecture/infrastructure_components/web_console.html),
run this command in a shell after starting Minishift to get the address:

```shell
minishift console
```

### Services

To access a service exposed via a node port, run this command in a shell after starting Minishift to get the address:

```shell
minishift service [-n NAMESPACE] [--url] NAME
```

## Mounted host folders

Some of drivers will mount a host folder within the VM so that you can easily share files between the VM and host.
These are not configurable at the moment and are different for each driver and OS that you use.

**Note:** Host folder sharing is not implemented in the KVM driver yet.

| Driver | OS | HostFolder | VM |
| --- | --- | --- | --- |
| Virtualbox | Linux | /home | /hosthome |
| Virtualbox | OSX | /Users | /Users |
| Virtualbox | Windows | C://Users | /c/Users |
| VMWare Fusion | OSX | /Users | /Users |
| Xhyve | OSX | /Users | /Users |

## Networking

The Minishift VM is exposed to the host system via a host-only IP address, that can be obtained
with the `minishift ip` command.

## Persistent volumes

Minishift supports [PersistentVolumes](https://docs.openshift.org/latest/dev_guide/persistent_volumes.html)
of type `hostPath`. These PersistentVolumes are mapped to a directory inside the Minishift VM.

The MiniShift VM boots into a tmpfs, so most directories will not be persisted across reboots (`minishift stop`).
However, MiniShift is configured to persist files stored under the following host directories:

* `/data`
* `/var/lib/minishift`
* `/var/lib/docker`

Here is an example PersistentVolume config to persist data in the '/data' directory:

```yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv0001
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 5Gi
  hostPath:
    path: /data/pv0001/
```

## Private container registries

To access a private container registry, follow the steps on [this page](http://kubernetes.io/docs/user-guide/images/).

We recommend you to use ImagePullSecrets, but if you would like to configure access on the
Minishift VM you can place the `.dockercfg` in the `/home/docker` directory or the `config.json`
in the `/home/docker/.docker` directory.

repository: lburgazzoli/minishift

repo_name: lburgazzoli/minishift


text:
---
title: Clean Data Is the Foundation of Effective Machine Learning
author: Vikram Bahl
type: post
date: 2020-09-26T02:34:53+00:00
url: /clean-data-is-the-foundation-of-effective-machine-learning/
ssb_old_counts:
  - 'a:5:{s:7:"twitter";i:0;s:9:"pinterest";i:0;s:7:"fbshare";i:0;s:6:"reddit";i:0;s:6:"tumblr";i:0;}'
ssb_cache_timestamp:
  - 450461
categories:
  - machine learning
  - data cleaning
  - AI
  - ML
  - Python
  - Data Science
tags:
  - machine learning
  - data cleaning
  - AI
  - ML
  - Python
  - Data Science
---
_Reproducing what I wrote for [The New Stack][1] where it first appeared._

Data science isn’t just for tech companies anymore; and it is slowly becoming a significant part of data teams, whether you are an enterprise looking for IT modernization or a startup trying to achieve the unicorn status. What used to be a bastion of data engineers, BI teams and data analysts talking SQL is increasingly being infiltrated by talks of linear regression and reinforcement learning. Machine learning and AI have solved some real-world business challenges for companies who are today reaping the rewards for making the right investment at the right time. Recommendation engines on Netflix, predictive analysis on Spotify, and multilingual speech-to-text systems by Google are among the most widely shared examples of this success.

With this in mind, CEOs and CFOs are reaching out to their CIOs and engineering heads with problem statements along the lines of

  1. Company X increased its revenue by 10% by doubling down on hidden market segments which were brought to light by Machine Learning. Can we apply AI to our product line and achieve similar results?
  2. We are paying $20,000 per month to this cloud provider for storing data. Can we do some kind of AI or machine learning on this idle data and get a win for the board and boost PR?

Data science teams get formed with largely this context in mind. They are given a business use case and a problem statement and are expected to deliver stellar results, increase revenues, and decrease costs for the company

But the hard reality is that data engineers and scientists often spend weeks, and in some cases months, trying to clean, organize, and make sense of the data that’s been handed over to them by the customer’s Business Intelligence team.

The reasons for underperforming data science teams are manifold. However, well-researched studies and thought pieces say that dirty data impacts machine learning results but also has important implications on the performance of dirty data as well as the performance of data science teams. (Read this for six factors why data science teams fail.)

In short, before applying any algorithm or technique to the dataset, it is paramount to have quality and consistency in the dataset.

Anyone who has attended a computer science or programming class is familiar with GIGO or Garbage In Garbage Out. The same principle applies to machine learning.

Data science may be “the sexiest job of the 21st Century” (based on a study conducted by Harvard), but let’s dig deeper and understand what data scientists actually do on a day-to-day basis. A recent study by Forbes concluded that data scientists spend 60% of their time just cleaning and organizing the data. Now that gives a whole new perspective to this promising career of Data Science.

## Typical Machine Learning Flow

Now that we have established that the majority of time on the job is consumed by data cleaning, let’s take a look at where it fits in the overall machine learning flow.

On a high level, ML happens in the following more or less sequential steps (let’s leave the feedback loops aside):

  1. Source Discovery
  2. Data Preparation & Segregation
  3. Feature extraction
  4. Modeling
  5. Model Training & Tuning
  6. Prediction
  7. Model Deployment

The focus of this article will primarily be data preparation and segregation.

## Characteristics of Bad Datasets

I have come to an understanding that the Anna Karenina principle is universal and applies to this case as well: “All good datasets are alike; each bad dataset is bad in its own way.”

The implication of this is that no matter the dataset or use case, there will always be a data cleaning step. No amount of automation or edge case analysis will take it away (maybe a ML model trained on highly unclean and dirty data will one day solve this problem).

Some of the commonly seen data which makes a dataset “unclean” are showcased below. The reasons for these errors are usually due to human error at some point in the data lifecycle, however, some can be machine-generated as well.

### Inconsistent Schema

Imagine you are in the data team of a football association that has requested all football clubs in the country to send detailed information of the players in the team. The guideline for clubs is to send personal details in a csv file (there was no time to build a web interface or an API). Now some of the clubs created the column headers in the file just as expected (name, age, place\_of\_birth etc). However, some of the clubs named it differently, missed a column or added some new columns (a column for firstname and a column for last_name). You as part of the football association will, therefore, be spending additional time on reconciling the data into a common schema.

### Extraneous Text

This is when a value for the column contains unnecessary information. An example:

  * date\_of\_birth column with “01/12/2018szuyl” instead of “01/12/2018”

This happens more often than you think and is a pain when it brings down the data pipeline or an entire day’s worth of data is not included in a query because _01/12/2018szuyl_ means nothing.

### Missing Data

This is fairly simple to debug as it is just values that are missing. Scientists clean this by adding default values or calculating a value based on certain assumptions. Implications of missing data can be large in downstream ML models. If unchecked, it could lead to overfitting and an undertrained ML model.

### Redundant Information

This is when the same data exists more than once in a dataset (think duplicated rows in a csv/xls dataset or duplicate key:value documents). Datasets with redundant data may lead to overfitting down the line. Duplicated rows are rare in a relational database these days since having primary keys being NOT NULL and UNIQUE is expounded in most computer science curriculums.

However, not all information is redundant, it depends on the context. If there is a IoT temperature sensor streaming two temperature records every five seconds then it is likely that there will be groups of data with the same temperature value. This may not represent duplicate data when calculating anomalies in temperature values in a time period.

### Contextual Errors

These are some of the toughest to correct because it often requires “business” knowledge or contextual know-how of the dataset itself. Some examples:

  * Consider a dataset of daytime temperatures in the most populated cities in the world. A temperature value 320 is impossible to be in Celsius or Fahrenheit
  * Age is -46.3 years
  * Earth’s axial tilt is 22 degrees. Is this even within the range of acceptable axial tilts?

Scientists often assume that the sanity check on data is “not my job”

Sorry to disappoint.

### Garbage Values

Imagine a record in the first_name column is_ â€™&#258._ This obviously requires cleaning up or UTF-8/Unicode conversion of some sort (unless it’s an Elon Musk thing). This is often the first pass which a data engineer needs to perform on the dataset. Unchecked, it could lead to some pretty hairy bugs in a downstream extract, transform and load (ETL) pipeline or feature engineering script.

## Data Cleaning in Action

The effort required to clean data scales with the breadth and depth of the dataset. I’d like to walk you through a scenario from a while back where I iterated through multiple rounds of data cleaning efforts while training a simple ML model.

The objective was to build an automatic speech recognition system that can recognize spoken digits from 0 to 9.

We chose Kaldi to build the solution. There were three steps involved in achieving the perfect model:

  1. Collect dataset and create training and test sets
  2. Train the model on training dataset
  3. Test the model accuracy based on the predictions from test dataset

Now that seems simple enough, doesn’t it?

We had identified over a thousand speakers who were required to send us the data to be trained upon. The requirements for each speaker were: 25 WAV files of audio from each speaker sampled at 16000Hz. Each audio file is to be named after the digits spoken in that audio, for example if you say “1 2 3”, the file should be named “_1\_2\_3.wav_”. This naming convention is enforced by Kaldi and will be considered as a valid input to the Automatic Speech Recognition (ASR) toolkit.

Each speaker optionally creates a **textfile** with the digits they have spoken:

Again, this may seem like a walk in the park. But it isn’t. Here are some of things which had to be fixed before the real ML work began:

  1. Wrong Filename
      1. _1_23.wav_ or _123.wav_ instead of_ 1\_2\_3.wav_

**Our solution:** We wrote a Python script to go over each file, validate the filename and make the change if required (string.split, string.replace was used)

  1. Wrong codec or container. This came under 2 cases
      1. The extension says .WAV but the encoding is MP3
      2. The extension says .MP3 but the encoding is WAV
      3. Our solution:This was a bit tough to solve, we used ffmpeg to validate that we are using the right codec here. If it was the codec, we used a shell script to reencode it.
  2. Wrong sample rate
      1. We expect each audio file to be recorded in 16k Hz (simple to do with Audacity). However, about 30-40% of the recordings we received were recorded in 44k Hz.

**Our solution:** We used ffmpeg and a script to downsample the audio to 16k Hz

### List of fixes for the text file

We made a mistake of having this as an optional file. We shouldn’t have made this a requirement. Instead, the right way to do this is to generate this file from the names of the audio files. We made this an option for the participants so they would be rigorous in the process of recording their audio. What you say is what you document.

However, at that point in time, we received the textfile from half the participants. For the second half which sent in their files, there were challenges with formatting which needed to be addressed, for example

  1. Missing delimiter or duplicated delimiter \r\n: we received files with_ 123789_ in one line instead of _123\r\n789._

What you expect:

What you get:

We abandoned the approach of fixing the textfiles sent in by the user as there were too many edge cases to handle. A decision was made to auto-generate the text file after fixing in the file naming issues with the audio files.

## Data Preparation

After cleaning the dataset, we had to do another additional step of data preparation to make the dataset ready for machine learning. This step was not as painful as cleaning the dataset but it was still another hurdle to jump before the machine learning could begin.

Why is data preparation required? Simply put, Kaldi expects the input to the ASR ML program in a certain format. With the cleaned dataset in hand, we had to create the following text files:

  * 1 _spk2gender_ file

  * 1 text file

  * 1 _utt2spk_ file

  * 1 _wav.scp_ file

These files can all be generated using the information provided by the audio files and the textfile. We spent a day writing the Python script to generate these files.

Once this was done, we spent all of two days running the ML algorithm customizations, model tuning, and deploying it to the cloud.

Here’s a shoutout to some of the tools used:

## Key Tenets to Keep in Mind

What are the final takeaways? This is the section that sums the post up with “tips and tricks” and “ninja tactics.” In short, you can put these points as TLDR or in a slide deck for management.

### Data Quality Plan

Start by creating KPIs for your dataset. Set guidelines for the quality of data in the system. This ensures that all data enters your system only after following a predefined set of rules. This process starts with profiling current data sources and points of entry and ends with defining the workflow of data in the system including any new data sources in the future.

### Data Standardization

This is a qualitative step that begins by asking the question “Is the right data entering my systems?” If unimportant data or incorrect data is entering your system without being sanitized, then it leads to additional effort down the road when time may be a critical factor in getting insights.

It is important to store raw data. There may be future use cases or legal/compliance needs which depend on it. However, it is prudent to have that stored in archival. For more day to day data needs, a layer of prepared data should be readily available in datastores.

### Data Validation

Verifying the accuracy of data is paramount to the success of the data team in delivering a business outcome. On a high level, this involves verifying the source of data and verifying the accuracy of a data point. If you are getting weekly dumps from a third-party provider, you need to assess whether that data is reliable, how they collect the data, what steps they take to clean it and how they manage access to it.

Having custom rules depending on the dataset to verify if the data points make sense is critical. If a value in the age column is negative, it’s more than likely that it is incorrect. If the justification is “No negative value in the age column represents the delta between the estimated year of death and today, then maybe that column should not be called age.”

### Data Hygiene

This involves end to end hygiene of the datasets to include fixing of:

  * Inconsistent schema
  * Extraneous text
  * Missing data
  * Redundant information
  * Contextual errors
  * Garbage values

Yes, but those scripts can be reused/automated to a high degree.

Although you have to manually write code to clean up every dataset since it is a unique schema, there is always room for automation.

### Enrich Data

Purists will argue that this is part of the ETL step or it “depends on xyz..” but it is valuable to do a first pass at enriching the data by appending or aggregating fields to create a cleaner schema. Getting a column in Fahrenheit and non-US members are tearing their hair out? Just add a column for Celsius. Append last name to the first name and have a single name column. The use case depends on how you plan to consume the data.

### Automation

All the points made above require thought, planning, and coding. It is important to automate these processes into systems as and when a pattern emerges. Think of it as a CI/CD pipeline for your data. All the automation gets integrated, gets tested and the ‘build’ is nothing but a dataset meeting its KPIs for quality and consistency.

## Closing Thoughts

At the end of the day, the tasks which bring maximum visible value to the business is the machine learning piece. That’s the reason teams go through the trouble of spending so much developer time with data cleaning and preparation.

However, there is a point to be made about the value of bringing rigor to the data science process by having a system or process around data cleaning and preparation. Once these systems are designed and automated, there is a path for the data to go from unclean to clean. Always remember that automating something once is far better than doing something manually way too many times. The process of getting there will be arduous and painful, but it will bear sweet fruits for everyone.

  Photo by Jeremy Perkins on Unsplash.

 [1]: https://thenewstack.io/clean-data-is-the-foundation-of-effective-machine-learning/

repository: svram/vikrambahl-com

repo_name: svram/vikrambahl-com


text:
# MacroBase

## What is MacroBase?

MacroBase is a new *analytic monitoring* engine designed to prioritize human attention in large-scale datasets and data streams. Unlike a traditional analytics engine, MacroBase is specialized for one task: finding and explaining unusual or interesting trends in data.

Under the hood, MacroBase is architected for rapid adaptation to new application domains. MacroBase has been successfully used for analyses in domains including datacenter and mobile application monitoring, industrial manufacturing, and video and satellite image analysis.

MacroBase supports streaming operation, meaning that once users identify an important behavior, they can track it continuously over time.

## How can I run MacroBase queries?

The simplest way to use MacroBase is through its UI. Users highlight key performance metrics (e.g., power drain, latency) and metadata attributes (e.g., hostname, device ID), and MacroBase reports explanations of abnormal behavior. For example, MacroBase may report that queries running on host 5 are 10 times more likely to experience high latency than the rest of the cluster or that devices of type X103 are 30 times more likely to experience high power drain. Check out the [Tutorial](getting-started/tutorial).

MacroBase is designed for extensibility, and there are a number of more advanced interfaces you can use to program the system and execute more complex query functionality.

## More information

Feel free to open GitHub issues or email us at<EMAIL_ADDRESS>The development branches contain most of the action.
A detailed technical overview of the project is available [on the arXiv](http://arxiv.org/pdf/1603.00567.pdf).

repository: sandeep-iitr/macrobase

repo_name: sandeep-iitr/macrobase


text:
# DependencyBuilder
Builds dependencies in topological order.

## Usage:
```Go
builder := dg.New()

// Add dependency type and it's constructor.
dg.AddDep(reflect.TypeOf(dep0{}), dep0Constructor)

// Build dependencies
dg.Build()
```