text:
# Sorcerer Task

[Back to Documentation](README.md)

## Configuration

Tasks are defined in the configuration files and then Sorcerer deserializes the YAML fields into an internal task object. To mark this, task definitions should be specified with a `!task` tag and delimited with `---`. The task fields are:

key  |required?|description
-----|---------|:-----------
name |yes      |task name unique in the task namespace
next |yes      |list of task names that are next in workflow
exec |no       |see [Provided task types](#)

### Provided task types

There are some provided task types that do not require you to implement. They are specified by populating the `exec` field in the task definition.

- #### fork
  This is a placeholder task that will inform Sorcerer to launch all of its `next` tasks in the next scheduled attempt of the pipeline.

- #### join
  Since Sorcerer by default will not execute a task until all tasks that specify the current one as `next` are completed, any specified task is technically a "join" task. Nonetheless a workflow definition may sometimes be more readable to have a explicit join task. Therefore we provide a placeholder task that will wait until all previous tasks complete successfully before it is scheduled.

##### Examples
```YAML
!task
	name: task_1
	next:
		- task_2
		- task_3
	exec: fork
---
!task
	name: task_2
	...
```

## Implementation

The Task implementation has two parts: Annotation and Methods

### Annotation

Sorcerer task implementation should be annotated with the `@SorcererTask` annotation with the `name` field populated. The name specified in the annotation will be used to map to the corresponding task definition in the configuration files.

```
@SorcererTask(name = "task_name")
```

### Methods

The `Task` class is an interface that requires the user to override and implement the following methods:

- #### `init(Context context)`
  Some tasks will require some initialization logic before execution. Since we use Guice injections to back instance creation, you could also implement some of your initialization code in a constructor.

  Similiarly, the `init()` method is provided to implement the task initialization code. It also provides a Context object which contains some parameters (like iteration number) as well as provides a way to pass parameters from the init method to the rest of the task without having to implement class variables.

  It is important to note that the `init()` method is called **before** before the call to `getDependencies()`. This can sometimes be useful becuase sometimes the dependency checking step can also require initialization. However the user should be aware of this because if the initialization step itself has dependencies, the user will have to implement those checks.

- #### `getDependencies(int iterNo)`

  This method returns a Collection of `Dependency` instances. These dependencies have to all resolve to true before Sorcerer proceeds with scheduling the task

- #### `exec(Context context)`

  This is the main method of the task. The actual business logic of the task should be put into method. The method also throws Exception so any exceptions thrown in the method will be caught, then wrapped in a SorcererException and logged.

##### Examples
```java
package com.example.tasks;

import com.turn.sorcerer.task.SorcererTask;

@SorcererTask(name = "new_task")
public class NewTask implements Task {

	private int scheduledHour;
	private int scheduledMinute;

    @Override
    public void init(Context context) {
        scheduledHour = 17;
        scheduledMinute = 30;
    }

    @Override
    public void exec(Context context) throws Exception {
        System.out.println("Running a new task!");
    }

    @Override
    public Collection<Dependency> getDependencies(int iterNo) {
    	return ImmutableList.<Dependency>builder()
				.add(new SorcererTaskDependency("prev_task_1"))
				.add(new TimeDependency(new DateTime()
						.withHourOfDay(scheduledHour)
						.withMinuteOfHour(scheduledMinute)))
				.build();
    }
}
```

## Dependency

The `Dependency` class is the object used by Sorcerer to check that a task's dependency is fulfilled. It only has one method to override:

- ##### `check(int iterNo)`

  This method returns true if the dependencies are fulfilled. The iteration number is provided to be used in the dependency checking.

### Provided Dependency implementations

There are some Dependency implementations already made available for some common task dependencies.

- #### SorcererTaskDependency

  This is a task dependency on another Sorcerer task. This can be used if a task is dependent on another task from the same or another pipeline.It provides a variety of ways to specify the task and/or iteration number. See the java docs for more details.

- #### TimeDependency

  This represents a task dependency based on time. This translates to a task being run only once a day. Sorcerer uses joda-time to represent time.

- #### HDFSPathDependency

  This is an abstract class that provides a `getPaths()` method where it will check for the existence of a the provided paths. See Javadocs for details.

## Context

In order to provide some context information for the task, a `Context` object is provided. The Context object contains fields useful for the task to both get and provide information outside the context of the task.

- ##### `getIterationNumber()`

  This will return an immutable `int` representing the current iteration number of the task.

- ##### `putProperty()`

  This puts a user parameter into a map that is passed on to the next tasks.

- ##### `getProperty()`

  This returns the property of the given key.

- ##### `putMetric()`

  This puts a metric in the metric map with the provided key and value.

---

[Back to Documentation](README.md)

repository: turn/sorcerer

gha_language: Java

repo_name: turn/sorcerer


text:
---
title: 'C++'
date: '2019-03-17'
tags: ['intro', 'cpp', '2019', 'march', '2019-March']
---

# Introduction #

This post is about C++ code.

### Hello World ###

The first thing is to observe the Hello World program in C++.

```cpp

  #include <iostream>

  int main () {
    std::cout << "Hello World" << std::endl;
    return 0;
  }

```

### Changes in C++11 ###

A vector of vector can now be defined as

```cpp
  std::vector<std::vector<int>> x;

  // Previously
  std::vector<std::vector<int> >x;
```

license: MIT

repository: rahulkadukar/site

branch_name: refs/heads/master

extension: md

license_type: permissive

repo_name: rahulkadukar/site


text:
# (->> expr forms**) threads value through calls as their last argument
Evaluates /expr/ and threads it through the supplied forms as their last argument. Any form that is not already a function call will be converted into one before threading.

## An Example

  (->> 0 (+ 10) (** 2) (// 5))

Will expand to `(// 5 (** 2 (+ 10 0)))` and return _0.25_. In order to better visualize what's going on, one might choose to insert a /,/ as a placeholder for the threaded value.

  (->> 0 (+ 10 ,) (** 2 ,) (// 5 ,))

repository: etsangsplk/sputter

repo_name: etsangsplk/sputter


text:
# lower_bound

**Description**: The function returns an iterator pointing to the key in the map container which is equivalent to k passed in the parameter.
In case k is not present in the map container, the function returns an iterator pointing to the immediate next element
which is just greater than k. If the key passed in the parameter exceeds the maximum key in the container,
then the iterator returned points to number of elements in the map as key and element = 0.

**Example**:
```cpp
#include <iostream>
#include <map>

int main() {

    // initialize container
    std::map<int, int> mp;

    // insert elements in random order
    mp.insert({ 2, 200 });
    mp.insert({ 1, 100 });
    mp.insert({ 5, 500 });
    mp.insert({ 4, 400 });

    // when 1 is present
    auto it = mp.lower_bound(1);
    std::cout << "The lower bound of key 1 is ";
    std::cout << (*it).first << " " << (*it).second << std::endl;  //Prints "The lower bound of key 1 is 1  100"

    // when 3 is not present
    // points to next greater after 3
    it = mp.lower_bound(3);
    std::cout << "The lower bound of key 3 is ";
    std::cout << (*it).first << " " << (*it).second;   //Prints "The lower bound of key 4 is 4  400"

    // when 6 exceeds
    it = mp.lower_bound(6);
    std::cout << "\nThe lower bound of key 6 is ";
    std::cout << (*it).first << " " << (*it).second;  //Prints "The lower bound of key 6 is 4  0"
    return 0;
}
```
**[Run Code](https://rextester.com/AWN42860)**

gha_license_id: MIT


text:
---
title: Integrating DataHub With Great Expectations
authors:
    name: Mayuri Nehate, John Joyce, Maggie Hays
    url: https://datahubproject.io
---

:::info
* Maintained By: DataHub
* Status: Beta
* Support/Contact: https://slack.datahubproject.io/
:::

### Introduction
This integration allows you to push the results of running Expectations into DataHub (https://datahubproject.io/). DataHub is a metadata platform which enables search & discovery, federated governance, and data observability for the Modern Data Stack.

### Technical background
There is a custom Action named `DataHubValidationAction` which allows you to view Expectation Results inside of DataHub.

:::note Prerequisites
 - Create a [Great Expectations Checkpoint](https://docs.greatexpectations.io/docs/terms/checkpoint)
 - [Deploy an instance of DataHub](https://datahubproject.io/docs/quickstart)
:::

`DataHubValidationAction` pushes Expectations metadata to DataHub. This includes

- **Expectation Details**: Details of assertions (i.e. Expectation) set on a Dataset (Table). Expectation set on a dataset in GE aligns with `AssertionInfo` aspect in DataHub. `AssertionInfo` captures the dataset and dataset fields on which assertion is applied, along with its scope, type and parameters.
- **Expectation Results**: Evaluation results for an assertion tracked over time.
Validation Result for an Expectation in GE align with `AssertionRunEvent` aspect in DataHub. `AssertionRunEvent` captures the time at which Validation was run, Batch(subset) of dataset on which it was run, the success status along with other result fields.

### Dev loops unlocked by integration
* View dataset and column level Expectations set on a dataset
* View time-series history of Expectation's outcome (pass/fail)
* View current health status of dataset

### Setup

Install the required dependency in your Great Expectations environment.
```shell
pip install 'acryl-datahub[great-expectations]'
```

## Usage

:::tip
Stand up and take a breath
:::

####  1. Ingest the metadata from source data platform into DataHub
For example, if you have GE Checkpoint that runs Expectations on a BigQuery dataset, then first
ingest the respective dataset into DataHub using [BigQuery](https://datahubproject.io/docs/generated/ingestion/sources/bigquery#module-bigquery) metadata ingestion source recipe.

```bash
datahub ingest -c recipe.yaml
```
You should be able to see the dataset in DataHub UI.

#### 2. Update GE Checkpoint Configurations
Add `DataHubValidationAction` in `action_list` of your Great Expectations Checkpoint. For more details on setting action_list, see [the configuration section of the GE Actions reference entry](https://docs.greatexpectations.io/docs/terms/action#configuration)
```yml
action_list:
  - name: datahub_action
    action:
      module_name: datahub.integrations.great_expectations.action
      class_name: DataHubValidationAction
      server_url: http://localhost:8080 #DataHub server url
```

**Configuration options:**
- `server_url` (required): URL of DataHub GMS endpoint
- `env` (optional, defaults to "PROD"): Environment to use in namespace when constructing dataset URNs.
- `platform_instance_map` (optional): Platform instance mapping to use when constructing dataset URNs. Maps the GE 'data source' name to a platform instance on DataHub. e.g. `platform_instance_map: { "datasource_name": "warehouse" }`
- `graceful_exceptions` (defaults to true): If set to true, most runtime errors in the lineage backend will be suppressed and will not cause the overall Checkpoint to fail. Note that configuration issues will still throw exceptions.
- `token` (optional): Bearer token used for authentication.
- `timeout_sec` (optional): Per-HTTP request timeout.
- `retry_status_codes` (optional): Retry HTTP request also on these status codes.
- `retry_max_times` (optional): Maximum times to retry if HTTP request fails. The delay between retries is increased exponentially.
- `extra_headers` (optional): Extra headers which will be added to the datahub request.
- `parse_table_names_from_sql` (defaults to false): The integration can use an SQL parser to try to parse the datasets being asserted. This parsing is disabled by default, but can be enabled by setting `parse_table_names_from_sql: True`.  The parser is based on the [`sqllineage`](https://pypi.org/project/sqllineage/) package.

#### 3. Run the GE checkpoint

```bash
great_expectations checkpoint run my_checkpoint #replace my_checkpoint with your checkpoint name
```

#### 4. Hurray!
The Validation Results would show up in Validation tab on Dataset page in DataHub UI.

## Further discussion

### Things to consider
Currently this integration only supports v3 API Datasources using `SqlAlchemyExecutionEngine`.

This integration does not support

- v2 Datasources such as `SqlAlchemyDataset`
- v3 Datasources using an Execution Engine other than `SqlAlchemyExecutionEngine` (Spark, Pandas)
- Cross-dataset Expectations (those involving > 1 table)

### When things don't work

- Follow [Debugging](https://datahubproject.io/docs/metadata-ingestion/integration_docs/great-expectations/#debugging) section to see what went wrong!
- Feel free to ping us on [DataHub Slack](https://slack.datahubproject.io/)!

### Other resources

 - [Demo](https://www.loom.com/share/d781c9f0b270477fb5d6b0c26ef7f22d) of Great Expectations Datahub Integration in action
 - DataHub [Metadata Ingestion Sources](https://datahubproject.io/docs/metadata-ingestion)

branch_name: refs/heads/develop


text:
# Events

You can add events callback in following ways

```JavaScript
const muneem = Muneem();

muneem.after("event", fn);//muneem.on("type", fn);
muneem.before("event", fn);

muneem.start();
```

Event name is case insensitive and multiple aliases ara available for your comfortability.

## Before and After

* pre, or preHandler : Before/After any pre handler

```JavaScript
muneem.after("pre", (asked, handlerName) => {
    //..
})
```

* post, or postHandler : Before/After any post handler

```JavaScript
muneem.after("post", (asked, handlerName) => {
    //..
})
```
* each, or eachHandler : Before/After pre & post handlers

```JavaScript
muneem.after("each", (asked, handlerName) => {
    //..
})
```
* main, or mainHandler : Before/After main handler
```JavaScript
muneem.after("main", (asked, handlerName) => {
    //..
})
```

## Before

 * addRoute : just before the route is added;

 ```JavaScript
muneem.before("addRoute", ( routeContext ) => {
    //..
})
```

 * serverStart, or start : just before server starts;
 ```JavaScript
 /*
serverOptions : {
    host : "local.host",
    port: 3002,
    http2 : false,
    https : false
}
*/
muneem.before("start", ( serverOptions ) => {
    //..
})
```

 * send, answer, or response : Before sending the response

```JavaScript
muneem.before("send", ( asked, answer ) => {
    //..
})
```

 * serverClose, or close : just before server's close is triggered

```JavaScript
muneem.before("serverClose", ( ) => {
    //..
})
```

## After

* serverStart, start : just after server starts;
  ```JavaScript
muneem.after("start", ( ) => {
    //..
})
```
* request : before route;
 ```JavaScript
muneem.after("request", ( nativeRequest, nativeResponse ) => {
    //..
})
```

* route : before all handlers;
```JavaScript
muneem.after("serverClose", (  asked, answer ) => {
    //..
})
```

* exceedContentLengthn, or fatBody: this event is triggered when the request stream is more than expected.
```JavaScript
muneem.after("fatBody", (asked ) => {
    //..
})
```

* send, answer, or response : After sending the response;
```JavaScript
muneem.after("send", ( asked ) => {
    //..
})
```

* serverclose, or close : just before server's close is triggered
```JavaScript
muneem.after("serverClose", ( ) => {
    //..
})
```

* routeNotFound : when no matching route is found
```JavaScript
muneem.after("routeNotFound", ( asked ) => {
    //..
})
```

* error : on error
```JavaScript
muneem.after("error", ( error, asked ) => {
    //..
})
```

repository: node-muneem/muneem

gha_language: JavaScript

repo_name: node-muneem/muneem


text:
# Contribution Guidelines

Please note that this project is released with a [Contributor Code of Conduct](https://github.com/spring-projects/spring-framework/blob/master/CODE_OF_CONDUCT.adoc). By participating in this project you agree to abide by its terms.

- 35 stars minimum are required to apply, it mean your project hold interest
- Use a meaningful name to your commit or I'll close it instantly, Update source.md is NOT a name
- Does your app bring something really interesting ?

- Is there a direct link so I we can check the details of your repo ?
- Flutter is all about UI, use screenshots and animated media for your widget
- Remember the team is giving its own time to help the community. This is not a paid job yet it take a lot of time to curate and review.

When your contribution is accepted you can add the Awesome Flutter Badge to your repository
thus signaling the quality of your work and linking to others authors Awesome projets

 ```
<a href="https://github.com/Solido/awesome-flutter">
    <img alt="Awesome Flutter" src="https://img.shields.io/badge/Awesome-Flutter-blue.svg?longCache=true&style=flat-square" />
</a>
```

Ensure your pull request adheres to the following guidelines:

- Search previous suggestions before making a new one, as yours may be a duplicate.
- Suggested packages should be tested and documented.
- The pull request should have a meaningful title and include a link to the package / resource and why it's awesome.
- Make an individual pull request for each suggestion.
- Use the following format: `[resource](link) - Description by [Author](link to author)`
- Use [title-casing](http://titlecapitalization.com) (AP style).
- Additions should be added to the bottom of the relevant category.
- New categories, or improvements to the existing categorization are welcome.
- Keep descriptions short and simple, but descriptive.
- Don't mention `Flutter` in the description as it's implied.
- Start the description with a capital.
- Check your spelling and grammar.
- Make sure your text editor is set to remove trailing whitespace.

Thank you for your suggestions!

repository: Solido/awesome-flutter

gha_language: Dart

repo_name: Solido/awesome-flutter


text:
# Software Architectures

This project is created for the final exam for Advanced Software Architectures.
It represents app for managing employees and departments inside one organization.

It consists of two applications, front-end and back-end. The front-end app is done in react and back-end are done in Spring Boot. Back-end consist of four micro services, one is Discovery service and the other three and made as clients. Discovery service runs on port 8761.

To start application, it's necessary to have previously installed mysql, java 15, and node js. The front-end app is hosted on port 3000.

branch_name: refs/heads/main


text:
# Changelog
All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.4.4] Unreleased Github Repo [develop]
## [0.4.3] - 2022-09-29
## Fix
- `wss_repl.py` dropped `websocket_help` dependency.
- `upyutils/develop/upynotify.py` pwm bug, `upytuils/ble/ble_uart_peripheral.py` rxbuf bug.
- wss WebSocketDevices now use ROOT CA-signed certificate
  (ROOT CA key/cert generated with `$ upydev kg ssl CA`)
- allow_abrev bug for `$ upydev kg ssl dev status -a` option (`--a` for python3.8+).
- fix fast get/dsync shasum animation for BleDevices.
- improve certificate info issuer/subject.
- fix dev_platform in `net scan` for esp8266
- now serial repl works with ``screen`` too (using ``repl -sc``) (in case ``picocom`` is not available)
- fix ssl error on `stream_test` in WebSocketDevices using wss.
- fix paste mode in serial repl missing to print first line output.
- fix net scan bug for esp8266
- fix dsync command help info.
- fix uconfig (config in shell-repl) help info.
- fix config `add_param` to allow custom `root_path`
- update otatool.py, ota.py to use ROOT CA certificate as CA.
## Added
- Added `ursyslogger.py` to `upyutils/develop` to enable remote logging with rsyslog as endpoint.
- Added `test_dev.py` to run parametric tests using yaml files.
- Added benchmark integration with ``pytest-benchmark``
- Added network tests with ``iperf3:uiperf3.py``
- Added multiple tests and benchmarks using yaml files.
- Added benchmarks from ``micropython/tests/perf_bench``
- `play` command to run custom tasks in ansible playbook style.
- Added example playbooks
- `uptime` and `cycles` utils/commands.
- upylog file rotation, so file log size does not go over a limit size. (2KB default)
- uptime.py and cycles.py to upyutils_dir (so it can be installed with `update_upyutils` cmd)
- `mv` command to change file names.
- `kg ssl CA` to generate a ROOT CA key/cert to be able to generate device's and host CA signed certificate.
- `kg ssl host` to generate a HOST key/cert signed by ROOT CA.
- `kg ssl dev add/export` to add or export a device certificate.
- `kg ssl CA add/export` to add or export a ROOT CA key/cert pair.
- `kg ssl dev/host/CA status` to check expiration date of certificates.
-  The new CA-host-dev certificate chain of trust model allows log in to wss WebSocketDevices from multiple locations/hosts (by exporting the ROOT CA key/cert pair to a new host and add them there to generate the proper key/cert for that host).
- nwatchdog timer kwargs.
- net scan results are sorted by RSSI value.
- local path in shell-repl prompt is now aware of current git branch.
## [0.4.2] - 2022-07-10
## Fix
- fix lcd ~ home user expand in shl & ssl key,cert gen for zerotier devices with custom port
- fix test_custom_cmd.py reload script on exception
- clarify extra free RAM in custom esp32 OTA SSL build
- fix file not found error in `put` command if file is indicated with absolute path
- fix `datetime` command missing
- `import os` missing with `put`, `get` commands if `-dir` indicated
## [0.4.1] - 2022-03-15
## Fix
- optimized autocompletion of subcommands and options
- upydev cli startup time
## [0.4.0] - 2022-03-13
## Fix
- hard reset `-hr` option for `reset` command
- added missing commands `touch`, `cd`.
- fix `-h` help for filesystem commands
- autocomplete options
## Added
- autocomplete commands with multiple options as subcommands
## [0.3.9] - 2022-03-12
## Fix
-  parse additional positional args in cli bug
## [0.3.8] - 2022-03-04
## Added
- added command history file to shell-repls
- added tests and example tests to run with devices
- added shell support for command concatenation (`&&`)
- added shell basic support for pipe device output to a local file (`>, >>, |`)
- `register` command to register a device name or group as a callable shell function
- `ota` command to do OTA Firmware updates (esp32). This needs `ota.py` for network (LAN/WiFi) or `otable.py` for BLE. (`upyutils` directory) and `firmware.bin` file in `build-GENERIC_OTA` or from micropython esp32-ota downloads.
- `ota` with `-sec` option to do OTA over TLS. (This needs `kg ssl` first).
- `rsa sign`, `rsa verify` commands to sign file or verify signatures made with device RSA key
- `kg rsa host`, to generate a host RSA key pair and send public key to device.
- `rsa sign host [FILE]` to sign a file with host RSA key.
- `rsa verify host [FILE]` to verify in the device a file signature made with host RSA key
- `rsa` lib in `upyutils` to support RSA key load, sign, verify, encrypt, decrypt,
generation and export in PEM format
- `kg rsa` added option `-rkey` to remove RSA private key from the host, so in combination with `-tfkey` option, the RSA private key will be stored only in the device.
- `rsa auth` to authenticate a device with a RSA encrypted challenge.
- `shasum.py` lib in `upyutils` to support hash SHA-256 check
- `shasum` and `shasum_c` to compute hash SHA-256 of files and check shasum files.
- `ls` command to improve ls from `upysh`, with multiple dirs and pattern matching
- `cat` command that accepts multiple files and pattern matching
- refactored help
- dropped wlan utils
- dropped prototype commands (this will be a stand alone cli or add-on for upydev)
# Fix
- `mpyx` command with multiple files
- `kg wr` now use RSA public key for password derivation and send encrypted password
that is decrypted and stored in device.
- Load time in MacOS caused by upydevice->BleDevice->bleak->corebluetooth
- Save in ecdsa key/cert directly in `DER`, no need for `openssl` in `PATH`.
- Allow mdns name in ssl certificate.
- Improved file io operations (`put`, `get`, `dsync` ...), more flexibility indicating files / dirs / pattern matching etc. Improved overall performance, specially for WebSocketDevices.
- fix `ping`, `probe` for zerotier devices (this needs updated config, see how-to section in docs.)
- fix `update_upyutils` if dir `lib` does not exists.
- refactored shells-repls, now `shl`, or `rpl` commands only
## [0.3.7] 2021-12-16
## Added
- `rssi` command in shell repls to get RSSI value (Wifi or Ble)
- `make_sgroup` / `mksg` command to create a subgroup of an existing group of devices.
- `set_hostname` command to set hostname of the device for dhcp service (needs *wpa_supplicant.py*)
- `set_localname` command to set localname of the device for ble gap/advertising name (needs *ble_uart_peripheral.py*)
- Now `-@` option accepts multiple devices, names with `*` wildcard or global group `gg` or other group names, e.g. `upydev check -i -@ esp\* dev{1..4} mytestgroup`
will expand to all devices that start with `esp` , `dev1 dev2 dev3 dev4` and devices configured in `mytestgroup`
## Fix
- `firmwaretools.get_fw_versions` update after `micropython.org\all` not working anymore.
- fix device name instead of `None` in `put`, `get`, `fget`, `dsync` in `sslweb_repl` with `-nem` mode enabled.
- fix `fget` disconnection error.

## [0.3.6] 2021-10-24
## Added
- `zerotier` compatibility through raspberry pi bridge (port forwarding + ufw route rules) and `-zt [HOST IP/BRIDGE IP]` option to indicate host zerotier ip and raspberry pi bridge local ip for ssl
shell-repl mode. Also `-zt` option with config command compatibility.
- mDNS `.local`/`dhcp_hostname` compatibility, so device configuration works across networks, e.g, `-t` argument now can accept `-t mydev.local` and resolve to device ip.
- added `-zt [IP]` in `keygen ssl` for zerotier `wss-sslwebrepl` mode
- `wpa_supplicant.py` utility in upyutils
- New buzzer effects in `buzzertools.py`: `error`, `warning_call`, `phone_call`, `door_bell`.
- Added docs for `zerotier` bridge configuration in `HOWTO`
## Fix
- `fget` error on connection, and unintended verbose output on close.
- `check -i`, `info` commands in `-apmd`, if connected to AP of the device.
- fix ssl cert authentication in `wss` mode if using `.local`/`dhcp_hostname` target.

## [0.3.5] 2021-09-09
## Fix
- `set_ntptime` with WebSocket Devices
- blerepl paste command
- paste command in sslweb_repl -nem mode
## Added
- get rssi value in blerepl

## [0.3.4] 2021-08-12
### Fix
- Fix autocomplete on tab in REPL while `import` or `from X import` in SERIAL REPL
- Command names change:
  * `filesize` --> `du`,
  * `filesys_info` --> `df`
  * `fw` --> `fwr`
  * `d_sync` --> `dsync`
  * `sync` --> `fget`

- Catching passwords too short for AP configuration
- Refactor help info organisation for easy reading.
- Refactor device management actions (configuration, groups...)
- Refactor firmware actions, `-i` option to check firmware and platform match. (from firmware file name)
- `tree` command in upysh2.py unix/linux like.
- drop ``mpy-cross`` dependency, better to build from source.
### Added
- Autocomplete `shr@` , `ssl@`, `wssl@` and `ble@` commands with saved devices in UPY_G global group
- `pytest` and `pytest-setup` command in SHELL-REPLS
- `pytest` and `pytest-setup` command mode in CLI.
- commands that start with `%` or not registered in SHELL-REPLS commands are forwarded to local shell (works with alias too)
- Ble SHELL-REPL `ble@[device]`
- `set` command to set current device configuration of a device saved in global group
- `check` command to see device configuration (`-i` to see more info if device available)
- `dm` command to see help about DEVICE MANAGEMENT
- `gc` command to see help about GENERAL COMMANDS
- `wu` command to see help about WLAN UTILS COMMANDS
- `sd` command to see help about SD UTILS COMMANDS
- `pro` command to see help about PROTOTYPE COMMANDS
- `gg` to see global group
- `-gg` flag to set -G flag to global group (`-gg` == `-G UPY_G`)
- `-ggp` flag to set -GP flag to global group (`-ggp` == `-GP UPY_G`)
- `%` before any command e.g. `%config` display help info about that command.
- `probe` command to test if a device/group is reachable
- `scan` command to look for devices (serial [-sr], network [-nt] or ble [-bl])
- `shl` / `shell`, and `rpl` / `repl` commands works with `@` or `-@` and will detect device type, redirecting to the proper SHELL-REPL / REPL type.
- `put`, `get`, `install`, `fget`, `dsync`, file operations now support indicating file/files/cwd/expression as a second argument, e.g, `upydev put this_file.py`, `upydev put demo_*.py`,  `upydev put fileone.py filetwo.py`, `upydev put cwd -dir lib` ...
- Alias and positional args for keygen/firmware/flash actions `upydev kg/keygen rsa/wr/ssl`, `upydev fwr get/list latest`, `upydev flash esp32-idf4-20200122-v1.12-76-gdccace6f3.bin`,`upydev flash pybv11-20200114-v1.12-63-g1c849d63a.dfu` ...
- Alias and positional args for `make_group, mg_group` actions `mkgroup/mkg, mggroup/mgg`, and `see`. e.g `upydev see MY_GROUP`, `upydev mkg MY_GROUP -devs mydevtest <IP_ADDRESS> mypasswd`, `upydev mgg MY_GROUP -add sdev2 /dev/tty.SLAB_USBtoUART 115200`
- Alias and positional args for SD actions `upydev sd enable/init/deinit/auto`
- upydev pypi version checking on `h`/ `help` command.
- `-gf` flag to indicate a group file operation (files are upload/download to/from a directory with the name of the device in the current working directory). It uses local group configuration file if available (unless -gg flag is indicated), otherwise it falls back to global group configuration
- `-rf` flag for `dsync` mode, to remove files or directories deleted in local dir.
- `-to [devname]` flag now can be used with `-tfkey` to transfer keys by USB/Serial
- `-d ` flag for `dsync` to sync from device to host.
- `backup` command == `dsync . -d` to make a backup of the device filesystem
- `rsync` command == `dsync [DIR] -rf -wdl` to recursively sync (deleting files too)

## [0.3.3] - 2020-06-07
### Fix
- Fix `git status dev` aware of current branch
- Fix `d_sync`, in SHELL-REPLS, support root directory `.` will sync local cwd into device cwd
- Fix `put` for pyboard in SERIAL SHELL-REPL mode
- Fix `fw` for downloading and flashing firmware to pyboard, esp in SERIAL SHELL-REPL mode, now asserts serial port is available after flashing.
- Fix `battery` error if command fails.

## [0.3.2] - 2020-05-27
### Fix
- Fix `set_localtime` for pyboard in SERIAL SHELL-REPL mode
- Fix [issue \#8](https://github.com/Carglglz/upydev/issues/8)
- Fix `d_sync` now supports sync of cwd, use `d_sync` without -dir flag to synchronize current working directory
- Fix error messages in SSLWebREPL
- Fix, now to toggle between shell and REPL modes, there is no need to press ENTER
- Fix, toggle local path in prompt without pressing ENTER
- Fix, `apscan` mode now show MAC address instead of bytes
### Added
- Watch mode with `-wdl` flag for `put -fre` and `d_sync` modes. Uploads only new or modified files

## [0.3.1] - 2020-04-23
### Fix
- Fix for host ip determination

## [0.3.0] - 2020-04-13
### Fix
- Minor fix for host ip determination (while ECDSA key is generated and for SSLWebREPL mode)

## [0.2.9] - 2020-02-24
### Added
- jupyter console integration through jupyter-micropython-upydevice
- 'jupyterc' command in upydev and in SHELLS
### Fix
- autocompletion for zsh
- git integration and 'git init dev' command for SHELLS
- sslgen_key command help

## [0.2.8] - 2020-02-16
### Added
- emacs in nw mode by default
- tig integration for git workflow: [tig](https://jonas.github.io/tig/)
### Fix
- in SERIAL SHELL-REPL 'fw update' command for esp8266

## [0.2.7] - 2020-02-08
### Added
- To be able to use this new commands update the SSL key and cert with 'upydev sslgen_key -tfkey' and do 'upydev update_upyutils'
- 'wss_repl.py' and 'wss_helper.py' to enable WebSecureREPL in the device
- 'wssl' command e.g. 'upydev wssl@esp_room1' or 'upydev<EMAIL_ADDRESS>for a E2E encryption Terminal (WebSecureREPL + SSL SHELL-REPL)
- 'wss on/off' command in SSL REPL to enable/disable WebSecureREPL
- 'set_wss' upydev command to switch WebREPL to WebSecureREPL and
'set_wss -wss' to switch back to WebREPL
- if WebSecureREPL enabled, upydev 'put' and 'get' commands support WebSecureREPL file transfer using '-wss' option e.g. 'upydev put -f foo.py -wss'

## [0.2.6] 2020-02-02
### Added
- 'bat' command output style configurable and line numbers (inspired by https://github.com/willmcgugan/rich)
- 'batstyle' command to configure 'bat' and 'batl' syntax highlight, 'batstyle -a' list all the themes available (from pygments), default is 'monokai'
- 'upy-config' command for shells, to configure network (connect to a WLAN or set an AP) and configure Interfaces (just I2C by now) through interactive dialogs
### Fix
- cat, bat in SSLWebREPL if last line does not end with new line ('\\n')
- ECDSA key (SECP256R1_SHA256 method to meet IETF recommendations)
- 'git status dev' now tracks all commits (inside or outside the shell)

## [0.2.5] 2020-01-26
### Fix
- put method in SERIAL SHELL for esp32/8266

## [0.2.4] - 2020-01-26
### Added
- 'timeit' command to measure execution time of a script/command (for SSL/SERIAL SHELLS)
- 'i2c' +'config/scan' to configure i2c and scan to find i2c devices (for SSL/SERIAL SHELLS)
- 'git' commands integration + 'git push dev', 'git log dev [-a]', 'git clone_dev' and 'git status dev' to integrate git workflow into a project (for SSL/SERIAL SHELLS) ('git' needs to be available in $PATH)
- 'emacs' to edit a file/script with emacs ('emacs' need to be available in $PATH)
### Fix
- Autocompletion in REPLS improved
- pyboard firmware improved (automatically enables DFU mode, jumper doesn't needed)
- Changed upydev 'sslgen_rsakey' to 'sslgen_key' key now is a ECDSA key (SECP384R1 method)
- Cipher suite is now TLSv1.2 @ ECDHE-ECDSA-AES128-CCM8 - 128 bits which is recommended for constrained devices / IOT (This requires a recent version of python-ssl)
- put method (faster) in SERIAL SHELL

## [0.2.3] - 2020-01-19
### Added
- fw , flash (and fw update which is 'fw get' + 'flash') commands for SERIAL SHELL
- 'du' command for disk usage statistics (unix like) (SSL/SERIAL SHELLS)
- 'dsync' command to recursively sync directories for (SSL/SERIAL SHELLS)
- 'uping' and 'lping' commands for device and host pings commands (SSL/SERIAL SHELLS)
- wildcard "\*" or [dir] for 'ls' and 'lsl' command e.g. "ls \*.py" or "ls my_dir" (SSL/SERIAL SHELLS)
- install available for pyboard too (experimental) (SERIAL SHELL)
- 'pkg_info' command to see the PGK-INFO file of a module if available (SSL/SERIAL SHELLS)
- 'upipl' to list available packages at pypi.org or micropython.org/pi
e.g. 'upipl' or 'upipl [module]' (SSL/SERIAL SHELLS)
- 'update_upyutils' command for SERIAL SHELL
### Fix
- fw and flash commands of upydev

## [0.2.2] 2020-01-13
### Added
- Autocompletion on REPLS (SSL/SERIAL) for "from foo import X" will show option of what to import/autocomplete on match for frozen modules too.
### Fix
- Autocompletion on REPLS (SSL/SERIAL) for "from foo import X" will show option of what to import/autocomplete on match (bug fix)

## [0.2.1] 2020-01-12
### Added
- Autocompletion on REPLS (SSL/SERIAL) for "from foo import X" will show option of what to import/autocomplete on match
- CTRL-u deprecated (no encryption toggle), CTRL-p now shows RAM STATUS (used/free)
- CTRL-e in SHELLS (SSL/SERIAL) moves cursor to end of the line (if not in edit mode)
- 'batl' command for SHELLS(SSL/SERIAL) to print local file with python syntax highlighting

## [0.2.0] - 2020-01-10
### Added
- serial terminal SHELL-REPL mode (same style as SSLWebREPL) "sh_srepl"
- "shr@[dev]" shorcut to "sh_srepl" mode
- commands view, bat, rcat for SSLWebREPL and Serial SHELL-REPL
### Fix
- ssl_wrepl netscan command for esp8266
- Add help of latest upydev commands
- see -c command help info text wraps at terminal length

## [0.1.9] - 2020-01-07
### Added
- Mode to generate RSA key and self signed certificate ('sslgen_rsakey')
- 'ssl_socket_client_server.py' and 'ssl_repl.py' added to the scripts that are updated with 'update_upyutils' mode
- 'upysh2.py' implements the 'tree' command to print filesystem in a tree view
- SSLWebREPL shell terminal mode (experimental) (see DOCS)
- "ssl@[dev]" shortcut to "ssl_wrepl" mode
- Progress bar length and percentage fix, and estimated time
- put and get commands now works in SSL mode too

## [0.1.8] - 2019-12-29
### Added
- New CryptoWebREPL shell local commands ('lsof', 'l_ifconfig', 'l_ifconfig_t') (last one works only on MacOS, linux pending...). Network utilities
- New CryptoWebREPL shell local command 'docs' to open MicroPython Docs site in browser
- New CryptoWebREPL shell local command 'reload' to delete a module/script from sys.path so it can be imported/run again
### Fix
- Paste mode in unencrypted mode
- Fix -h command error if there is no 'UPY_G.config' global group file
- Installation dependencies

## [0.1.7] - 2019-12-24
### Added
- Mode to generate RSA private key, and send it to the device ('gen_rsakey')
- Mode to refresh the WebREPL password with a random password
 and don't leave explicit trace in tcp Websocket packets.
 (this needs upysecrets.py in the device, more info in DOCS/help) ('rf_wrkey')
 This can be automatically after a wrepl session using '-rkey' option (e.g. upydev wrepl -rkey)
- 'upysecrets.py' added to the scripts that are updated with 'update_upyutils' mode
- 'upysh2.py' implements the 'tree' command to print filesystem in a tree view
- "crypto"-webrepl-shell mode (experimental) (see DOCS)
- New option '-@' to redirect a command to a specific device saved in global
  Group 'UPY_G' (this needs to be created with the 'make_group' command )
  (e.g. "upydev make_group -g -f UPY_G -devs foo_device <IP_ADDRESS> myfoopass")
  Then commands can be redirected to this device from anywhere using -@ foo_device (e.g. upydev ping -@ foo_device )
### Fix
- Progress bar animation now available in put/get/sync modes and auto-adjust to terminal size
- sync mode improved

## [0.1.6] - 2019-12-08
### Added
- New 'update_upyutils' mode to update the latest versions of sync_tool.py,
  upylog.py and upynotify.py (will be uploaded to '/lib' folder)
- New 'debug' mode to execute a local script line by line in the target
  upydevice, use -f option to indicate the file
- improved 'fw' mode options of 'list latest' and 'get latest', now option
  '-n' can be used to filter the results (e.g: -n ota or -n idf4)
### Fix
- BSSID column of netscan command now shows mac address instead of bytes format
- refactor of sync mode (shows transfer progress animation also)

## [0.1.5] - 2019-11-23
### Added
- New mode d_sync to sync recursively a directory (all files and subfolders)
(this needs the new version of *sync_tool.py* from upyutils directory to be
  uploaded to the device)
  * the *-tree* option to get the dir tree view is from:
  "https://stackoverflow.com/questions/9727673/list-directory-tree-structure-in-python" @abstrus
  * to sync to an sd card, sd must be mounted as 'sd', and then use '-s sd' option
- New modes to execute scripts:
  * sysctl: to execute scripts in the upydevice in non-blocking mode (won't
            follow the output) (use -start and -stop options)
  * log: to execute scripts and log its output; use -dslev and -dflev to
        indicate log level of sys.stdout and log file. To stop use CTRL-C
        Use option -daemon to log in 'daemon-like' mode, the output is logged
        to daemon log file. To follow use -follow option. (detailed explanation
          in docs)

- New mode 'stream_test': (this needs the new version of *sync_tool.py*)
  To test  wireless transmission speed of data through sockets (TCP).
   In the default test the upydevice sends 10 MB of data in chunks of 20 kB,
   then speed is obtained from total time elapsed and amount of data received.
   See docs for more info.

### Fix
- Code refactoring using upydevice

## [0.1.4] - 2019-11-12
### Added
- New 'upyutils' script 'upylog.py' a modification of logging module to be able to log messages or exceptions to file and two formats to choose ([NAME] [LVL]
  MSG) or ([DATETIME] [NAME] [LVL] MSG)
- option to log to file in sd (mounted as 'sd')
- New error.log content test in diagnose mode
- New errlog command mode to see 'error.log' file if any with option to indicate
source sd.
- New 'upyutils' script 'upynotify.py' to make easier physical debugging (led, and buzzer notifications)
### Fix
- Command options -g , -st, -rep, -apmd (do not need any argument, they store true if used)
- Option "-md local" in diagnose option (deprecated) use -apmd instead

## [0.1.3] - 2019-11-02
### Fix
- package requests dependency added
### Added
- KeyboardInterrupt command (kbi) to stop for/while loops
- New mode diagnose
- New -apmd option to set target to <IP_ADDRESS> (AP default)

## [0.1.2] - 2019-10-14
### Added
- '-dir' option to put/get commands to select in which directory to put file or get a file from

## [0.1.1] - 2019-10-01
### Fix
- Dedent with shift-tab in wrepl (normal mode and paste mode)
### Added
- find command to scan for upy devices in the WLAN
- WIFI UTILS commands

## [0.1.0] - 2019-09-19
### Fix
- toggle wrepl in synchronized mode for ssh session
- Group command mode -devs option fix
### Added
- toggle autosuggest mode (fish shell like)
- mg_group command to manage groups to add or remove devices

## [0.0.9] - 2019-08-22
### Added
- NEW WebREPL Terminal mode (Custom keybindings and autocompletion on tab)

## [0.0.8] - 2019-08-18
### Fix
- Fix long output indentation (with head or cat upysh command for example)
### Added
- Autocompletion groups for -G and -GP options

## [0.0.7] - 2019-08-16
### Fix
- Fix help info file.

## [0.0.6] - 2019-08-16
### Added
- Buzz interrupt reverse operation mode (falling)
- Serial repl using picocom
- Group command parallel mode more reliable
- Sensors i2c pin configuration with -i2c option

## [0.0.5] - 2019-08-12

### Added
- Group command parallel mode

repository: Carglglz/upydev

gha_language: Python

repo_name: Carglglz/upydev


text:
# Gitup

Gitup shows some recent stats about your `git`-activity such as _commits_, `diffs` and various stats that helps to remind you of what you have acomplished during the day or week. The name is a combination of _git_ and _standup_.

![Show branch additions and deletions](https://i.imgur.com/PWQTWfp.png)

## Features

## Requirements

## Extension Settings

## Known Issues

## Release Notes

### 1.0.0

-----------------------------------------------------------------------------------------------------------

## Development

This project is based on [**vscode-webview-react** by _rebornix_](https://github.com/rebornix/vscode-webview-react)

### Run

Run following commands in the terminal

```shell
yarn
yarn build
```
And then press `F5`, in Extension Development Host session, run `Gitup` command from command palette.

### Under the hood

Things we did on top of Create React App TypeScript template

* We inline `index.html` content in `ext-src/extension.ts` when creating the webview
* We set strict security policy for accessing resources in the webview.
  * Only resources in `/build` can be accessed
  * Only resources whose scheme is `vscode-resource` can be accessed.
* For all resources we are going to use in the webview, we change their schemes to `vscode-resource`
* Since we only allow local resources, absolute path for styles/images (e.g., `/static/media/logo.svg`) will not work. We add a `.env` file which sets `PUBLIC_URL` to `./` and after bundling, resource urls will be relative.
* We add baseUrl `<base href="${vscode.Uri.file(path.join(this._extensionPath, 'build')).with({ scheme: 'vscode-resource' })}/">` and then all relative paths work.

repository: jesperorb-caspeco/gitup-react

gha_language: TypeScript

repo_name: jesperorb-caspeco/gitup-react


text:
node-bandcamp [![npm version](https://badge.fury.io/js/node-bandcamp.svg)](http://badge.fury.io/js/node-bandcamp)
=================================================================================================================

node.js (unofficial) Bandcamp API.

---

### Getting Started

1. `npm install --save node-bandcamp`
2. `var bandcamp = require('node-bandcamp')`

---

### Searching for tracks

Specific tracks can be searched for using the `trackSearch` method. This function takes the following parameters:

- `query` (str): The search query for which you want results.
- `limit` (defaults to 20): The maximum number of results you want returned. Defaults to `20`.

The function will recursively query Bandcamp until results are exhausted or the limit is hit.

```javascript
var bandcamp = require('node-bandcamp');

bandcamp.trackSearch('tibetan pop stars', 30).then(function(results){
  // do something with search results
}).catch(function(err) {
  // handle error
});
```

Returns a `promise`, which will eventually resolve as an array of search results. Results are in the format:

```json
[
  {
    "title": "title",
    "album": "album name",
    "artist": "artist name",
    "image": "URL to track artwork",
    "url": "URL to be passed to getTrack method"
  },
  ...
]

```

---

### Streaming a track

A specific track can be streamed using the `getTrack` method. This function takes just one parameter, the Bandcamp URL string retrieved using the previously discussed `trackSearch` method.

```javascript
var bandcamp = require('node-bandcamp');

bandcamp.getTrack('http://hopalong.bandcamp.com/track/tibetan-pop-stars').then(function(stream) {
  // pipe audio stream to res, etc.
}).catch(function(err) {
  // handle error
});
```

Returns a `promise`, which will eventually resolve as an audio stream.

---

### Getting track details

The details for a specific track (title, duration, etc.) can be retrieved using the `getDetails` method. This function takes the Bandcamp URL string for the track as a parameter, and returns a `promise` which will eventually resolve as an object in the same format as search results:

```javascript
var bandcamp = require('node-bandcamp');

bandcamp.getDetails('http://hopalong.bandcamp.com/track/tibetan-pop-stars').then(function(details) {
  // return details to res, etc.
}).catch(function(err) {
  // handle error
});
```

and the promise resolves with an object in the format:
```json
{
  "title": "title",
  "album": "album name",
  "artist": "artist name",
  "image": "URL to track artwork",
  "url": "URL to be passed to getTrack method"
}
```

---

### Testing

All tests for this package are within the `__tests__/` directory. If you wish to run the tests:

1. `git clone git@github.com:jakemmarsh/node-bandcamp.git`
2. `cd node-bandcamp`
3. `npm install`
4. `npm test`

repository: jakemmarsh/node-bandcamp

repo_name: jakemmarsh/node-bandcamp


text:
---
seo:
  title: Sender Reputation
  description: A sender has a reputation score that helps identify whether they are a spammer.
  keywords: spam, reputation, email sender, sender reputation, reputation score
title: Sender Reputation
weight: 0
layout: page
navigation:
  show: false
---

![]({{root_url}}/images/reputation_1.png "Sender Reputation")

Your Reputation score is given in a range from 0-100 (100 being the best) and is a calculated based on information abour your sending habits, specifically how many requests are able to be delivered. This includes:

-	Low bounce rates, drop rates, unsubscribes
-	Few spam complaints
-	The ratio of invalid emails sent out
-	Consistent mailing volume

It is calculated by taking the percentage of requests that are able to be delivered.

To achieve the highest reputation possible, we recommend that you have an industry standard opt-in process, don't rent or buy email lists and always keep your lists clean.

{% anchor h2 %}
The Effect of Reputation on your Account
{% endanchor %}

Depending on your package level, your internal SendGrid Reputation can matter more or less. At all levels, if your Reputation is low enough, your account will come under review by our Compliance team and your account may be suspended for review.

If you have an account lower than Pro 100k, for the first month your account is in a "New" IP cluster while we learn your sending habits. After that, you're placed in an IP cluster with other customers with the same reputation range as you. This keeps the good senders together, and prevents the poorer senders from harming the good ones.

If you're on a Pro 100k package or higher, you have a dedicated IP, and our internal reputation metric may not have as much weight as external reputation metrics. Instead, you should keep a closer eye on your external reputation. Every receiving server has their own score of you, which they generally "forget" after a couple of weeks of no sending habits. Sites such as [SenderScore](https://www.senderscore.org/) are a good way to get an idea of your external reputation.

{% anchor h2 %}
SendGrid's Reputation Policy
{% endanchor %}

We regularly monitor our customers' account reputation for abuse and undesirable sending practices.

Our general guidelines for account reputation take into account the following reputation thresholds:

* **Reputation above 80%:** Congratulations, you have a good sending reputation and there are no issues at this time.
* **Reputation between 70% and 80%:** This is considered a poor reputation and actions should be taken immediately to identify and fix problems with your sending practices.
* **Reputation below 70%:** Accounts with reputations below 70% may be subject to immediate suspension or termination.

repository: jstvz/docs

gha_language: HTML

repo_name: jstvz/docs


text:
# KBase Index Runner / Knowledge Engine

This is a background service that listens to events from the KBase Workspace
and automatically generates graph data and search indexes to enrich the
workspace object data.

It consumes event from a Kafka topic and sends data into ArangoDB and
Elasticsearch. It is designed to have the service replicated in Rancher. Be
sure to partition the topic to at least the number of running workers.

## Development

Start the servers:

```sh
docker-compose up
```

Run the tests (servers need not be running, and will be shut down if they are):

```sh
make test
```

Note that `docker-compose run`, and therefore `make test`, will only build a Docker image for the
application if no image exists, and therefore `Dockerfile` changes will not be included in any
tests. To include `Dockerfile` changes in tests, do one of the following, in order of
destructiveness:

1. `docker-compose build` will build any images with changed `Dockerfile`s using the build cache.
2. Delete the `index_runner_deluxe_app` image. `make test` will then rebuild the application image
   from scratch.
3. `make reset`. This will remove all test images and rebuild them from scratch.

## Config

You can set the following env vars:

* `SKIP_RELENG` - skip imports into the relation engine (ArangoDB)
* `SKIP_FEATURES` - skip any importing or indexing of genome features
* `SKIP_INDICES` - comma-separated list of index names that the service will not write into.
* `ELASTICSEARCH_HOST` - host name of the elasticsearch server to use (do not prepend protocol)
* `ELASTICSEARCH_PORT` - port to use for the elasticsearch server
* `KBASE_ENDPOINT` - URL of kbase API services (default is "https://ci.kbase.us/services")
* `WS_URL` - URL of the workspace API (default is to append "/ws" to KBASE_ENDPOINT)
* `CATALOG_URL` - URL of the catalog API (default is to append "/catalog" to KBASE_ENDPOINT)
* `RE_URL` - URL of the relation engine API (default is to append "/relation_engine_api" to KBASE_ENDPOINT)
* `GLOBAL_CONFIG_URL` - Optional URL of a specific index_runner_spec configuration file to use. Set this to use a specific config release that will not automatically update.
* `GITHUB_RELEASE_URL` - Optional URL of the latest release information for the index_runner_spec. Defaults to "https://api.github.com/repos/kbase/index_runner_spec/releases/latest". Use this setting to have the config file automatically keep up-to-date with the latest config changes. Ignored if GLOBAL_CONFIG_URL is provided.
* `GITHUB_TOKEN` - Optional Github token (https://github.com/settings/tokens) to use when fetching config updates. Avoids any Github API usage limit errors.
* `WORKSPACE_TOKEN` - Required KBase authentication token for accessing the workspace API
* `MOUNT_DIR` - Directory that can be used for local files when running SDK indexer apps (defaults to current working directory).
* `RE_API_TOKEN` - Required KBase auth token for accessing the relation engine API
* `KAFKA_SERVER` - URL of the Kafka server
* `KAFKA_CLIENTGROUP` - Name of the Kafka client group that the consumer will join
* `ERROR_INDEX_NAME` - Name of the index in which we store errors (defaults to "indexing_errors")
* `ELASTICSEARCH_INDEX_PREFIX` - Name of the prefix to use for all indexes (defaults to "search2")
* `KAFKA_WORKSPACE_TOPIC` - Name of the topic to consume workspace events from (defaults to "workspaceevents")
* `KAFKA_ADMIN_TOPIC` - Name of the topic to consume indexer admin events from (defaults to "indexeradminevents")

## Admininstration

Some CLI admin tools are provided in the `indexer_admin` script. For a running container:

```sh
docker exec container_name indexer_admin
```

Examples:

Show command help: `indexer_admin -h`

_Reindex a specific object_

```sh
# Reindex only if the doc does not exist
indexer_admin reindex --ref 11/22

# Reindex and overwrite any existing doc
indexer_admin reindex --ref 11/22 --overwrite
```

_Reindex a workspace_

```sh
# Reindex only objects that do not exist
indexer_admin reindex --ref 11

# Reindex and overwrite any existing docs
indexer_admin reindex --ref 11 --overwrite
```

_Reindex a range of workspaces_

```sh
# Index workspaces 1 through 10
indexer_admin reindex_range --min 1 --max 10

# Index workspaces 1 through 10, overwriting existing docs
indexer_admin reindex_range --min 1 --max 10 --overwrite

# Index from workspace 1 through 1000
indexer_admin reindex_range  --max 1000
```

_Get a count of indexing errors_

```sh
# Total error count
indexer_admin err_count

# Error count by type
indexer_admin err_count --by-type
```

_Get all the upas that had an error_

Note: this outputs ALL upas to stdout. You might want to pipe this into a file.

```sh
indexer_admin err_upas
```

_Index all objects of a certain type_

```sh
indexer_admin reindex_type --type "KBaseNarrative.Narrative"
```

### Deployment

Build the image:

```sh
IMAGE_NAME=kbase/index_runner2:{VERSION} sh hooks/build
```

Push to docker hub

```sh
docker push kbase/index_runner2:{VERSION}
```

### Project anatomy

* The main process and entrypoint for the app lives in `./src/index_runner/main.py`
* The entrypoint for ES indexing is in `./src/index_runner/es_indexer.py`
  * Local elasticsearch indexers live in `./src/index_runner/es_indexers`
* The entrypoint for Arango imports is in `./src/index_runner/releng_importer.py`
  * Type-specific RE importers are found in `./src/index_runner/releng`

## KBase Relation Engine/Knowledge Engine Stack

* [Index Runner](https://github.com/kbaseIncubator/index_runner_deluxe) - Kafka worker to construct indexes and documents and save them to Elasticsearch and Arango.
* [Search API](https://github.com/kbaseIncubator/search_api_deluxe) - HTTP API for performing search queries.
* [Search Config](https://github.com/kbaseIncubator/search_config) - Global search configuration.
* [Relation Engine API](https://github.com/kbase/relation_engine_api) - HTTP API for querying the relation engine
* [Relation Engine Spec](https://github.com/kbase/relation_engine_spec) -  specifications for types, queries, views, etc in the graph database

## Creating an SDK indexer application

The index runner has the ability to use SDK modules for indexing. Modules should use the following as input:

```
typedef structure {
	string obj_data_path;
	string ws_info_path;
	string obj_data_v1_path;
} inputParams;
```

Each of the above fields are paths to JSON files that contain relevant object
data (written to disk to avoid using too much memory).

The output for the indexer module is expected to be written to a file.
Each line of the returned file corresponds to one JSON document to be saved in Elasticsearch.

_Example output:_

```json
{ "doc": { "key1": "value1", "key2": "value2" } }
```

If the module outputs more than one Elasticsearch document, the `sub_type` and `sub_id` fields must be specified for all but the 'parent' document. The `sub_type` field should be the same for all the sub objects and should only contain alphanumeric characters, excluding whitespaces. The `sub_id` field MUST be unique for each subobject. If the `sub_id` field is not unique, only one of the objects will be indexed.

_Example output:_

```
{"doc": {"key1": 1, "key2": "string of interest"}}
{"doc": {"sub_key1": 3, "sub_key2": "value"}, "sub_type": "feature", "sub_id": "LKJCID"}
{"doc": {"sub_key1": 6, "sub_key2": "different value"}, "sub_type": "feature", "sub_id": "OIMQME"}
```

The output from your SDK module should contain the filepath for the above data. The file must be written to the scratch space of the sdk application, whose path can be accessed from `config['scratch']`.

```
# Example output from your SDK module's indexer method:
result = {
	"filepath": os.path.join(config['scratch'], "my_output.json")
}
```

NOTE: Before you can use an SDK module as an indexer, it must be registered in the KBase environment you are using.

### Registering your SDK indexer module

To register your application with the index runner, you must open a pull request against the [search config repo](https://github.com/kbaseIncubator/search_config). In the `config.yaml` file, the `sdk_indexer_apps` section is a mapping from KBase object types (excluding version) to their appropriate SDK module and method.

Example:

```
  KBaseMatrices.MetaboliteMatrix:
    sdk_app: kbasematrices_indexer # required
    sdk_func: run_kbasematrices_indexer # required
    sub_obj_index: attribute_mapping # required for indexers that have subobjects
```

NOTE: `attribute_mapping` is the string name of the sub_obj_index for `KBaseMatrices.MetaboliteMatrix`.

You may also specify `sdk_version`, which corresponds to the version of the SDK module you would like to use.

An Elasticsearch type mapping is also required. This can be added to the `mappings` field in the config. Several examples are available in the `config.yaml` file.


text:
# Chapter 7: Transmission Media
## Introduction
Transmission media are actually located below the physical layer and are directly controlled by the physical layer.

![](./static/ch7_1.png)

In telecommunications, transmission media can be divided into two broad categories: guided and unguided. Guided media include twisted-pair cable, coaxial cable, and fiber-optic cable. Unguided medium is free space.

![](./static/ch7_2.png)

## Guided Media
Guided media, which are those that provide a conduit from one device to another, include twisted-pair cable, coaxial cable, and fiber-optic cable. A signal traveling along any of these media is directed and contained by the physical limits of the medium.
- `Twisted-pair` and `coaxial cable`: use metallic (copper) conductors that accept and transport signals in the form of electric current.
- `Optical fiber`: is a cable that accepts and transports signals in the form of light.

### Twisted-Pair Cable

![](./static/ch7_3.png)

![](./static/ch7_5.png)

![](./static/ch7_6.png)

### Coaxial Cable

![](./static/ch7_7.png)

![](./static/ch7_8.png)

![](./static/ch7_9.png)

### Fiber-Optic Cable

![](./static/ch7_11.png)

![](./static/ch7_15.png)

![](./static/ch7_16.png)

## Unguided Media: Wireless
Unguided medium transport electromagnetic waves without using a physical conductor.

![](./static/ch7_17.png)

## Summary
- Transmission media are actually located below the physical layer and are directly controlled by the physical layer. We could say that transmission media belong to layer zero.
- A guided medium provides a physical conduit from one device to another.
- Twisted-pair cable consists of two insulated copper wires twisted together. Twisted-pair cable is used for voice and data communications.
- Coaxial cable consists of a central conductor and a shield. Coaxial cable is used in cable TV networks and traditional Ethernet LANs.
- Fiber-optic cables are composed of a glass or plastic inner core surrounded by cladding, all encased in an outside jacket. Fiber-optic transmission is becoming increasingly popular due to its noise resistance, low attenuation, and high-bandwidth capabilities. Fiber-optic cable is used in backbone networks, cable TV networks, and Fast Ethernet networks.
- Unguided media (free space) transport electromagnetic waves without the use of a physical conductor.
- Wireless data are transmitted through ground propagation, sky propagation, and line-of-sight propagation. Wireless waves can be classified as radio waves, microwaves, or infrared waves. Radio waves are omnidirectional; microwaves are unidirectional. Microwaves are used for cellular phone, satellite, and wireless LAN communications. Infrared waves are used for short-range communications such as those between a PC and a peripheral device. They can also be used for indoor LANs.

repository: Dakobrah/Life-long-Learner

repo_name: Dakobrah/Life-long-Learner


text:
# Change Log

## v0.2.1

* `Client` no longer implements `DerefMut` which was not needed anyways.
* `Client.transaction` now returns a wrapped transaction object which
    utilizes the statement cache of the wrapped client.

## v0.2.0

* First release

repository: vpzomtrrfrt/deadpool

gha_license_id: NOASSERTION

repo_name: vpzomtrrfrt/deadpool


text:
## Chapter 9

# Technologies Used in the Reference Implementation (Chapter Title)

# Windows Azure Service Bus

    Comment [DRB]:
	Cover Brokered Messages, Queues, Topics, Subscriptions

This section is not intended to provide an in-depth description of the Windows Azure Service Bus, rather it is intended to highlight those features that may prove useful in implementing the CQRS pattern and event sourcing. The section "Further Information" below, includes links to additional resources for you to learn more.

The Windows Azure Service Bus provides a cloud-hosted, reliable messaging service. It operates in one of two modes:

* **Relayed.** Relayed messaging provides a direct connection between clients who need to perform request/response messaging, one-way messaging, or peer-to-peer messaging.
* **Brokered.** Brokered messaging provides durable, asynchronous messaging between clients that are not necessarily connected at the same time. Brokered messaging supports both queue and publish/subscribe topologies.

In the context of CQRS and Event Sourcing, brokered messaging can provide the necessary messaging infrastructure for delivering commands and events reliably between elements of an application. The Windows Azure Service Bus also offers scalability in scenarios that must support high volumes of messages.

## Queues

Windows Azure Service Bus queues provide a durable mechanism for senders to send one-way messages for delivery to a single consumer.

Figure 1 shows how a queue delivers messages.

![Figure 1][fig1]

**Windows Azure Service Bus Queue**

The following list describes some of the key characteristics of queues.

* Queues deliver messages on a First In, First Out (FIFO) basis.
* Multiple senders can send messages on the same queue.
* A queue can have multiple consumers, but an individual message is only consumed by one consumer. Multiple consumers compete for messages on the queue.
* Queues offer "temporal decoupling." Senders and consumer do not need to be connected at the same time.

## Topics and Subscriptions

Windows Azure Service Bus topics provide a durable mechanism for senders to send one-way messages for delivery to a multiple consumers.

Figure 2 shows how a topic distributes messages.

![Figure 2][fig2]

**Windows Azure Service Bus Topic**

The following list describes some of the key characteristics of topics.

* Topics deliver a copy of each message to each subscription.
* Multiple senders can publish messages to the same topic.
* Each subscription can have multiple consumers, but an individual message in a subscription is only consumed by one consumer. Multiple consumers compete for messages on the subscription.
* Topics offer "temporal decoupling." Senders and consumer do not need to be connected at the same time.
* Individual subscriptions support filters that limit the messages available through that subscription.

## Useful API Features

The following sections highlight some of the Windows Azure Service Bus API features that are used in the project.

### Reading Messages

A consumer can use one of two modes to retrieve messages from queues or subscriptions: **ReceiveAndDelete** mode and **PeekLock** mode.

In the **ReceiveAndDelete** mode, a consumer retrieves a message in a single operation: the Service Bus delivers the message to the consumer and marks the message as deleted. This is the simplest mode to use, but there is a risk that a message could be lost if the consumer fails between retrieving the message and processing it.

In the **PeekLock** mode, a consumer retrieves a message in two steps: first, the consumer requests the message, the Service Bus delivers the message to the consumer and marks the message on the queue or subscription as locked; then, when the consumer has finished processing the message, it informs the Service Bus so that it can mark the message as deleted. In this scenario, if the consumer fails between retrieving the message and completing its processing, the message is re-delivered when the consumer restarts. A timeout ensures that locked messages become available again if the consumer does not complete the second step.

In the **PeekLock** mode, it is possible that a message could be delivered twice in the event of a failure. This is known as *at least once* delivery. You must ensure that either the messages are idempotent, or add logic to the consumer to detect duplicate messages and ensure *exactly once* processing. Every message has a unique, unchanging Id which facilitates checking for duplicates.

### Expiring Messages

When you create a **BrokeredMessage** object, you can specify an expiry time using the **ExpiresAtUtc** property or a time to live using the **TimeToLive** property. When a message expires you can specify either to send the message to a dead letter queue or discard it.

### Delayed Message Processing

In some scenarios, you may want to send the message now, but to delay delivery until some future time. You can do this by using the **ScheduleEnqueueTimeUtc** property of the **BrokeredMessage** instance.

## Further Information

For general information about the Windows Azure Service Bus, see
[Service Bus][sb] on MSDN.

For more information about Service Bus topologies and patterns, see
[Overview of Service Bus Messaging Patterns][sbpatterns] on MSDN.

For information about scaling the Windows Azure Service Bus
infrastructure, see [Best Practices for Performance Improvements Using
Service Bus Brokered Messaging][sbperf] on MSDN.

[fig1]:           images/Reference_09_ServiceBusQueue.png?raw=true
[fig2]:           images/Reference_09_ServiceBusTopic.png?raw=true

[sb]:             http://msdn.microsoft.com/en-us/library/ee732537.aspx
[sbperf]:         http://msdn.microsoft.com/en-us/library/hh528527.aspx
[sbpatterns]:     http://msdn.microsoft.com/en-us/library/hh410103.aspx

repository: raelyard/cqrs-journey-doc

extension: markdown

repo_name: raelyard/cqrs-journey-doc


text:
# channels

Web application framework to support discrete channels/streams of content/chat items as part of (or instead of) web pages.

[Specification](SPEC.md)

repository: channels-org/channels-spec

repo_name: channels-org/channels-spec


text:
# Conocybe watlingii Hauskn., 1996 SPECIES

#### Status
ACCEPTED

#### According to
The Catalogue of Life, 3rd January 2011

#### Published in
Öst. Z. Pilzk. 5: 193 (1996)

#### Original name
Conocybe watlingii Hauskn., 1996

### Remarks
null

repository: mdoering/backbone

repo_name: mdoering/backbone


text:
# Teknik Web Services

Teknik is a suite of services with attractive and functional interfaces.

## Features
  * File Upload w/ encryption
  * Album Support
  * Pastebin
  * URL Shortening
  * Blogs
  * Git Repositories
  * Podcasts
  * Easy to use API
  * Flexible installation and configuration
  * And much more...

You can see a live demo [here](https://www.teknik.io).

## Requirements
In order to run Teknik on your server, you'll need:

  * IIS 7+ with URL Rewrite module or Apache with `mod_rewrite` enabled (Requires conversion of `web.config` files)
  * ASP.NET MVC 5
  * .NET Framework 4.6.2
  * MS SQL Server
  * hMailServer (If running email as well)
  * Web Mail Client (If you would like to have webmail)
  * Gogs Service (If you want to have Git integration)

## Installation
  * Clone the Teknik repository to your web root directory, or anywhere else you want to run Teknik from

```nohighlight
cd /var/www
git clone https://git.teknik.io/Teknikode/Teknik
```

  * Open the `Teknik.sln` file to build the project
  * Copy the build files to your `wwwroot` directory
  * Create a `ConnectionStrings.config` file in the `App_Data` directory and fill it with the following template and put in your SQL server connection details

  ```nohighlight
  <connectionStrings>
    <add name="TeknikEntities"
      providerName="System.Data.SqlClient"
      connectionString="Data Source=<server name>,<port>\<sql server name>;Initial Catalog=<database>;Integrated Security=False;User Id=<username>;Password=<password>" />
  </connectionStrings>
  ```

  * After the first run, a `Config.json` file will be created in the `App_Data` directory. This will need to be edited with your configuration options.

That's it, installation complete! If you're having problems, let us know through the [Contact](https://contact.teknik.io/) page.

## Authors and contributors
  * [Chris Woodward](https://www.teknik.io) (Creator, Developer)
  * [dronedaddy](https://www.behance.net/dronedaddy) (Logo Designer)

## License
[BSD 3-Clause license](http://opensource.org/licenses/BSD-3-Clause)

## Development
You can view Teknik's [Development Branch](https://dev.teknik.io/) to see the current new features (It may not work, as it is a development branch).

## Contributing
If you are a developer, we need your help. Teknik is a young project and we have lots of stuff to do. Some developers are contributing with new features, others with bug fixes. Any help you can give would be greatly appreciated!

## Further information
If you want to know more about the features of Teknik, check the [Help](https://help.teknik.io/) page. Also, if you're having problems with Teknik, let us know through the [Contact](https://contact.teknik.io/) page. Don't forget to give feedback and suggest new features! :)

repository: PradeepTheDev/Teknik

repo_name: PradeepTheDev/Teknik


text:
# Hander
Simple and open sourced hand-to utility, for your roleplay game's needs.

## Getting Started
To build the place from scratch, use:

```bash
rojo build -o "Hander.rbxlx"
```

Next, open `Hander.rbxlx` in Roblox Studio and start the Rojo server:

```bash
rojo serve
```

## Setting up feedback
By default, Hander do not send any feedback back to the user, and the person who received the tool. To implement feedbacks, you can connect the RemoteEvent Hander uses, which is `Hander_Event`. After that, simply check whether the first parameter is `receive`, if so, the second parameter will be: `{client's name, tool's name}`.

To implement feedback for the original sender, simply just use the boolean returned by the RemoteFunction call inside `Shared.Cook.init.lua`, where true means successful, false means fail.

## Credits
Hander uses the following libraries/softwares:

- [EtiLibs/Logger](https://www.roblox.com/library/6600363372/EtiLibs)
- [EtiLibs/Lang](https://www.roblox.com/library/6600363372/EtiLibs)
- [Promise](https://github.com/evaera/roblox-lua-promise)
- [spr](https://github.com/Fraktality/spr/)
- [Matcher](https://github.com/rgieseke/textredux/blob/main/util/matcher.lua)
- [Rojo](https://rojo.space/)

## License
Hander is licensed under the MIT license, read it [here](./LICENSE)


text:
### [CVE-2015-6839](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-6839)
![](https://img.shields.io/static/v1?label=Product&message=n%2Fa&color=blue)
![](https://img.shields.io/static/v1?label=Version&message=n%2Fa&color=blue)
![](https://img.shields.io/static/v1?label=Vulnerability&message=n%2Fa&color=brighgreen)

### Description

The parse function in MSA vot.Ar 3.1 does not check whether a candidate receives more than one vote, which allows physically proximate attackers to cast multiple votes for a candidate via a crafted RFID ballot tag.

### POC

#### Reference
- https://www.youtube.com/watch?v=CTOCspLn6Zk

#### Github
No PoCs found on GitHub currently.

repository: trickest/cve

repo_name: trickest/cve


text:
---
title: oVirt 3.6 Release Management
category: release
authors: bproffitt, danken, sandrobonazzola
---

# oVirt 3.6 Release Management

## Key Milestones
| Date | Milestone |
|---|---|
| **2014-10-17**    | oVirt 3.5.0 GA Releases                                                             |
| **2014-10-22**    | Release criteria discussion start                                                   |
| <s>2014-11-12</s>
<s>2014-11-19</s>
**2014-11-26** | Release criteria ready |
| **2015-04-22**    | Feature Review - Feature Submission Closed                                          |
| <s>2015-05-06</s>
 <s>2015-05-12</s>
 **2015-05-19**     | Alpha Release: [oVirt 3.6 Release Notes](/develop/release-management/releases/3.6/)        |
| <s>2015-05-12</s>
 <s>2015-05-19</s>
 <s>2015-05-26</s>
 <s>2015-05-27</s>  | <s>Alpha Release Test Day: [oVirt 3.6 Test Day](oVirt 3.6 Test Day)</s>  |
| **2015-06-15**    | Feature freeze                                                                      |
| <s>2015-06-17</s>
 <s>2015-06-24</s>
 <s>2015-06-25</s>
 <s>2015-06-26</s>
 **2015-06-29**     | Second Alpha Release: [oVirt 3.6 Release Notes](/develop/release-management/releases/3.6/) |
| **2015-07-28**    | Third Alpha Release: [oVirt 3.6 Release Notes](/develop/release-management/releases/3.6/)  |
| <s>2015-07-15</s>
 **2015-08-03**     | Beta Release                                                                        |
| <s>2015-07-20</s>
 **2015-08-10**     | Beta Release Test Day                                                               |
| <s>2015-08-11</s>
 **2015-08-14**     | Second Beta Release                                                                 |
| <s>2015-08-03</s>
 **2015-08-19**     | String Freeze                                                                       |
| **2015-08-20**    | Third Beta Release                                                                  |
| <s>2015-09-02</s>
 **2015-09-07**     | Fourth Beta Release                                                                 |
| **2015-09-09**    | Fifth Beta Release                                                                  |
| **2015-09-16**    | Sixth Beta Release                                                                  |
| <s>2015-08-19</s>
 <s>2015-09-02</s>
 <s>2015-09-16</s>
 <s>2015-09-22</s>
 **2015-09-28**     | Release Candidate                                                                   |
| <s>2015-08-24</s>
 <s>2015-09-08</s>
 <s>2015-09-17</s>
 <s>2015-09-24</s>  | <s>Release Candidate Test Day</s>                                                   |
| **2015-10-14**    | Second Release Candidate                                                            |
| **2015-10-21**    | Third Release Candidate                                                             |
| <s>2015-09-08</s>
 <s>2015-09-22</s>
 <s>2015-09-29</s>
 <s>2015-10-12</s>
 **2015-11-04**     | Release                                                                             |

**NOTE** this is a tentative planning according to [Release process](Release process)

## External Project Schedules

Links to other significant project schedules, useful for seeing how oVirt aligns with them.

*   Debian Jessie 8.2: [2015-09-05](https://lists.debian.org/debian-announce/2015/msg00003.html)
*   Fedora 24: [2016-05-17](https://fedoraproject.org/wiki/Releases/23/Schedule)
*   Fedora 23: [2015-10-27](https://fedoraproject.org/wiki/Releases/23/Schedule)
*   Fedora 22 End Of Life: [2016-06-17 (1 month after Fedora 24 release)](https://fedoraproject.org/wiki/Releases/24/Schedule)
*   Fedora 21 End Of Life: [2015-11-27 (1 month after Fedora 23 release)](https://fedoraproject.org/wiki/Releases/23/Schedule)
*   Foreman 1.8.4: [2015-09-15](http://projects.theforeman.org/rb/releases/foreman)
*   Foreman 1.9.2: [2015-10-13](http://projects.theforeman.org/rb/releases/foreman)
*   Foreman 1.10.0: [2015-11-01](http://projects.theforeman.org/rb/releases/foreman)
*   Foreman 1.11.0: [2016-02-01](http://projects.theforeman.org/rb/releases/foreman)
*   GlusterFS 3.7: [2015-05-06](http://www.gluster.org/community/documentation/index.php/Planning37)
*   GlusterFS 4.0: [No date yet](http://www.gluster.org/community/documentation/index.php/Planning40)
*   OpenStack Kilo: [2015-04-30](https://wiki.openstack.org/wiki/Kilo_Release_Schedule)
*   OpenStack Liberty: [2015-10-15](https://wiki.openstack.org/wiki/Liberty_Release_Schedule)
*   QEMU 2.5.0: [2015-12-10](http://wiki.qemu.org/Planning/2.5)
*   Tiny Core 6.4: [2015-09-08](http://forum.tinycorelinux.net/index.php/topic,18818.0.html)

## Nightly Builds

Nightly builds are available from oVirt snapshots repositories:

[`http://resources.ovirt.org/pub/ovirt-master-snapshot-static/`](http://resources.ovirt.org/pub/ovirt-master-snapshot-static/)
[`http://resources.ovirt.org/pub/ovirt-master-snapshot/`](http://resources.ovirt.org/pub/ovirt-master-snapshot/)

Please refer to [Install nightly snapshot](/develop/dev-process/install-nightly-snapshot/) guide for enabling those repositories

## oVirt Live

oVirt Live has been rebased on EL7.

Please help us testing it! Latest nightly iso is available here: <http://jenkins.ovirt.org/job/ovirt_live_create_iso/>

## Features Status Table

To try and improve 3.6 planning over the wiki approach in 3.3, this google doc <http://goo.gl/9X3G49> has been created.

## Translation Status

Translations are handled by Zanata. You can join the translators team and see current translation status here:
<https://translate.zanata.org/zanata/iteration/view/ovirt/master/languages>

## Key Proposed Changes

The following list is a subset of the [features proposed for oVirt 3.6](/develop/release-management/releases/3.6/proposed-feature/)

*   [Features/Management Network As A Role](/develop/release-management/features/network/management-network-as-a-role/)
*   [Features/DetailedHostNetworkingApi](/develop/release-management/features/network/detailedhostnetworkingapi/)
*   [Features/HostNetworkingApi](/develop/release-management/features/network/hostnetworkingapi/)
*   [Features/IsolatedNetworks](/develop/release-management/features/network/isolatednetworks/)
*   [Features/Cluster parameters override](/develop/release-management/features/engine/cluster-parameters-override/)
*   Drop support for Fedora <= 20
*   [Add support for Fedora 22](/develop/release-management/features/os-support/fedora-22-support/)
*   [Add support for Ubuntu hosts](/develop/release-management/features/debian-support-for-hosts/)
*   No support for new features on el6. el6 hosts would be allowed only in [3.5 compatibility mode](http://lists.ovirt.org/pipermail/users/2014-September/027421.html).
*   Hosted Engine support only on hosts supporting 3.6 compatibility level (EL7 and Fedora). A guide will be provided for migrating from EL6

## Tracker Bug

*   - Tracker: oVirt 3.6 release

## Release Criteria

### Alpha Release Criteria

1.  MUST: All sources must be available on ovirt.org
2.  MUST: All packages listed by subprojects must be available in the repository

### Beta Release Criteria

1.  MUST: Release Notes have feature-specific information: [oVirt 3.6 Release Notes](/develop/release-management/releases/3.6/)
2.  MUST: Alpha Release Criteria are met
3.  MUST: Supported localizations must be at least at 70% of completeness for being included in the release
4.  MUST: All accepted features must be substantially complete and in a testable state and enabled by default -- if so specified by the change

### Candidate Release Criteria

1.  MUST: Beta Release Criteria are met
2.  MUST: All test cases defined for each accepted feature must be verified and pass the test for the build to be released
3.  MUST: All lower-level components must be available on supported distributions at required version [1]
4.  MUST: No known regressions from previous releases should be present
5.  MUST: Each subproject must allow upgrade from a previous version, either providing documented manuals or automated upgrade procedures
6.  MUST: All features working on previous release must still work in the new release if not dropped as deprecated

[1] This means we're not hosting any package we don't develop inside oVirt repository just because it's not yet released officially.

repository: msbai/ovirt-site

repo_name: msbai/ovirt-site


text:
---
layout: docs
title: Securing Quasar
permalink: /docs/securingquasar/
docs_navbar: true
---

There are several methods of restricting access to Quasar.  This Quick Guide
focuses on configuring Quasar to run on a specific host with Nginx providing
basic http authorization on a separate host.

This quick guide does not give detailed instructions on how to setup
firewall rules or configure applications other than Quasar and Nginx.

This guide covers:

* Assumptions
* Architecture Overview
* Installing Quasar Manually
* Testing Quasar
* Installing Nginx
* Configuring Nginx
{:toc}

## Assumptions

For the example in this quick guide, we'll continue with the following
assumptions:

* <IP_ADDRESS>: IP address of MongoDB host, already running on port 27017
* <IP_ADDRESS>: IP address of Quasar host
* <IP_ADDRESS>: IP address of Nginx host
* If network security policy dictates, then network communication between hosts is
  restricted via ipchains or other firewall mechanism.  This allows
  simpler Quasar and Nginx configuration files.
* The reader has at least a basic understanding of JSON formatting and the Linux OS
* Running on Ubuntu 14.04 or similar.  If using RedHat, use 'yum' rather than 'apt-get' for software management.
* You have version 2.2.1 or newer of the Quasar source code.

If your IP addresses differ, change as appropriate.

## Architecture Overview

As can be noted from the above diagram, the network communications path is straight forward:

1. A request comes into the Nginx proxy.  Note that the proxy in this configuration
   has no special restrictions about who can access it.
2. Nginx first authenticates the user via standard http auth.  More about this in the
   authorization section below.
3. After appropriate authentication, Nginx acts as a proxy for the request and will send the
   request to Quasar on behalf of the client.
4. Quasar runs the appropriate tasks on the data source, MongoDB in this case, and
   returns the results back upstream, through Nginx to the client.

## Installing Quasar Manually

The first step is to download Quasar and build the source:

### Download and Build Quasar

git clone https://github.com/quasar-analytics/quasar
cd quasar
./sbt test
./sbt 'project web' oneJar
./sbt 'project core' oneJar

These commands will download the latest source code, run the test suite,
and build both the Web and Core projects jar files.

### Configure Quasar

Configure the Quasar server to connect to MongoDB.  Assuming there is a MongoDB
databased called 'testdb' running on <IP_ADDRESS>, your configuration file may
be called quasar-config.json and look like this:

    {
      "mountings": {
          "/local": {
              "mongodb": {
                  "connectionUri": "mongodb://<IP_ADDRESS>/testdb"
              }
          }
      },
      "server": { "port": 8080 }
    }

For further details on the format of the configuration file, please see the
Quasar [Configuration](/docs/quasarconfiguration) documentation.

## Testing Quasar

### Start the Core jar file

From the Quasar server, run the core jar file to verify you have connectivity and your mounting is correct:

java -jar \
~/quasar/core/target/scala-2.11/core_2.11-2.2.1-SNAPSHOT-one-jar.jar \
~/quasar-config.json

Once launched, a [REPL](https://en.wikipedia.org/wiki/Read–eval–print_loop) console will appear representing
a virtual file system where each MongoDB database is a directory, and each database directory contains one
or more MongoDB collections.

Notice how the the OS-like file system commands and SQL commands are executed directly after the $ prompt:

    💪 $ ls
    local@
    💪 $ cd local
    💪 $ ls
    local/
    quasar-test/
    testdb/
    💪 $ cd testdb
    💪 $ ls
    coll1
    💪 $ select * from coll1;
    Mongo
    db.coll1.find();

Results:

    Query time: 0.0s
     name    | age   | gender  | minor  |
    ---------|-------|---------|--------|
     Johnny  |  42.0 | male    |  false |
     Jenny   |  27.0 | female  |  false |
     Deb     |  33.0 | female  |  false |
     Billy   |  15.0 | male    |   true |

### Start the Web jar file

Once you have verified proper connectivity between Quasar and MongoDB, stop the Core jar file
and now start the Web jar file with a slightly different syntax to point to the configuration file:

java -jar \
~/quasar/web/target/scala-2.11/web_2.11-2.2.1-SNAPSHOT-one-jar.jar \
-c ~/quasar-config.json

Congratulations!  You now have two of the three necessary systems up and running for this configuration.

## Installing Nginx

### OS X
On OS X systems, consider using [HomeBrew](http://brew.sh/) to install Nginx:

brew install nginx

### Redhat / CentOS
On RedHat or CentOS systems:

sudo yum install nginx

### Ubuntu / Debian
On Ubuntu or Debian systems:

sudo apt-get install nginx

## Configuring Nginx

There are two main reasons we'll modify the Nginx configuration file in this guide:

1. To force user authentication, thus restricting access to known individuals
2. To redirect queries to the Quasar engine on another host, thus limiting the
   access path to the Quasar API to individuals authenticated with Nginx.

By combining firewall rules on the Quasar server that only allow HTTP
requests from the Nginx server, we essentially limit all communication
to the Quasar API from only a single system which also forces user
authentication.

This example will use the 'default' Nginx site configuration. Nginx has many
configuration files and is a versatile tool, please contact your Nginx
application administrator if you have questions, or visit the Nginx web site.

If Nginx is already running, stop it:

sudo service nginx stop

### Setting up Authentication

To allow http authentication, we'll need to create a file which stores the
names and passwords of allowed individuals.  To do this, use the apache2-utils
package which provides those tools:

sudo apt-get install apache2-utils

Now create the htpasswd file we'll use to store the encrypted data:

sudo htpasswd -c /etc/nginx/.htpasswd exampleuser

Replace 'exampleuser' with a real username.  You'll then be prompted for
a password and verification password.

This creates the file /etc/nginx/.htpasswd which we will reference in the
Nginx configuration file below.

Assuming the Nginx site configuration file is located at
/etc/nginx/sites-available/default, paste the following code directly after the
'server {' declaration, somewhere around line 20:

    listen 80;
    server_name your_nginx_server_name;
    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    location / {
            auth_basic "Restricted";
            auth_basic_user_file /etc/nginx/.htpasswd;
            proxy_set_header X-Forwarder-For $proxy_add_x_forwarded_for;
            proxy_set_header Host $http_host;
            proxy_redirect off;
            proxy_pass http://<IP_ADDRESS>:8080;
    }

The configuration above ensures the following important actions:

1. Nginx runs on port 80
2. Enables http authentication (lines beginning with 'auth_basic')
3. Nginx acts as a reverse proxy to the Quasar service, requesting data for the client

Save the configuration and start or restart Nginx:

sudo service nginx restart

At this point the Quasar application should be shielded from all requests
except those coming directly from the Nginx host.  Additionally all requests
coming from the Nginx host should only originate from authenticated users via
http authentication.

Test authentication with a browser.  Use the URL based on the quasar-config.json
file above:

    http://<IP_ADDRESS>/fs/data/local/testdb/coll1

An authentication prompt should appear.  Enter the username and password you
specified with the htpasswd command above.  If successful, Quasar should then
you the entire 'coll1' collection in JSON format

Note that you are sending the request to Nginx (IP .210), which authenticates
a username, sends the request to Quasar, then returns the results directly to
the browser.  The browser itself is not redirected as that would defeat the purpose
of securing with Nginx.


text:
# MetaCall Mailchimp DynamoDB

Hello There! Welcome to another video-example of MetaCall.

My name is Jose and Today we are going to use MetaCall to add a newsletter to a static website, then create a webhook to replicate the user's data into DynamoDB.

So, Let's jump right into it...

[![How to create a Newsletter self-contained solution with MetaCall & DynamoDB](https://img.youtube.com/vi/yo3eJmje_A8/0.jpg)](https://www.youtube.com/watch?v=yo3eJmje_A8)


text:
---
title:  "Scraping Song Lyrics for Password Attacks"
date:   2017-04-26
layout: post
excerpt: "Generate targeted password wordlists based on song lyrics for specific artists."
---

*Update (2019): I've since released a much more powerful tool for cracking passphrases. Check it out [on my GitHub page here](https://github.com/initstring/passphrase-wordlist). I'll keep this old blog up as the output from lyricpass can be fed through the cleanup.py script and then used with the hashcat rules in the new project. Enjoy!*

There’s been a lot of news in the media lately about using tools like encryption and password managers. Both of these can leverage a single password to unlock a ton of vital information. Because of this, people are looking to create longer, more complex “master keys”. This blog demonstrates a method of guessing some of those keys.

These “master keys” are for things like:

- Unlocking their password manager
- Decrypting their OS when booting
- Decrypting volumes with sensitive files
- WiFi encryption using WPA
- etc.

I’ve noticed a trend when people are discussing new strategies here – using a string of words together, including spaces, that is easy to remember. Song lyrics come readily to mind, and it seems that a good deal of people assume these will be difficult to crack.

## So What?

I wanted to create a short program to show that this type of password is also insecure. Using Python with a few simple libraries, I created [this script](http://github.com/initstring/lyricpass) that generates a password list based on a given artist. Discovering someone’s favorite band is pretty easy... that sort of thing is plastered all over social media, and it’s usually something people will provide when asked by anyone.

```
$ python ./lyricpass.py -h
usage: lyricpass.py [-h] [--lower] [--punctuation] artist output

positional arguments:
  artist         Define a specific artist for song lyric inclusion. Please
                 place the artist name in quotes.
  output         Output to file name in current directory.

optional arguments:
  -h, --help     show this help message and exit
  --lower        Switches all letters to lower case.
  --punctuation  Preserves punctuation, which is removed by default.
  ```

The script currently allows a few inputs to tailor the password file, which is pure text and can be used for brute force password attacks. It will chug away and output a nice deduplicted file when it is done.

![screenshot](/images/post-lyricpass/1.png)

Are you using any lyric-based passwords today? Try out [this script](http://github.com/initstring/lyricpass) and see if it can crack yours.

## Update
This script is now linked to on [WeakPass](https://weakpass.com/links)! :)