text:
# Contributing and Maintaining

First, thank you for taking the time to contribute!

The following is a set of guidelines for contributors as well as information and instructions around our maintenance process. The two are closely tied together in terms of how we all work together and set expectations, so while you may not need to know everything in here to submit an issue or pull request, it's best to keep them in the same document.

## Ways to contribute

Contributing isn't just writing code - it's anything that improves the project. All contributions for Insert Special Characters are managed right here on GitHub. Here are some ways you can help:

### Reporting bugs

If you're running into an issue with the plugin, please take a look through [existing issues](https://github.com/10up/insert-special-characters/issues) and [open a new one](https://github.com/10up/insert-special-characters/issues/new) if needed. If you're able, include steps to reproduce, environment information, and screenshots/screencasts as relevant.

### Suggesting enhancements

New features and enhancements are also managed via [issues](https://github.com/10up/insert-special-characters/issues).

### Pull requests

Pull requests represent a proposed solution to a specified problem. They should always reference an issue that describes the problem and contains discussion about the problem itself. Discussion on pull requests should be limited to the pull request itself, i.e. code review.

For more on how 10up writes and manages code, check out our [10up Engineering Best Practices](https://10up.github.io/Engineering-Best-Practices/).

## Workflow

The `develop` branch is the development branch which means it contains the next version to be released. `stable` contains the current latest release and `master` contains the corresponding stable development version. Always work on the `develop` branch and open up PRs against `develop`.

## Release instructions

1. Starting from `develop`, cut a release branch named `release/X.Y.Z` for your changes.
2. Version bump: Bump the version number in `insert-special-characters.php` and `package.json` if it does not already reflect the version being released.
3. Changelog: Add/update the changelog in `CHANGELOG.md` and `readme.txt`
4. Check to be sure any new files/paths that are unnecessary in the production version are included in `.distignore`.
5. Merge: Make a non-fast-forward merge from your release branch to `develop` (or merge the pull request), then do the same for `develop` into `master`. `master` contains the stable development version.
6. Push: Push your master branch to GitHub, e.g. `git push origin master`.
7. Git tag: Create a [new release](https://github.com/10up/insert-special-characters/releases/new) as `X.Y.Z` on the `master` branch in GitHub.
8. Deploy to WordPress.org: Head to the [Actions](https://github.com/10up/insert-special-characters/actions) tab in the repo and wait for the Deploy to WordPress.org workflow to finish if it hasn't already. If it doesn't succeed, figure out why and head back to delete the tag and try again.
9. Edit the [X.Y.Z milestone](https://github.com/10up/insert-special-characters/milestone/#) with release date (in the `Due date (optional)` field) and link to GitHub release (in the `Description` field), then close the `X.Y.Z` milestone.
10. If any open issues or PRs which were milestoned for `X.Y.Z` do not make it into the release, update their milestone to `X.Y.Z+1` or `Future Release`.


text:
## Configuration Transformations

All base configuration files are in the folder **Configurations**.

When building the solution a configuration file matching the current configuration (if it exists) is applied as [Xdt-transforms](https://msdn.microsoft.com/en-us/library/dd465326(v=vs.110).aspx) to the base configuration file.

By default .Debug-files are ignored to each developer can have their own configuration (e.g. database connection strings or Find-index).

The .Release-files are prefilled with Octopus Deploy variables used for substitution with Octopus.
See the section about [Octopus Deploy](octopus.html) for more information.

### Technical details and adding a new config-file
The configuration transform is applied because of the [Your solution name].Web.wpp.targets-file. This is automatically run when in a project of type Web Project, and tells MSBuild to run the TransformXml-task from the Microsoft.Web.Publishing dll.

The ItemGroup-element determines which configurations are eligible for transform. If you need additional configuration files transformed add this to the ItemGroup-element:

```xml
<ConfigName Include="NewConfig">
    <Ext>config</Ext>
</ConfigName>
```

Optionally, you can also modify the csproj-file to nest the .Release and .Debug-files under the base.

license: Unlicense

branch_name: refs/heads/master

extension: md

gha_license_id: Unlicense

license_type: permissive


text:
# poker
A simple pygame poker application

This stemmed from a project euler problem that required you to find the winner of 10,000 poker hands.  Now it is a full on 5 card draw poker application made using pygame.  Can be run with pygame installed on your machine and by running PokerView.py.

license: MIT

repository: drewtorg/poker

gha_language: Python

gha_license_id: MIT

repo_name: drewtorg/poker


text:
# Introduction

```{toctree}
:maxdepth: 2
:hidden:

self
changelog
faq
```

```{toctree}
:caption: Basics
:maxdepth: 2
:hidden:

installation
components
templates
actions
child-components
django-models
```

```{toctree}
:caption: Features
:maxdepth: 2
:hidden:

validation
redirecting
loading-states
dirty-states
partial-updates
polling
visibility
messages
advanced
queue-requests
```

```{toctree}
:caption: Misc
:maxdepth: 2
:hidden:

troubleshooting
settings
cli
architecture
code-of-conduct
PDF <https://www.django-unicorn.com/docs/unicorn-latest.pdf>
GitHub <https://github.com/adamghill/django-unicorn>
Sponsor <https://github.com/sponsors/adamghill>
```

`Unicorn` is a reactive component framework that progressively enhances a normal Django view, makes AJAX calls in the background, and dynamically updates the DOM. It seamlessly extends Django past its server-side framework roots without giving up all of its niceties or re-building your website.

## Related projects

`Unicorn` stands on the shoulders of giants, in particular [morphdom](https://github.com/patrick-steele-idem/morphdom) which is integral for merging DOM changes.

### Inspirational projects in other languages

- [Livewire](https://laravel-livewire.com/), a full-stack framework for the PHP web framework, Laravel.
- [LiveView](https://github.com/phoenixframework/phoenix_live_view), a library for the Elixir web framework, Phoenix, that uses websockets.
- [StimulusReflex](https://docs.stimulusreflex.com), a library for the Ruby web framework, Ruby on Rails, that uses websockets.
- [Hotwire](https://hotwire.dev), "is an alternative approach to building modern web applications without using much JavaScript by sending HTML instead of JSON over the wire". Uses AJAX, but can also use websockets.

### Full-stack framework Python packages

- [Reactor](https://github.com/edelvalle/reactor/), a port of Elixir's `LiveView` to Django. Especially interesting for more complicated use-cases like chat rooms, keeping multiple browsers in sync, etc. Uses Django channels and websockets to work its magic.
- [Flask-Meld](https://github.com/mikeabrahamsen/Flask-Meld), a port of `Unicorn` to Flask. Uses websockets.
- [Sockpuppet](https://sockpuppet.argpar.se/), a port of Ruby on Rail's `StimulusReflex`. Requires Django channels and websockets.
- [Django inertia.js adapter](https://github.com/zodman/inertia-django) allows Django to use inertia.js to build an SPA without building an API.
- [Hotwire for Django](https://github.com/hotwire-django) contains a few different repositiories to integrate [Hotwire](https://hotwire.dev) with Django.
- [Lona](https://lona-web.org/) is a web application framework, designed to write responsive web apps in full Python.

#### Comparison

| Repo                                                          | Django | Flask | AJAX | Websockets |                                                       Stars                                                        |
| :------------------------------------------------------------ | :----: | :---: | :--: | :--------: | :----------------------------------------------------------------------------------------------------------------: |
| [Unicorn](https://github.com/adamghill/django-unicorn)        |   ✔    |       |  ✔  |            |   ![GitHub Repo stars](https://img.shields.io/github/stars/adamghill/django-unicorn?label=%20&style=flat-square)   |
| [Reactor](https://github.com/edelvalle/reactor/)              |   ✔   |       |      |     ✔     |      ![GitHub Repo stars](https://img.shields.io/github/stars/edelvalle/reactor?label=%20&style=flat-square)       |
| [Sockpuppet](https://github.com/jonathan-s/django-sockpuppet) |   ✔   |       |      |     ✔     | ![GitHub Repo stars](https://img.shields.io/github/stars/jonathan-s/django-sockpuppet?label=%20&style=flat-square) |
| [Flask-Meld](https://github.com/mikeabrahamsen/Flask-Meld)    |        |  ✔   |      |     ✔     |  ![GitHub Repo stars](https://img.shields.io/github/stars/mikeabrahamsen/Flask-Meld?label=%20&style=flat-square)   |

### Django packages to integrate lightweight frontend frameworks

- [django-htmx](https://github.com/adamchainz/django-htmx) which has extensions for using Django with [htmx](https://htmx.org/).

### Django component packages

- [django-components](https://github.com/EmilStenstrom/django-components/), which lets you create "template components", that contains both the template, the Javascript and the CSS needed to generate the front end code you need for a modern app.
- [django-component](https://gitlab.com/Mojeer/django_components), which provides declarative and composable components for Django, inspired by JavaScript frameworks.
- [django-page-components](https://github.com/andreyfedoseev/django-page-components), a minimalistic framework for creating page components and using them in your Django views and templates.
- [slippers](https://mitchel.me/slippers/), helps build reusable components in Django without writing a single line of Python.


text:
# dbt + Databricks Demo!

This is a modified version of our public [tutorial](https://docs.getdbt.com/tutorial/setting-up)
intended for users of dbt on Databricks.

Any questions<EMAIL_ADDRESS>
## Sample data

Create Databricks tables `jaffle_shop.orders`, `jaffle_shop.customers`,
and `stripe.payments` from these CSV files, which are located in a public S3 bucket ([docs](https://docs.databricks.com/data/tables.html#create-a-table-using-the-ui)):

```
s3://dbt-tutorial-public/jaffle_shop_orders.csv
s3://dbt-tutorial-public/jaffle_shop_customers.csv
s3://dbt-tutorial-public/stripe_payments.csv
```

## Getting started

The instructions below assume you are running dbt on macOS. Linux and Windows
users should adjust the bash commands accordingly.

1. Clone this github repo
2. Install [dbt-spark](https://github.com/fishtown-analytics/dbt-spark): `pip install dbt-spark`
3. Copy the example profile to your `~/.dbt` folder (created when installing dbt):
```bash
$ cp ./sample.profiles.yml ~/.dbt/profiles.yml
```
4. Populate `~/.dbt/profiles.yml` with your Databricks host, API token, cluster ID, and schema name
```bash
open ~/.dbt
```
5. Verify that you can connect to Databricks
```
$ dbt debug
```
6. Verify that you can run dbt
```
$ dbt run
```

### Resources:
- Learn more about dbt [in the docs](https://docs.getdbt.com/docs/introduction)
- Check out [Discourse](https://discourse.getdbt.com/) for commonly asked questions and answers
- Join the [chat](http://slack.getdbt.com/) on Slack for live discussions and support
- Find [dbt events](https://events.getdbt.com) near you
- Check out [the blog](https://blog.getdbt.com/) for the latest news on dbt's development and best practices
- Watch our [Office Hours](https://www.youtube.com/watch?v=C9WgnKEnwmg) on dbt + Spark


text:
[![Contributors][contributors-shield]][contributors-url]
[![Forks][forks-shield]][forks-url]
[![Stargazers][stars-shield]][stars-url]
[![Issues][issues-shield]][issues-url]
[![MIT License][license-shield]][license-url]
[![LinkedIn][linkedin-shield]][linkedin-url]

HCPairing

    Bridge the gap between underserved communities and healthcare providers.

    Explore the services »

    View Demo
    ·
    Report Bug
    ·
    Request Feature

  Table of Contents

      About The Project

Built With

      Getting Started

Prerequisites

Installation

Usage

Roadmap

Contributing

License

Contact

Acknowledgements

## About The Project

It is very difficult for underserved communities to access the right healthcare provider with the specialties for their symptoms. So we try to design the journey to bridge the gap between them.

### Built With

The project is mainly built with the well-known frameworks and libraires like [React.js](https://reactjs.org/), [Redux](https://redux.js.org/), [Material-UI](https://material-ui.com/).

* Based on the requirements above, we decide to build a website with a human-friendly interface as the implementation. We simply divide the service into frontend and backend, the former includes SDK integration and UI presentation, and the latter contains the conversion system and storagement.
* The project is driven by the Scrumban cycle, we maintain a public bulletin board and a knowledge base using HackMD throughout the development, it allows us to converge ideas quickly and control the schedule and resources.
* We finally deploy the service on Microsoft Azure via CI CD pipeline, maintaining the equipment by ourselves is too much trouble for us, the free quota of public cloud is a great choice for the hackathon.

## Getting Started

Besides using the published version, you can compile your own and make pull requests if you think this project can be better!

### Prerequisites

|          | Version  |
| -------- | -------- |
|[Node.js](https://nodejs.org/en/) | v14.17.0 |
|[Npm](https://www.npmjs.com/) | v6.14.13 |
|[React](https://reactjs.org/) | v17.0.2 |
|[TypeScript](https://www.typescriptlang.org/) | v4.3.2 |
|[Redux](https://redux.js.org/) | v7.2.4 |
|[Material-UI](https://material-ui.com/) | v4.11.4 |

### Step one - Clone the repo
`git clone https://github.com/RainrainWu/hcpairing-frontend.git`
### Step two - Install dependencies
`npm install`
### Step three - Build
`npm run build-dev`

The code will be generated in `dist/` directory in development mode

or

`npm run build`

The code will be generated in `build/` disrectory in production mode

### Step four - Host with http-server
`npx http-server ./build -p [port]`

Be sure that you have already installed npx through `npm install -g npx`.

## Usage

Users can pick the tags that describe their symptoms well, our service will help them convert into corresponding healthcare specialties, and display the candidates of healthcare providers which meet their needs.

## Roadmap

See the [open issues](https://github.com/RainrainWu/hcpairing-frontend/issues) for a list of proposed features (and known issues).

## Contributing

Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are **greatly appreciated**. Please follow the instructions below to start contributing to our project:

1. Fork the Project
2. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)
3.  Push to the Branch (`git push origin feature/AmazingFeature`)
5. Open a Pull Request

## License

Distributed under the MIT License. See `LICENSE` for more information.

## Contact

* Chun-Hao Huang - [@f26401004](https://github.com/f26401004) -<EMAIL_ADDRESS>* Cheng-Feng Wu - [@RainrainWu](https://github.com/RainrainWu) -<EMAIL_ADDRESS>

## Acknowledgements
* [Best README Template](https://github.com/othneildrew/Best-README-Template)

[contributors-shield]: https://img.shields.io/github/contributors/RainrainWu/hcpairing-frontend.svg?style=for-the-badge
[contributors-url]: https://github.com/RainrainWu/hcpairing-frontend/graphs/contributors
[forks-shield]: https://img.shields.io/github/forks/RainrainWu/hcpairing-frontend.svg?style=for-the-badge
[forks-url]: https://github.com/RainrainWu/hcpairing-frontend/network/members
[stars-shield]: https://img.shields.io/github/stars/RainrainWu/hcpairing-frontend.svg?style=for-the-badge
[stars-url]: https://github.com/RainrainWu/hcpairing-frontend/stargazers
[issues-shield]: https://img.shields.io/github/issues/RainrainWu/hcpairing-frontend.svg?style=for-the-badge
[issues-url]: https://github.com/RainrainWu/hcpairing-frontend/issues
[license-shield]: https://img.shields.io/github/license/RainrainWu/hcpairing-frontend.svg?style=for-the-badge
[license-url]: https://github.com/RainrainWu/hcpairing-frontend/blob/master/LICENSE.txt
[linkedin-shield]: https://img.shields.io/badge/-LinkedIn-black.svg?style=for-the-badge&logo=linkedin&colorB=555
[linkedin-url]: https://linkedin.com/in/f26401004
[product-screenshot]: images/screenshot.png

branch_name: refs/heads/main


text:
# jQuery plugins to handle events that require timing

## Abstract

Once in a while you look for an event handler, and find, to your surprise that:

1. It does not exist
1.  Handling the event you're looking for requires conjecture that depends on a timer

This repo contains several such handlers, converted into jQuery plugins, so they can easily be dropped into web or mobile web projects, and used as seamlessly as possible.

## Demo

The `demo.html` page contains a script using all the plugins in the repo. Just clone the entire repo, and open the html in your browser. While style is minimal (and abyssmal), you can easily understand how to use each plugin.

## 1. scrollStopped

### Purpose

Finding out when the user stopps scrolling an element (so you can e.g. update your view to show data pertinent to the contents that just scrolled in).

### Usage

`$(element).scrollStopped(handler)` - the handler will be called on scroll stop.

- All handlers have the signature `function(e) {}`.
- Avoid using ES6 arrow functions, unless you don't need access to `$(this)` in the handler.

## 2. doubleClick

### Purpose

While jQuery does have a `.dblClick` (or `.on('dblClick')`) event, the problem starts when you want the same element to support both `click` and `dblClick`. This plugin attaches itself to the element, and supports both events.

### Usage

`$(element).doubleClick(singleClickHandler, doubleClickHandler, duration)`

- `singleClickHandler` will be called when the user clicks the element once in `duration` milliseconds
- `doubleClickHandler` will be called when the user clicks the element twice in `duration` milliseconds
- `duration` - milliseconds to wait before calliing the single click handler. Defaults to 500

## 3. longPress

### Purpose

Simulate a long press you get on a mobile device - while still maintaining a short press (click) behavior on the element.

### Usage

`$(element).longPress(shortPressHandler, longPressHandler, duration)`

- `shortPressHandler` will be called when the user clicks and exits the element within `duration` milliseconds
- `longPressHandler` will be called when the user clicks and stays on the element before  `duration` milliseconds passed
- `duration` - milliseconds to wait before calliing the short press handler. Defaults to 750

## Thanks and inspiration

1. scrollStopped - was inspired by this [StackOverflow answer](https://stackoverflow.com/questions/14035083/jquery-bind-event-on-scroll-stop)
1. doubleClick - was inspired by this [StackOverflow answer](https://stackoverflow.com/questions/5497073/how-to-differentiate-single-click-event-and-double-click-event)
1. longPress - my initial approach was to look at [Vaidik Kapoor's solution](https://github.com/vaidik/jquery-longpress), although my logic is quite different now

repository: TravelingTechGuy/my-plugins

repo_name: TravelingTechGuy/my-plugins


text:
# OASIS: A Large-Scale Dataset for Single Image 3D in the Wild

![OASIS](teaser.gif)

This repository contains the code for the following paper:

    OASIS: A Large-Scale Dataset for Single Image 3D in the Wild,
    Weifeng Chen, Shengyi Qian, David Fan, Noriyuki Kojima, Max Hamilton, Jia Deng
    Conference on Computer Vision and Pattern Recognition (CVPR), 2020.

Please check the [project site](https://pvl.cs.princeton.edu/OASIS) for more details.

## Installation

The code has been tested on python 3.7, cuda 10.0, pytorch 1.1.0, gcc 8.4.0

```bash
conda create --name oasis python=3.7
conda activate oasis

conda install pytorch==1.1.0 torchvision==0.3.0 cudatoolkit=10.0 -c pytorch
conda install opencv==3.4.2 h5py scipy pillow==6.1.0 scikit-learn
pip install sacred easydict pyyaml imageio==2.6.0 tb-nightly future tqdm
```

## Data Preparation

Please go to the [download page](https://oasis.cs.princeton.edu/download) and download all the images and annotations. Then untar:

```bash
mkdir OASIS
tar -xzf OASIS_images_v1.tar.gz  -C OASIS
tar -xzf OASIS_trainval_annotations_v1.tar.gz -C OASIS
```

The folder tree after these steps should look like:

```bash
OASIS
    - LICENSE
    - OASIS_trainval
        - image
        - meta
        - OASIS_train.csv
        - OASIS_val.csv
        - depth
        - normal
        - fold
        - occlusion
        - mask
        - DIW_style_rel_depth
        - segmentation
            - planar_instance
            - continuous_instance
    - OASIS_test
        - image
        - meta
        - OASIS_test.csv
```

## Experiments

The `experiment` folder contains code to reproduce the results for the following experiments:

* Depth Estimation
* Surface Normal Estimation
* Fold and Occlusion Boundary Detection
* Planar Instance Segmentation

Please refer to the README files under each folder for instructions on how to run the code.

To run on pretrained models, please first download the pretrained models [experiments.tar.gz](https://drive.google.com/file/d/1XE--nVIUEROud5YwNRUuvqJH3I_cR9kI/view?usp=sharing), and `tar -xzf experiments.tar.gz`.


text:
[![Build Status](https://travis-ci.org/Antash/exercisync.svg?branch=master)](https://travis-ci.org/Antash/exercisync)

## Credits
This project is based on https://github.com/cpfair/tapiriik code by Collin Fair.

## Licensing
exercisync is an Apache 2.0 Licensed open-source project.

## Translation
To use locale translation, it requires the package gettext.
To launch a new language, you have to generate the locale with : python3 manage.py makemessages -l 'fr'
Secondly, after translation (edition of po files), compile the files : python manage.py compilemessages


text:
# Restocket
This library allows you to build API's like in Express, but that also expose a websocket and http(s) endpoint.

## Example Usage
### Server Side
```javascript
import { RestocketServer, RestocketRouter } from 'restocket';
const server = new RestocketServer()

// Receive an event when a client connects via websocket
server.onSocketConnected(function (req, res) {
  console.log('Websocket is connected...')
})

// Add a new route
server.get('/hello/:name', function (req, res) {
  res.send({ message: `Hello ${req.params.name}` })
})

// You can also use other Restocket instances
const routes = new RestocketRouter()
routes.get('/goodbye/:name', function (req, res) {
  res.send({ message: `Goodbye ${req.params.name}` })
})

server.use(routes)

// Start the websocket and http instance
await server.start({
  port: 3000
})

console.log('Listening')
```

### Client Side
```javascript
import { RestocketClient } from 'restocket';

const api = new RestocketClient({
  host: '<IP_ADDRESS>:3000'
})

const result = await api.get('/hello/tester')
console.log(result)
```

## How it works
In the above example the endpoints `/hello/:name` and `/goodbye/:name` have been created on the websocket and http servers.

You can make a HTTP request to http://<IP_ADDRESS>/hello/tester as you would normally in express.

You can make a websocket request to ws://<IP_ADDRESS> with the following data:
```json
// string Method, string Path, optional object Headers, optional any Body
["GET", "/hello/tester"]
```

### Correlation ID
Unlike http requests, websocket queries do not respond one for one. Or at least there is no built in way to related a request with a response.

If you pass a `_cid` header with your request, the response from the server will also contain the same correlation id in the header.

For example:

1. Send the following to the websocket
```javascript
["GET", "/hello/tester", { "_cid": 1 }]
```

2. Receive the following from the server
```javascript
["RESP", { "_cid": 1 }, { "message": "Hello tester" }]
```

#### Using RestocketClient
The above is a low level description of using the websocket api directly. However if you use the RestocketClient this is hidden from you in the form of promises.

repository: markwylde/restocket

gha_language: JavaScript

repo_name: markwylde/restocket


text:
---
# Display name
title: Sean Choi

# Is this the primary user of the site?
superuser: true

# Role/position
role: Research Scientist @ Facebook, Ph.D. in Electrical Engineering, M.S. in Computer Science

# Organizations/Affiliations
organizations:
- name: Stanford University
  url: ""

# Short bio (displayed in user profile at end of posts)
bio: My research interests include distributed robotics, mobile computing and programmable matter.

interests:
- Data Center Network
- Cloud Infrastructure
- Cloud Services
- Serverless Compute
- SmartNICs
- Programmable Network

education:
  courses:
  - course: Ph.D. in Electrical Engineering
    institution: Stanford University
    year: 2019
  - course: M.S. in Computer Science
    institution: Stanford University
    year: 2013
  - course: Dual B.S. in Computer Science & Business Economics and Management
    institution: California Institute of Technology
    year: 2011

# Social/Academic Networking
# For available icons, see: https://sourcethemes.com/academic/docs/page-builder/#icons
#   For an email link, use "fas" icon pack, "envelope" icon, and a link in the
#   form<EMAIL_ADDRESS>or "#contact" for contact widget.
social:
- icon: envelope
  icon_pack: fas
  link<EMAIL_ADDRESS># For a direct email link, use<EMAIL_ADDRESS>- icon: linkedin
  icon_pack: fab
  link: https://www.linkedin.com/in/seanschoi/
- icon: medium
  icon_pack: fab
  link: https://medium.com/@seanschoi
- icon: google-scholar
  icon_pack: ai
  link: https://scholar.google.com/citations?user=_N-bcZgAAAAJ
- icon: github
  icon_pack: fab
  link: https://github.com/yo2seol/
# Link to a PDF of your resume/CV from the About widget.
# To enable, copy your resume/CV to `static/files/cv.pdf` and uncomment the lines below.
- icon: cv
  icon_pack: ai
  link: files/cv.pdf

# Enter email to display Gravatar (if Gravatar enabled in Config)
email: ""

# Organizational groups that you belong to (for People widget)
#   Set this to `[]` or comment out if you are not using People widget.
user_groups:
- Researchers
- Visitors
---
Sean Choi is current a Research Scientist at Facebook, working on scaling and improving Facebook's data center networks.

Previously, Sean completed his Ph.D. in electrical engineering at Stanford University. He was advised by Professor Mendel Rosenblum, one of the founders of VMWare and the recipient of ACM Thacker breakthrough in computing award. Sean completed a M.S. degree in computer science at Stanford University and B.S. degree in computer science and business economics and management at California Institute of Technology. During the intrim between the end of his M.S. and the start of his Ph.D., Sean worked at LinkedIn as a data engineer and worked at VMware as a senior member of technical staff. He also founded multiple companies in the area of machine learning and image recognition.

His research focus is on adding programmability to network switches and network interface cards (NIC) and finding new use cases for the newly available programmable hardware in the areas of data analytics and machine learning.

repository: seanschoi/academic-kickstart

repo_name: seanschoi/academic-kickstart


text:
# gauss-binomial-mp99
Python package for calculating and visualizing Gaussian and Binomial distributions.
The package contains methods for performing computations pertaining to Guassian and Binomial distributions.

Link to the PyPI project:
[https://pypi.org/project/gauss-binomial-mp99/](https://pypi.org/project/gauss-binomial-mp99/)

## Installation

```
$ pip install gauss-binomial-mp99
```

# Files

## Generaldistribution.py
The parent class, contains `__init__` and `read_data_file` methods.

Import:
```
>>> from gauss_binomial_mp99 import Gaussian, Binomial
>>> gauss = Gaussian()
>>> gauss.read_data_file('sample.txt')
```

## Gaussiandistribution.py
Gaussian distribution class for calculating and visualizing a Gaussian distribution. Methods to compute Gaussian distribution features include-

* `calculate_mean()`: Function to calculate the mean of the data set.
* `calculate_stdev()`: Function to calculate the standard deviation of the data set.
* `plot_histogram()`: Function to output a histogram of the instance variable data using matplotlib pyplot library.
* `pdf(x)`: Probability density function calculator for the gaussian distribution.
* `plot_histogram_pdf`: Function to plot the normalized histogram of the data and a plot of the probability density function along the same range

**Default**: *mu*= 0 and *sigma*=1
*Form: guass(mu, sigma)*

Example:
```
>>> gauss.calculate_mean()
78.0909090909091
>>> gauss.calculate_stdev()
92.87459776004906
```

Provides the functionality of adding two Gaussian distribution objects.

## Binomialdistribution.py
Binomial distribution class for calculating and visualizing a Binomial distribution. Methods to compute Gaussian distribution features include-

* `calculate_mean()`: Function to calculate the mean from p= probability and n= size
* `calculate_stdev()`: Function to calculate the standard deviation from p and n.
* `plot_bar()`: Function to output a histogram of the instance variable data using matplotlib pyplot library.
* `pdf(x)`: Probability density function calculator for the gaussian distribution.
* `plot_bar_pdf`: Function to plot the pdf of the binomial distribution.

**Default**: *p*= 0.5 and *n*=20
*Form: bin(p, n)*

```
>>> bin= Binomial(.45, 78)
>>> bin.pdf(34)
0.08798942891783665
```

Provides the functionality of adding two Binomial distribution objects with the same probablity.


text:
# update
The `update` subcommand is a simple alias for:
```sh
nix flake update
```
As it sounds, this will update your lock file.

## Updating Package Sources
If you pass directory name then it will update that input if the directory
contains a flake.nix, with an optional arguement to update only a specific
input in the subflake.

For example, you can update any
[package sources](../outputs/pkgs.md#automatic-source-updates) you may have
declared in _pkgs/flake.nix_:
```sh
flk update pkgs
```
or just its _nixpkgs_:
```sh
flk update pkgs nixpkgs
```

repository: repos-nixos/config

repo_name: repos-nixos/config


text:
---
layout: default
title: delete(url,body,output)
parent: ws.async
tags: command ws async asynchronous api default
comments: true
---

### Description
> This command is similar to [its `ws` counterpart](../ws/delete(url,body,var)), except for the asynchronous HTTP
invocation and response handling.

This command sends a HTTP DELETE request to perform delete operation with the given `url` and `body` (request payload).
The response will be saved to `output`. To learn about how to manipulate the response returned from the target URL,
visit [HTTP Response](index#http-response).

### Parameters
- **url** - the target URL.
- **body** - the payload to be sent to target endpoint. If not required, use `(empty)` to denote no payload to be sent.
- **output** - the file to save the [request and response information](index.html#http-response)

### See Also
- [`get(url,queryString,output)`](get(url,queryString,output))
- [`head(url,output)`](head(url,output))
- [`patch(url,body,output)`](patch(url,body,output))
- [`post(url,body,output)`](post(url,body,output))
- [`put(url,body,output)`](put(url,body,output))

repository: nexiality/documentation

repo_name: nexiality/documentation


text:
# Contributing Templates

## Add a New Template

In the [`data/templates`](data/templates) directory, use the example to create your own template directory. The name of the directory will be the primary alias for that meme.

## Create a Pull Request

Create a pull request to be merged after review.

# Contributing Code

## Setup

### Requirements

* Python 3.6+
    * Windows: https://www.python.org/downloads
    * macOS: `$ brew install python3`
    * Linux: `python3.6` and `python3.6-dev` packages
* pipenv: http://docs.pipenv.org
* Make:
    * Windows: http://mingw.org/download/installer
    * Mac: http://developer.apple.com/xcode
    * Linux: http://www.gnu.org/software/make
* OpenJPEG:
    * Windows: ?
    * macOS: `$ brew install openjpeg`
    * Linux: ?
* libfreetype:
    * Windows: ?
    * macOS: `$ brew install freetype`
    * Linux: ?

### Installation

After cloning the repository, create a virtualenv:

```sh
$ make install
```

## Development

### Run the application

Run the server locally:

```sh
$ make run
```

or also launch it in your browser:

```sh
$ make launch
```

### Run the tests

Manually run the tests and checkers:

```sh
$ make ci
```

or keep them running on change:

```sh
$ make watch
```

> In order to have OS X notifications, `brew install terminal-notifier`.

### Create a Pull Request

After checks pass, create a pull request to be merged after review.

repository: flavienbwk/memegen

gha_license_id: NOASSERTION

repo_name: flavienbwk/memegen


text:
# idataapi-transform

**idataapi-transform** is a pure python, **full async** support **Toolkit**, to transform data from one location/format to another location/format, provide **command line interface** for easy use, and **python module** with powerful features

It's currently used in [IDataAPI](http://www.idataapi.cn/) team for efficiency work

* [中文文档](https://github.com/zpoint/idataapi-transform/blob/master/README_CN.md)

Provide

* [Command line interface](#command-line-interface-example)
* [Python API](#build-complex-routine-easily)

![diagram](https://github.com/zpoint/idataapi-transform/blob/master/idataapi-transform.png)

You can read data from one of

 * **API(http)**
 * **ES(ElasticSearch)**
 * **CSV**
 * **XLSX**
 * **JSON**
 * **Redis**
 * **MySQL**
 * **MongoDB**

and convert to

 * **CSV**
 * **XLSX**
 * **JSON**
 * **TXT**
 * **Redis**
 * **MySQL**
 * **MongoDB**
 * **Kafka**

Features:

* with asyncio support and share same API
* Baesd on Template Method and Factory Method
* Every failure network read operation will log to error log before retry 3(default) times
* Every failure network write operation will log to error log before retry 3(default) times
* Command line support for simple usage, python module provide more features
* Every Getter and Writer support filter, you can alter or drop your data in filter
* Auto header generation(csv/xlsx)/table generation(mysql) based on your data
* APIGetter, will request next page automatically，each page will retry max_retry before fail
* APIBulkGetter support bulk processing for APIGetter/url object, you can feed iterable object or async generator
* easy to use concurrent control for APIBulkGetter
* support normal call back and async call back
* persistent task to disk
-------------------

### catalog

* [Requirment](#requirment)
* [Installation](#installation)
* [Command line interface Example](#command-line-interface-example)
	* [Elasticsearch to CSV](#read-data-from-elasticsearch-convert-to-csv)
	* [API to xlsx](#read-data-from-api-convert-to-xlsx)
	* [json to csv](#read-data-from-json-convert-to-csv)
	* [csv to xlsx](#read-data-from-csv-convert-to-xlsx)
	* [Elasticsearch to csv with parameters](#read-data-from-elasticsearch-convert-to-csv-with-parameters)
	* [API to redis](#read-data-from-api-write-to-redis)
	* [redis to csv](#read-data-from-redis-write-to-csv)
	* [API to MySQL](#read-data-from-api-write-to-mysql)
	* [MySQL to redis](#read-data-from-mysql-write-to-redis)
	* [MongoDB to csv](#read-data-from-mongodb-write-to-csv)
* [Python module support](#python-module-support)
	* [ES to csv](#es-to-csv)
	* [API to xlsx](#api-to-xlsx)
	* [CSV to xlsx](#csv-to-xlsx)
	* [API to redis](#api-to-redis)
    * [redis to MySQL](#redis-to-mysql)
    * [MongoDB to redis](#mongodb-to-redis)
	* [Bulk API to ES/MongoDB/Json](#bulk-api-to-es-or-mongodb-or-json)
	* [Extract error info from API](#extract-error-info-from-api)
	* [call_back](#call_back)
	* [filter](#redis-to-mysql)
    * [done_if](#api-to-xlsx)
    * [persistent to disk(resume from break point)](#persistent-to-disk)
* [REDIS Usage](#redis-usage)
* [ES Base Operation](#es-base-operation)
	* [Read data from ES](#read-data-from-es)
	* [Write data to ES](#write-data-to-es)
	* [DELETE data from ES](#delete-data-from-es)
	* [API to ES in detail](#api-to-es-in-detail)
	* [Get ES Client](#get-es-client)
* [Config](#config)
	* [ini file](#ini-file)
	* [manual config in program](#manual-config-in-program)
* [Doc String](#doc-string)
* [Change Log](#changelog)
* [License](#license)

-------------------

#### Requirment
* python version >= 3.5.2
* If you need MySQL enable, your python version should be >= 3.5.3
* If you need MongoDB enable, your platform should not be **windows**
-------------------

#### Installation

	python3 -m pip install idataapi-transform
    # shell, run
    transform --help # explanation of each parameter and create configure file
    # edit ~/idataapi-transform.ini to config elasticsearch hosts, redis, mysql etc...

    # Install MySQL module, if your python version >= 3.5.3
    python3 -m pip install 'PyMySQL<=0.9.2,>=0.9'
    python3 -m pip install aiomysql

    # Install MongoDB module, if your platform is not Windows
    python3 -m pip install motor

-------------------

#### Command line interface Example

* Read data from **[API, ES, CSV, XLSX, JSON, Redis, MySQL, MongoDB]**
* Write data to **[CSV, XLSX, JSON, TXT, ES, Redis, MySQL, MongoDB]**

##### read data from Elasticsearch convert to CSV

will read at most **500** items from given **index**: **knowledge20170517**, and write to ./result.csv

	transform ES csv "knowledge20170517" --max_limit=500

##### read data from API convert to XLSX

will read all items from given api url, until no more next page, and save to dest(/Users/zpoint/Desktop/result.xlsx), **dest is optional, default is ./result.xlsx**

	transform API xlsx "http://xxx/post/dengta?kw=中国石化&apikey=xxx" "/Users/zpoint/Desktop/result"

##### read data from JSON convert to csv

will read items from json file, and save to **./result.csv**

	transform JSON csv "/Users/zpoint/Desktop/a.json"

##### read data from CSV convert to xlsx

will read items from csv file, and save to **./result.xlsx**

	transform CSV xlsx "./a.csv"

##### read data from Elasticsearch convert to CSV with parameters
* save csv with file encoding "gbk" **(--w_encoding)**
* specific index: knowledge20170517, **(knowledge20170517)**
* when read from Elasticsearch, specific request body **(--query_body)**

    	body = {
        	"size": 100,
        	"_source": {
            	"includes": ["location", "title", "city", "id"]
                }
              }

* before write to csv, add timestamp to each item, and drop items with null city **(--filter)**

        # create a file name my_filter.py (any filename will be accepted)
        import time
        def my_filter(item): # function name must be "my_filter"
            item["createtime"] = int(time.time())
            if item["city"]:
                return item # item will be write to destination
            # reach here, means return None, nothing will be write to destination

* Shell:

    	transform ES csv "knowledge20170517" --w_encoding gbk --query_body '{"size": 100, "_source": {"includes": ["location", "title", "city", "id"]}}' --filter ./my_filter.py

##### Read data from API write to Redis

* redis key name: my_key
* redis store/read support LIST and HASH, default value is LIST, you can change it with  --key_type parameter

will read data from ./a.csv, and save to redis LIST data structure, KEY: my_key

	transform API redis "http://xxx/post/dengta?kw=中国石化&apikey=xxx" my_key

##### Read data from Redis write to csv

will read data from redis key **my_key**, read at most 100 data， and save to **./result.csv**

	transform Redis csv my_key --max_limit 100

##### Read data from API write to MySQL

* auto create table if not exist

will read data from **API**, read at most 50 data， and save to MySQL table: **my_table**

	transform API MYSQL 'http://xxx' my_table --max_limit=50

##### Read data from MySQL write to redis

will read data from MySQL table **my_table**, each read operation fetch 60 items， and save to a redis LIST name **result**, **result** is the default key name if you don't provide one

	transform MYSQL redis my_table --per_limit=60

##### Read data from MongoDB write to csv

* you can provide --query_body

will read at most 50 data from "my_coll", and save to **./result.csv**

	transform mongo csv my_coll --max_limit=50

-------------------

#### Python module support

##### ES to csv

    import asyncio
	from idataapi_transform import ProcessFactory, GetterConfig, WriterConfig

	async def example():
        body = {
            "size": 100,
            "_source": {
                "includes": ["likeCount", "id", "title"]
                }
        }
        es_config = GetterConfig.RESConfig("post20170630", "news", max_limit=1000, query_body=body)
        es_getter = ProcessFactory.create_getter(es_config)
        csv_config = WriterConfig.WCSVConfig("./result.csv")
        with ProcessFactory.create_writer(csv_config) as csv_writer:
            async for items in es_getter:
                # do whatever you want with items
                csv_writer.write(items)

	if __name__ == "__main__":
        loop = asyncio.get_event_loop()
        loop.run_until_complete(example())

##### API to xlsx

    import time
    import asyncio
	from idataapi_transform import ProcessFactory, GetterConfig, WriterConfig

    yesterday_ts = int(time.time()) - 24 * 60 * 60

    def my_done_if(items):
        # RAPIConfig will fetch next page until
        # 1. no more page or
        # 2. reach max_limit or
        # 3. error occurs
        # if you want to terminate fetching in some condition, you can provide done_if function
        if items[-1]["publishDate"] < yesterday_ts:
        	return True
        return False

	async def example():
        api_config = GetterConfig.RAPIConfig("http://xxxx")
        # or you can use: api_config = GetterConfig.RAPIConfig("http://xxxx", done_if=my_done_if)
        getter = ProcessFactory.create_getter(api_config)
        xlsx_config = WriterConfig.WXLSXConfig("./result.xlsx")
        with ProcessFactory.create_writer(xlsx_config) as xlsx_writer:
        	async for items in getter:
                # do whatever you want with items
                xlsx_writer.write(items)

	if __name__ == "__main__":
        loop = asyncio.get_event_loop()
        loop.run_until_complete(example())

##### CSV to xlsx

    import asyncio
	from idataapi_transform import ProcessFactory, GetterConfig, WriterConfig

    async def example():
        csv_config = GetterConfig.RCSVConfig("./result.csv")
        getter = ProcessFactory.create_getter(csv_config)
        xlsx_config = WriterConfig.WXLSXConfig("./result.xlsx")
        with ProcessFactory.create_writer(xlsx_config) as xlsx_writer:
            for items in getter:
                # do whatever you want with items
                xlsx_writer.write(items)

    if __name__ == "__main__":
        loop = asyncio.get_event_loop()
        loop.run_until_complete(example())

##### API to redis

    import asyncio
	from idataapi_transform import ProcessFactory, GetterConfig, WriterConfig

    async def example():
        api_config = GetterConfig.RAPIConfig("http://xxx")
        getter = ProcessFactory.create_getter(api_config)
        redis_config = WriterConfig.WRedisConfig("key_a")
        with ProcessFactory.create_writer(redis_config) as redis_writer:
            async for items in getter:
                # do whatever you want with items
                await redis_writer.write(items)

    if __name__ == "__main__":
        loop = asyncio.get_event_loop()
        loop.run_until_complete(example())

##### redis to MySQL

    import asyncio
	from idataapi_transform import ProcessFactory, GetterConfig, WriterConfig

    def my_filter(item):
        # I am a filter
        # Every getter or writer created by ProcessFactory.create can set up a filter
        # every data will be pass to filter before return from getter, or before write to writer
        # you can alter data here, or drop data here
    if item["viewCount"] > 10:
        return item
        # if don't return anything(return None) means drop this data

	async def example():
        api_config = GetterConfig.RAPIConfig("http://xxxx", filter_=my_filter)
        getter = ProcessFactory.create_getter(api_config)
        mysql_config = WriterConfig.WMySQLConfig("my_table")
        with ProcessFactory.create_writer(mysql_config) as mysql_writer:
        	async for items in getter:
                # do whatever you want with items
                await mysql_writer.write(items)

        # await mysql_config.get_mysql_pool_cli() # aiomysql connection pool
        # mysql_config.connection # one of the connection in previous connection pool
        # mysql_config.cursor # cursor of previous connection
        # you should alaways call 'await mysql_config.get_mysql_pool_cli()' before use connection and cursor
        # provided by GetterConfig.RMySQLConfig and WriterConfig.WMySQLConfig

	if __name__ == "__main__":
        loop = asyncio.get_event_loop()
        loop.run_until_complete(example())

##### MongoDB to redis

    import asyncio
	from idataapi_transform import ProcessFactory, GetterConfig, WriterConfig

	async def example():
        mongo_config = GetterConfig.RMongoConfig("coll_name")
        mongo_getter = ProcessFactory.create_getter(mongo_config)
        redis_config = WriterConfig.WRedisConfig("my_key")
        with ProcessFactory.create_writer(redis_config) as redis_writer:
        	async for items in mongo_getter:
                # do whatever you want with items
                await mysql_writer.write(items)

        # print(mongo_config.get_mysql_pget_mongo_cli()) # motor's AsyncIOMotorClient instance
        # provided by GetterConfig.RMongoConfig and WriterConfig.WMongoConfig

	if __name__ == "__main__":
        loop = asyncio.get_event_loop()
        loop.run_until_complete(example())

##### Bulk API to ES or MongoDB or Json

    import asyncio
	from idataapi_transform import ProcessFactory, GetterConfig, WriterConfig

    """
    RAPIConfig support parameter:
    max_limit: get at most max_limit items, if not set, get all
    max_retry: if request fail, retry max_retry times
    filter_: run "transform --help" to see command line interface explanation for detail
    """

    def url_generator():
        for i in range(10000):
            yield url % (i, ) # yield RAPIConfig(url  % (i, )) will be OK

    async def example():
        # urls can be any iterable object, each item can be api url or RAPIConfig
        # RAPIBulkConfig accept a parameter: interval，means interval between each async generator return
        # if you set interval to 2 seconds，the async for will wait for 2 seconds before return，every data you get between these 2 seconds will be returned

        urls = ["http://xxxx", "http://xxxx", GetterConfig.RAPIConfig("http://xxxx"), ...]
        api_bulk_config = GetterConfig.RAPIBulkConfig(urls, concurrency=100)
        api_bulk_getter = ProcessFactory.create_getter(api_bulk_config)
        es_config = WriterConfig.WESConfig("profile201712", "user")
        with ProcessFactory.create_writer(es_config) as es_writer:
            async for items in api_bulk_getter:
                # do whatever you want with items
                await es_writer.write(items)

    async def example2mongo():
        urls = url_generator()
        api_bulk_config = GetterConfig.RAPIBulkConfig(urls, concurrency=50)
        api_bulk_getter = ProcessFactory.create_getter(api_bulk_config)
        # you can config host.port in configure file，or pass as parameters，parameters have higher priority than configure file
        mongo_config = WriterConfig.WMongoConfig("my_coll")
        with ProcessFactory.create_writer(mongo_config) as mongo_writer:
            async for items in api_bulk_getter:
                # do whatever you want with items
                await mongo_writer.write(items)

    # ******************************************************
    # Below is "async generator" example for RAPIBulkConfig
    # keyword "yield" in "async" function only support for python3.6+，for python 3.5+ please refer below
    # https://github.com/python-trio/async_generator
    # Only idataapi-transform version >= 1.4.4 support this feature
    # ******************************************************

    async def put_task2redis():
        writer = ProcessFactory.create_writer(WriterConfig.WRedisConfig("test"))
        await writer.write([
            {"keyword": "1"},
            {"keyword": "2"},
            {"keyword": "3"}
        ])

    async def async_generator():
        """
        I am async generator
        """
        getter = ProcessFactory.create_getter(GetterConfig.RRedisConfig("test"))
        async for items in getter:
            for item in items:
                r = GetterConfig.RAPIConfig("http://xxx%sxxx" % (item["keyword"], ), max_limit=100)
                yield r

    async def example2json():
        # await put_task2redis()
        urls = async_generator()
        api_bulk_config = GetterConfig.RAPIBulkConfig(urls, concurrency=30)
        api_bulk_getter = ProcessFactory.create_getter(api_bulk_config)
        json_config = WriterConfig.WJsonConfig("./result.json")
        with ProcessFactory.create_writer(json_config) as json_writer:
            async for items in api_bulk_getter:
                # do whatever you want with items
                json_writer.write(items)

	if __name__ == "__main__":
        loop = asyncio.get_event_loop()
        loop.run_until_complete(example())

##### Extract error info from API

    import asyncio
	from idataapi_transform import ProcessFactory, GetterConfig

	async def example_simple():
        # if return_fail set to true, after retry 3(default) times,
        # still unable to get data error info will be returned in "bad_items"
        url = "xxx"
        config = GetterConfig.RAPIConfig(url, return_fail=True)
        reader = ProcessFactory.create_getter(config)
        async for good_items, bad_items in reader:
        	print(good_items)
            if len(bad_items) > 0:
            	err_obj = bad_items[0]
                print(err_obj.response) # http body, if network down，it will be None
                print(err_obj.tag) # tag you pass to RAPIConfig, default None
                print(err_obj.source) #  url you pass to RAPIConfig
                print(err_obj.error_url) # the url that elicit error

    async def example():
        unfinished_id_set = {'246834800', '376796200', '339808400', ...}
        config = GetterConfig.RAPIBulkConfig((RAPIConfig(base_url % (i,), return_fail=True, tag=i) for i in unfinished_id_set), return_fail=True, concurrency=100)
        reader = ProcessFactory.create_getter(config)
        async for good_items, bad_items in reader:
            # A: When you set RAPIBulkConfig's return_fail to True,
            # 1）normal url will retuen error info
            # 2) RAPIConfig Object with return_fail set to True will retuen error info
            # 3) RAPIConfig Object with return_fail set to False will not retuen error info
            # B: Whrn you set RAPIBulkConfig's return_fail to False(default)
            # None of the previous situitions will return error info, same as the API to ES example above
        	print(bad_items)

    if __name__ == "__main__":
        loop = asyncio.get_event_loop()
        loop.run_until_complete(example_simple())

##### call_back

    import asyncio
	from idataapi_transform import ProcessFactory, GetterConfig, WriterConfig

    """
    call_back can be normal function or async funcion
     only RAPIConfig support call_back parameter, call_back will be used after filter，
    whatever call_back return will return to user
    """

    async def fetch_next_day(self, items):
        """ for each item, fetch seven days in order, combine data and return to user """
        prev_item = items[0]
        date_obj = # get a timestamp or datetime object from prev_item
        if date_obj not in seven_days:
            call_back = None
        else:
            call_back = self.fetch_next_day
        getter_config = GetterConfig.RAPIConfig(url, call_back=call_back)
        getter = ProcessFactory.create_getter(getter_config)
        async for items in getter:
            item = items[0]
            item = combine(item, prev_item)
            return [item]

    async def fetch_next_day_with_return_fail(self, good_items, bad_items):
        """ for each item, fetch seven days in order, combine data and return to user """
        if bad_items:
            # error in current level
            pass
        prev_item = good_items[0]
        date_obj = # get a timestamp or datetime object from prev_item
        if date_obj not in seven_days:
            call_back = None
        else:
            call_back = self.fetch_next_day_with_return_fail
        getter_config = GetterConfig.RAPIConfig(url, call_back=call_back, return_fail=True)
        getter = ProcessFactory.create_getter(getter_config)
        async for good_items, bad_items in getter:
            if bad_items:
                # next level retrn error, how to handle it?
                return None, bad_items

            item = good_items[0]
            item = combine(item, prev_item)
            return [item], bad_items

    async def start(self):
        id_set = {...a set of id...}
        url_generator = (GetterConfig.RAPIConfig(base_url % (id_, ), call_back=fetch_next_day) for id_ in self.id_set)
        bulk_config = GetterConfig.RAPIBulkConfig(url_generator, concurrency=20, interval=1)
        bulk_getter = ProcessFactory.create_getter(bulk_config)
        with ProcessFactory.create_writer(...) as writer:
            async for items in bulk_getter:
                await writer.write(items)

    async def start_with_return_fail(self):
        id_set = {...a set of id...}
        url_generator = (GetterConfig.RAPIConfig(base_url % (id_, ), call_back=fetch_next_day) for id_ in self.id_set, return_fail=True)
        bulk_config = GetterConfig.RAPIBulkConfig(url_generator, concurrency=20, interval=1, return_fail=True)
        bulk_getter = ProcessFactory.create_getter(bulk_config)
        with ProcessFactory.create_writer(...) as writer:
            async for items in bulk_getter:
                await es_writer.write([i for i in good_items if i is not None])

    if __name__ == "__main__":
        loop = asyncio.get_event_loop()
        loop.run_until_complete(start())

##### persistent to disk

    import asyncio
	from idataapi_transform import ProcessFactory, GetterConfig, WriterConfig

    ### persistent -> only RAPIBulkConfig support this parameter，it's a boolean value, means whether to persist finished job ID to disk, default value is False
    ### if persistent set to True, and you provide a batch of jobs to RAPIBulkConfig, because of the non-blocking event driven architecture，we can't simply record an order to disk, we create a json file in the working directory，and persist every finished job ID to disk, when program restart，it will load this json file，and won't execute those job that has been executed before
    ### persistent_key -> the json file name, to identify record of which batch of jobs
    ### persistent_start_fresh_if_done -> if all jobs done, whether remove the persistent json file, if the persistent file hasn't been removed and all of the jobs finished, next time you run the program, there will be no job to schedule, default is True
    ### persistent_to_disk_if_give_up ->  if there's a job fail after retry max_retry times, whether regard this job as success and persistent to disk or not, default is True

    async def exapmle():
        urls = [
            "http://xxx",
            "http://xxx",
            GetterConfig.RAPIConfig("http://xxx", persistent_to_disk_if_give_up=True)
        ]
        getter = ProcessFactory.create_getter(GetterConfig.RAPIBulkConfig(urls, persistent=True, persistent_to_disk_if_give_up=False))
        async for items in getter:
            print(items)

    if __name__ == "__main__":
        loop = asyncio.get_event_loop()
        loop.run_until_complete(example())

### REDIS Usage

    import asyncio
	from idataapi_transform import ProcessFactory, GetterConfig, WriterConfig

    async def example_simple():
        # default key_type is LIST in redis
        # you can pass parameter "encoding" to specify how to encode before write to redis, default utf8
        json_lists = [...]
        wredis_config = WriterConfig.WRedisConfig("my_key")
        writer = ProcessFactory.create_writer(wredis_config)
        await writer.write(json_lists)

        # get async redis client
        client = await wredis_config.get_redis_pool_cli()
        # if you instance a getter_config, you can get client by 'getter_config.get_redis_pool_cli()'
        # then, you can do watever you want in redis
        r = await client.hset("xxx", "k1", "v1")
        print(r)

    async def example():
        # specify redis's key_type to HASH, default is LIST
        # compress means string object is compressed by zlib before write to redis,
        # we need to decompress it before turn to json object
        # you can pass parameter "need_del" to specify whether need to del the key after get object from redis, default false
        # you can pass parameter "direction" to specify whether read data from left to right or right to left, default left to right(only work for LIST key type)
        getter_config = GetterConfig.RRedisConfig("my_key_hash", key_type="HASH", compress=True)
        async for items in reader:
            print(items)

    if __name__ == "__main__":
        loop = asyncio.get_event_loop()
        loop.run_until_complete(example())

#### ES Base Operation

##### Read data from ES

    import asyncio
	from idataapi_transform import ProcessFactory, GetterConfig

	async def example():
        # max_limit: means get at most max_limit items, if you don't provide it, means read all items
        # you can provide your query_body to es_config
        es_config = GetterConfig.RESConfig("post20170630", "news", max_limit=1000)
        es_getter = ProcessFactory.create_getter(es_config)
        async for items in es_getter:
            print(items)

	if __name__ == "__main__":
        loop = asyncio.get_event_loop()
        loop.run_until_complete(example())

##### Write data to ES

    import asyncio
	from idataapi_transform import ProcessFactory, WriterConfig

    def my_hash_func(item):
        # generate ES_ID by my_hash_func
        return hashlib.md5(item["id"].encode("utf8")).hexdigest()

	async def example():
        json_lists = [#lots of json object]
        # actions support create, index, update default index
        # you can ignore "id_hash_func" to use default function to create ES_ID, see "API to ES in detail" below
        es_config = WriterConfig.WESConfig("post20170630", "news", actions="create")
        es_writer = ProcessFactory.create_writer(es_config)
        await es_writer.write(json_lists)

	if __name__ == "__main__":
        loop = asyncio.get_event_loop()
        loop.run_until_complete(example())

##### DELETE data from ES

    import asyncio
    import json
    from idataapi_transform import ProcessFactory, WriterConfig

	async def example():
        # wrapper of delete_by_query API
        body = {"size": 100,  "query": {"bool": {"must": [{"term": {"createDate": "1516111225"}}]}}}
        writer = ProcessFactory.create_writer(WriterConfig.WESConfig("post20170630", "news"))
        r = await writer.delete_all(body=body)
        print(json.dumps(r))

	async def example_no_body():
        # same as above, without , delete all
        writer = ProcessFactory.create_writer(WESConfig("post20170630", "news"))
        r = await writer.delete_all()
        print(json.dumps(r))

	if __name__ == "__main__":
        loop = asyncio.get_event_loop()
        loop.run_until_complete(example())

##### API to ES in detail

    import time
    import asyncio
    from idataapi_transform import ProcessFactory, WriterConfig, GetterConfig
    """
    Every es document need an _id
    There are two rules to generate _id for ES inside this tool
    1) iF privide "id_hash_func" parameter when create WESConfig Object, _id will be id_hash_func(item)
    2) if rule 1 fail to match and the data(dictionary object) has key "id" and key "appCode"，_id will be md5(appCode_id)
    3) if rule 1 and rule 2 both fail to match, _id will be md5(str(item))
    """

    # global variables
    now_ts = int(time.time())

	def my_filter(item):
        # I am a filter
        # Every getter or writer created by ProcessFactory.create can set up a filter
        # every data will be pass to filter before return from getter, or before write to writer
        # you can alter data here, or drop data here
        if "posterId" in item:
        	return item
        # if don't return anything(return None) means drop this data

    async def example():
        # urls can be any iterable object, each item can be api url or RAPIConfig
        urls = ["http://xxxx", "http://xxxx", "http://xxxx", RAPIConfig("http://xxxx", max_limit=10)]
        # set up filter，drop every item without "posterId"
        api_bulk_config = GetterConfig.RAPIBulkConfig(urls, concurrency=100, filter_=my_filter)
        api_bulk_getter = ProcessFactory.create_getter(api_bulk_config)
        # you can also set up filter here
        # createDate parameter set same "createDate" for every data written by this es_writer
        # Of course，you can ignore "createDate", es_writer will set every data's "createDate" to the current system's timestamp when it performs write operation, you can disable it by parameter auto_insert_createDate=False
        # add parameter "appCode" for every data so that it can generate _id by rule 1
        es_config = WriterConfig.WESConfig("profile201712", "user", createDate=now_ts)
        with ProcessFactory.create_writer(es_config) as es_writer:
            async for items in api_bulk_getter:
                # do whatever you want with items
                await es_writer.write(items)

	if __name__ == "__main__":
        loop = asyncio.get_event_loop()
        loop.run_until_complete(example())

##### Get ES Client

    import asyncio
    import json
	from idataapi_transform import ProcessFactory, WriterConfig

	async def example():
        writer = ProcessFactory.create_writer(WriterConfig.WESConfig("post20170630", "news"))
        client = writer.config.es_client
        # a client based on elasticsearch-async, you can read offical document

	if __name__ == "__main__":
        loop = asyncio.get_event_loop()
        loop.run_until_complete(example())

-------------------

#### Config

##### ini file

By default, program will load config file in the following order
* the ini file your specific(see below)
* ./idataapi-transform.ini
* ~/idataapi-transform.ini

if none of the configure file exists, program will create **~/idataapi-transform.ini** automatically and use it as default

##### manual config in program

By default, program will log to file configured in **idataapi-transform.ini**, and also log to console, all of the log will be formatted
If you don't want any of it, you can disable it

    from idataapi_transform import ManualConfig
    ManualConfig.disable_log()

If you want to specific your own configure file

    from idataapi_transform import ManualConfig
    ManualConfig.set_config("/Users/zpoint/Desktop/idataapi-transform.ini")

If you want to change log directory in the run time

    from idataapi_transform import ManualConfig
    # at most 5MB per log file
    ManualConfig.set_log_path("/Users/zpoint/Desktop/logs/", 5242880)

-------------------

#### doc string

	from idataapi_transform import GetterConfig, WriterConfig

    # run help on config to see detail
    help(GetterConfig.RAPIConfig)
    """
    ...
    will request until no more next_page to get, or get "max_limit" items

    :param source: API to get, i.e. "http://..."
    :param per_limit: how many items to get per time
    :param max_limit: get at most max_limit items, if not set, get all
    :param max_retry: if request fail, retry max_retry times
    :param random_min_sleep: if request fail, random sleep at least random_min_sleep seconds before request again
    :param random_max_sleep: if request fail, random sleep at most random_min_sleep seconds before request again
    :param session: aiohttp session to perform request
    :param args:
    :param kwargs:

    Example:
        api_config = RAPIConfig("http://...")
        api_getter = ProcessFactory.create_getter(api_config)
        async for items in api_getter:
            print(items)
    ...
	"""

-------------------
#### API to Kafka
```python
import asyncio

from idataapi_transform import ProcessFactory, GetterConfig, WriterConfig, ManualConfig

ManualConfig.set_config("conf/idataapi-transform.ini")

async def example():
    urls = [
        "http://api01.idataapi.cn:8000/article/idataapi?kw=%E9%9A%86%E5%9F%BA%E8%82%A1%E4%BB%BD&KwPosition=3&size=20&catLabel2=%E8%82%A1%E7%A5%A8&apikey=test",
    ]
    api_bulk_config = GetterConfig.RAPIBulkConfig(urls, concurrency=1)
    api_bulk_getter = ProcessFactory.create_getter(api_bulk_config)
    kafka_config = WriterConfig.WKafkaConfig()
    with ProcessFactory.create_writer(kafka_config, topic="news") as kafka_writer:
        async for items in api_bulk_getter:
            # do whatever you want with items
            await kafka_writer.write(items)

if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(example())

```

---------------------------
#### ChangeLog
v 1.6.6 - 1.6.9
* redis manual db fix
* keep_other_fields, keep_fields
* mysql charset
* self define http headers

v 1.6.3 - 1.6.4
* persistent to disk
* debug mode support

v 1.5.1 - 1.6.1
* random sleep float seconds support
* es specific host && headers
* RAPIGetter HTTP POST support
* xlsx/csv headers, append mode support

v 1.4.7 - 1.5.1
* done_if param support
* manual success_ret_code config for user
* xlsxWriter replace ilegal characters automatically

v 1.4.4 - 1.4.6
* RAPIBulkGetter support async generator
* ini config relative path support, manual config support

v 1.4.3
* fix logging bug
* max_limit limit number of data before filter
* report_interval add for APIGetter

v 1.4.1
* call_back support
* mongodb auth support and motor 2.0 support
* mongodb support
* fix APIBulkGetter incompleted data bug
* 3.5 compatiable
* ESGetter get all data instead of half
* compatible with elasticsearch-async-6.1.0
* ESClient singleton

v 1.2.0
* mysql support
* redis support
* retry 3 times for every write operation
* ES create operation
* shorter import directory

v.1.0.1 - 1.1.1
* fix es getter log error
* unclose session error for elasticsearch
* fix ES infinity scroll
* fix bug (cli)
* es_client msearch support
* fix XLSX reader
* return_fail for APIGetter
* compatible for aiohttp 3.x

v.1.0
* fix ESWriter log bug
* timeout add for ESWriter

v.0.9
* filter for every getter
* createDate for ESWriter
* APIGetter per_liimt bug fix
* new session for all RAPIBulkConfig

v.0.8
* error logging when unable to insert to target for ESWriter
* actions parameter add for WESConfig
* id_hash func change for ESWriter

v.0.7
* remove APIGetter infinity loop for empty result

v.0.6
* No error when read empty item from ESGetter

v.0.5
* fetch more items for ESGetter in CLI per request
* per_limit param fix for ESGetter in CLI

v.0.4
* appCode for ESWriter Config
* ESGetter CLI bug fix

v.0.3
* doc string for each config
* RAPIBulkConfig support

-------------------

#### License

http://rem.mit-license.org

repository: zpoint/idataapi-transform

repo_name: zpoint/idataapi-transform


text:
---
order: 99
title: FAQ
type: Documents
---

## path

Many functions require passing `path` parameter, it uses `/` to represent form object path, following JSON Schema is an example:

```ts
schema: SFSchema = {
  properties: {
    app: {
      type: 'string',
      title: 'APP',
    },
    user: {
      type: 'object',
      properties: {
        name: {
          type: 'string',
        },
        age: {
          type: 'number',
        },
      }
    },
  }
};
```

Following `path` are all valid:

- `/app`
- `/user/name`

Starting with `/` indicates to search from root path, otherwise, search from current path.

## Why some customized validations are not working in non-realtime validation?

Because non-realtime validation (set `liveValidate:false`) doesn't revalidate every element, although it can be done, customized validation may involve asynchronized validation, cannot guarantee the order of execution, so non-realtime validation actually only validates JSON Schema itself.

## How to use Schema dynamically?

There are two common scenarios:

**1. Schema is limited by remote data after definition**

```ts
@ViewChild('sf', { static: false }) sf: SFComponent;
schema: SFSchema = {
  properties: {
    app: {
      type: 'string',
      title: 'Affiliated Application',
      ui: 'select',
      enum: []
    }
  }
};

ngOnInit() {
  this.http.get('/apps').subscribe((res: any) => {
    this.schema.properties.app.enum = res;
    this.sf.refreshSchema();
  });
}
```

**2. Remote Schema**

```ts
schema: SFSchema = {
  properties: {}
};

ngOnInit() {
  this.http.get('/schema').subscribe((res: any) => this.sf.refreshSchema(res));
}
```

## When to use `default`?

`default` of Schema is used to set initialization, usually, need to provide `formData` when editting forms, but for adding forms, should rely on `default` to provide a better user friendly experience form to users.

## How to refresh a specific Schema?

You can get a specific attribute of Schema by calling `getProperty` function, the attribute includes Schema data and Ui data, you can modify these data and re-render the Schema by calling `reset` function, for example:

```ts
const statusProperty = this.sf.getProperty('/status')!;
statusProperty.schema.enum = ['1', '2', '3'];
statusProperty.widget.reset('2');
```


text:
# MIDI Websocket

Control MIDI devices via Websockets in JavaScript.

This is the client for https://github.com/PeterSR/python-midi-websocket-server


text:
---
layout: page
title: "Getting Started"
date: 2016-08-26 08:36:06
author: Alexander Thiemann
permalink: /tutorials/getting-started
redirect_from:
  - /tutorial
  - /tutorial/
---

## Setup

Before using Spock, you'll need to install the Haskell toolchain on your machine. We recommend using the
[stack](http://haskellstack.org/) tool to quickly get started! Our guide will be `stack` based, but you can
easily translate this to `cabal`.
Next, you can prepare a directory for your first Spock powered application:

1. Create a new project using `stack new Spock-example`
1. Jump into the directory `cd Spock-example`

## Dependencies

First we will add `Spock` to our dependencies by opening `Spock-example.cabal` and adding `Spock >=0.14`, `mtl` and `text` to `build-depends` in the
`executable Spock-example-exe` section. Also, the `extra-deps:` section in the generated `stack.yaml` file to:

```
extra-deps:
  - Spock-<IP_ADDRESS>
  - Spock-core-<IP_ADDRESS>
  - reroute-<IP_ADDRESS>
  - stm-containers-1.2
  - focus-<IP_ADDRESS>
  - stm-hamt-<IP_ADDRESS>
  - primitive-extras-0.8
  - primitive-unlifted-<IP_ADDRESS>
```

Next we build everything once: `stack build --fast --pedantic`.

## Hello world

Now it is time to write some Haskell code. Open `app/Main.hs` in your favorite editor and replace the content with:

{% highlight haskell %}
{% raw %}{-# LANGUAGE OverloadedStrings #-}{% endraw %}
module Main where

import Web.Spock
import Web.Spock.Config

import Control.Monad.Trans
import Data.IORef
import qualified Data.Text as T

data MySession = EmptySession
data MyAppState = DummyAppState (IORef Int)

main :: IO ()
main =
    do ref <- newIORef 0
       spockCfg <- defaultSpockCfg EmptySession PCNoDatabase (DummyAppState ref)
       runSpock 8080 (spock spockCfg app)

app :: SpockM () MySession MyAppState ()
app =
    do get root $
           text "Hello World!"
       get ("hello" <//> var) $ \name ->
           do (DummyAppState ref) <- getState
              visitorNumber <- liftIO $ atomicModifyIORef' ref $ \i -> (i+1, i+1)
              text ("Hello " <> name <> ", you are visitor number " <> T.pack (show visitorNumber))
{% endhighlight %}

Next, run `stack build --fast --pedantic` again to build the project. `stack exec Spock-example-exe` should start the executable - you may now point your browser to `http://localhost:8080` and `http://localhost:8080/hello/[YOUR_NAME]`.

## Code explained

Let us take a quick look at the interesting parts of the example. First, we import the core Spock modules:
{% highlight haskell %}
import Web.Spock
import Web.Spock.Config
{% endhighlight %}

Then, in the `main` function, we start out by configuring Spock. To do that, we need to describe what an
empty session for an individual user will look like (in our case, the `EmptySession` from our `MySession` type),
if and how we would like to use a database (no database, for now, so `PCNoDatabase`) and how our initial global
application state will look like. This is very useful to pass around configuration or other globally shared
information. We'll use it to implement a small hit counter by putting an `IORef Int` in our state. Let's not worry
about the last line in `main` just now, and move on to `app`.

The definition of a Spock application lives in the `SpockM conn sess st a` monad. The `conn` type parameter describes
what our database connection looks like (`()` for no database), the `sess` is the type of our session and `st` the
type of our global application state. Thus, for us: `SpockM () MySession MyAppState ()`. Inside the `SpockM` monad,
we'll wire URLs to actions. You can think of it as a [`Writer` monad](https://hackage.haskell.org/package/mtl-2.2.1/docs/Control-Monad-Writer-Lazy.html).
To connect an URL to an action, we use *routes*. A route is either

- a static route piece, such as `"hello"` or `"blog"` or `root` (the `/` route)
- a route parameter: `var`
- or a combination of the above, using the `<//>` operator, such as `"hello" <//> var`.

A static route piece will match the exact text counterpart, eg. a route `"hello" <//> "world"` would match the URL
`/hello/world` and `/hello/world/`. A route parameter matches anything that does not contain a `/` and parses as the
inferred type of the parameter. The type of the parameter is inferred by the action bound to the route. In the example above, our action is:
{% highlight haskell %}
\name -> text ("Hello " <> name <> "!")
{% endhighlight %}
This lambda function takes a `Text`, integrates it between `"Hello "` and `"!"`, and returns it to the framework. Thus,
the `var` in the route `"hello" <//> var` will require the parameter to be a text. If our function would take more than one argument, or our route would not contain a parameter this would result in a type error.

Putting it all together, we first need an HTTP-Verb to match against. In the example, we match `GET` requests, so we'll use the `get` function to
wire our action. Next we specify the route (e.g. `root`), and then the handler (e.g. `text "Hello world!"`). Handlers run in the `SpockAction conn sess st a` monad. You can use various functions to read headers and the HTTP body, and return content to the browser. Note that after a function returning content to the browser is called (such as `text`), the action is terminated.

After having defined the application, you can create a [`Wai.Middleware`](https://hackage.haskell.org/package/wai) from it using `spock spockCfg app` and then run it using `runSpock 8080` (choosing any port you like). Internally, the application is run by the [warp](https://hackage.haskell.org/package/warp) web server.

## Next Steps

- Read the documentation on [Hackage](https://hackage.haskell.org/package/Spock)
- Check out and/or contribute to the [example project](https://github.com/agrafix/funblog)
- Contribute on [Github](https://github.com/agrafix/Spock) by reporting or pull-requesting missing features and bugs
- Check out the *Addons* and *Works great with* section in the pages footer
- Watch our [blog]({{ "/blog/" | prepend: site.baseurl }})
- And of course: build your Spock powered application

repository: agrafix/Spock

gha_language: Haskell

repo_name: agrafix/Spock


text:
# API version {#concept_109364_zh .concept}

This topic describes the MQ for MQTT API releases.

It also provides the functions of each API version. See the following table.

|API version|Release date|Description|
|-----------|------------|-----------|
|3.1.0|2019-02-15| -   Incompatible with the 2.x.x version and has the same functions as the online console.
-   Provided all the functions of instantiation.
-   Changed the consumerId and producerId parameters to the groupId parameter.
-   Removed the OnsRegionId parameter to simplify API calling.

 |
|2.1.4|2019-01-25| -   Compatible with the 2.x.x version.
-   Instantiation fields returned by calling OnsTopicList.

 |
|2.0.1|2018-04-17|Added an operation for creating an MQTT group ID.|
|2.0.0|2017-09-19| -   Added an operation for message trace query.
-   Added access control to the write operation.

 |
|1.3.3|2017-08-10| -   Added an operation for read and write prohibiting.
-   Added an operation for prohibiting ConsumerId reading.

 |

path: /intl.en-US/O&M API Reference/API version.md

filename: API version.md


text:
---
layout: page
title: NorthHaven Church Missions Information
---

## Current Partnerships ##

* [Cooperative Baptist Fellowship](http://www.cbf.net/)
* [Cooperating Baptist Fellowship of Oklahoma](http://www.cbfok.org/)
* [Baptist General Convention of Texas](http://texasbaptists.org/)
* [Baptist Center for Ethics](http://www.ethicsdaily.com/)
* [Baptist Joint Committee](http://bjconline.org/)
* [Baptist News Global](https://baptistnews.com/)

## Other Possible Partnerships ##

* [Alliance of Baptists](http://allianceofbaptists.org/)
* [Central Baptist Seminary](http://cbts.edu/)
* [HisNets](http://hisnets.org/)
* [Center for Children and Families](http://www.ccfinorman.org/ )
* [Food and Shelter](http://www.foodandshelterinc.org/ )
* [Regional Food Bank of Oklahoma](https://www.regionalfoodbank.org )
* [Cleveland County Habitat for Humanity](http://www.cchfh.org )
* [Oklahoma Conference of Churches](http://okchurches.org )
* [Baptist Women in Ministry](http://bwim.info)

repository: rlridenour/jekyll-blog

repo_name: rlridenour/jekyll-blog


text:
# Btcmon
===========

Btcmon is an application designed to provide a user friendly interface to monitor Bitcoin prices published by different exchanges.

Platforms
-----------------
* Browsers

Start Services
-----------------
* Server
```
cd server
npm install
node app.js
```

* Client
```
cd hybrid-client
npm install
ionic serve
```

You should expect the application running on http://localhost:8100


text:
---
name: Conceptual question
about: A question about the project (how to use, not sure if something is wrong, etc.)

---

<!-- Not following the template might result in your issue being closed without further notice -->
**_Question_**

* [ ] I've checked documentation and searched for existing issues
* [ ] I tried the [spectrum channel](https://spectrum.chat/?t=dad48299-3dfc-4e10-b6da-9af1e39498a3)

<!-- Write your question below -->

repository: NoFishLikeIan/mobx-state-tree

repo_name: NoFishLikeIan/mobx-state-tree


text:
Documentation
=============

The documentation of munkireport is a work in progresss, if you want to contribute, please fork the wiki and send a pull request.

Installation
---

The installation is covered in setup.md

Performance
---

Running munkireport with up to 400 (300 daily active) clients with the SQLite backend should be ok. If you're adding more clients you'll need a database backend that allows for more concurrency like MySQL.

Security
---

Although munkireport should be pretty secure out-of-the-box, there are some areas where you could improve the security.

#### Use https for the webserver

Logging into the munkireport webapp sends your username and password in cleartext, someone might intercept your credentials and use them to
login. If you use https, the communication between your browser and the webserver is encrypted.

#### Remove the database from the documentroot

Add instructions.

#### Remove the webapp from the documentroot

Add instructions.

repository: tuxudo/munkireport-php

gha_language: PHP

repo_name: tuxudo/munkireport-php


text:
# Npm-registry-as-cli
Get the JSON payload returned by the NPM registry for a given package.

## Getting Started
This package is available in the Node Package Repository and can be easily installed with [npm](https://docs.npmjs.com/getting-started/what-is-npm) or [yarn](https://yarnpkg.com).

```bash
$ npm install npm-registry-as-cli -g
# or
$ npx npm-registry-as-cli
```

## Usage example
When installed globally the `npmreg` executable will be exposed in your terminal

```bash
$ npmreg httpie
```

for a given version of a package

```bash
$ npmreg httpie 1.1.2
```

## License
MIT

repository: fraxken/npm-registry-as-cli

repo_name: fraxken/npm-registry-as-cli


text:
---
description: The AMQP1_0 sink connector pulls messages from Pulsar topics and persists messages to AMQP brokers.
author: ["StreamNative"]
contributors: ["StreamNative"]
language: Java
document:
source: "https://github.com/streamnative/pulsar-io-amqp-1-0/tree/branch-2.7.1/io-amqp1_0-impl/src/main/java/org/apache/pulsar/ecosystem/io/amqp"
license: Apache License 2.0
tags: ["Pulsar IO", "AMQP", "Qpid", "JMS", "Sink"]
alias: AMQP1_0 sink
features: ["Use AMQP1_0 sink connector to sync data from Pulsar"]
license_link: "https://www.apache.org/licenses/LICENSE-2.0"
icon: "/images/connectors/amqp-logo.png"
download: "https://github.com/streamnative/pulsar-io-sqs/releases/download/v2.7.0/pulsar-io-sqs-2.7.0.nar"
support: StreamNative
support_link: https://streamnative.io
support_img: "/images/connectors/streamnative.png"
dockerfile:
owner_name: "StreamNative"
owner_img: "/images/streamnative.png"
id: "amqp-1-0-sink"
---

The AMQP1_0 sink connector pulls messages from Pulsar topics and persists messages to [AMQP 1.0](https://www.amqp.org/).

![](/images/connectors/amqp-1-0-sink.png)

# How to get

You can get the AMQP1_0 sink connector using one of the following methods:

* Download the NAR package from [here](https://github.com/streamnative/pulsar-io-amqp-1-0/releases/download/v<IP_ADDRESS>/pulsar-io-amqp1_0-<IP_ADDRESS>.nar).

* Build it from source code.

  1. Clone the source code to your machine.

        ```bash
        git clone https://github.com/streamnative/pulsar-io-amqp-1-0
        ```

  2. Assume that `PULSAR_IO_AMQP1_0_HOME` is the home directory for the `pulsar-io-amqp1_0` repo. Build the connector in the `${PULSAR_IO_AMQP1_0_HOME}` directory.

        ```bash
        mvn clean install -DskipTests
        ```

        After the connector is successfully built, a `NAR` package is generated under the `target` directory.

        ```bash
        ls pulsar-io-amqp1_0/target
        pulsar-io-amqp1_0-{version}.nar
        ```

# How to configure

Before using the AMQP1_0 sink connector, you need to configure it.

You can create a configuration file (JSON or YAML) to set the following properties.

| Name | Type|Required | Default | Description
|------|----------|----------|---------|-------------|
| `protocol` |String| true | "amqp" | The AMQP protocol. |
| `host` | String| true | " " (empty string) | The AMQP service host. |
| `port` | int |true | 5672 | The AMQP service port. |
| `username` | String|false | " " (empty string) | The username used to authenticate to ActiveMQ. |
| `password` | String|false | " " (empty string) | The password used to authenticate to ActiveMQ. |
| `queue` | String|false | " " (empty string) | The queue name that messages should be read from or written to. |
| `topic` | String|false | " " (empty string) | The topic name that messages should be read from or written to. |
| `activeMessageType` | String|false |0 | The ActiveMQ message simple class name. |
| `onlyTextMessage` | boolean | false | false | If it is set to `true`, the AMQP message type must be set to `TextMessage`. Pulsar consumers can consume the messages with schema ByteBuffer. |

**Example**

* JSON

    ```json
    {
        "tenant": "public",
        "namespace": "default",
        "name": "amqp1_0-sink",
        "inputs": ["user-op-queue-topic"],
        "archive": "connectors/pulsar-io-amqp1_0-{version}.nar",
        "parallelism": 1,
        "configs":
        {
            "protocol": "amqp",
            "host": "localhost",
            "port": "5672",
            "username": "guest",
            "password": "guest",
            "queue": "user-op-queue-pulsar"
        }
    }
    ```

* YAML

    ```yaml
    tenant: "public"
    namespace: "default"
    name: "amqp1_0-sink"
    inputs:
      - "user-op-queue-topic"
    archive: "connectors/pulsar-io-amqp1_0-{version}.nar"
    parallelism: 1

    configs:
        protocol: "amqp"
        host: "localhost"
        port: "5672"
        username: "guest"
        password: "guest"
        queue: "user-op-queue-pulsar"
    ```

# How to use

You can use the AMQP1_0 sink connector as a non built-in connector or a built-in connector as below.

## Use as non built-in connector

If you already have a Pulsar cluster, you can use the AMQP1_0 sink connector as a non built-in connector directly.

This example shows how to create an AMQP1_0 sink connector on a Pulsar cluster using the command [`pulsar-admin sinks create`](http://pulsar.apache.org/tools/pulsar-admin/2.8.0-SNAPSHOT/#-em-create-em--24).

```
PULSAR_HOME/bin/pulsar-admin sinks create \
--name amqp1_0-sink \
--archive pulsar-io-amqp1_0-{version}.nar \
--classname org.apache.pulsar.ecosystem.io.amqp.AmqpSink \
--sink-config-file amqp-sink-config.yaml
```

## Use as built-in connector

You can make the AMQP1_0 sink connector as a built-in connector and use it on standalone cluster, on-premises cluster, or K8S cluster.

### Standalone cluster

1. Prepare AMQP service using Solace.

    ```
    docker run -d -p 8080:8080 -p:8008:8008 -p:1883:1883 -p:8000:8000 -p:5672:5672 -p:9000:9000 -p:2222:2222 --shm-size=2g --env username_admin_globalaccesslevel=admin --env username_admin_password=admin --name=solace solace/solace-pubsub-standard
    ```

2. Copy the NAR package of the AMQP1_0 sink connector to the Pulsar connectors directory.

    ```
    cp pulsar-io-amqp1_0-{version}.nar
    $PULSAR_HOME/connectors/pulsar-io-amqp1_0-{version}.nar
    ```

3. Start Pulsar in standalone mode.

    ```
    PULSAR_HOME/bin/pulsar standalone
    ```

   You can find the similar logs as below.

    ```
    Searching for connectors in /Volumes/other/apache-pulsar-2.8.0-SNAPSHOT/./connectors
    Found connector ConnectorDefinition(name=amqp1_0, description=AMQP1_0 source and AMQP1_0 connector, sourceClass=org.apache.pulsar.ecosystem.io.amqp.AmqpSource, sinkClass=org.apache.pulsar.ecosystem.io.amqp.AmqpSink, sourceConfigClass=null, sinkConfigClass=null) from /Volumes/other/apache-pulsar-2.8.0-SNAPSHOT/./connectors/pulsar-io-amqp1_0-2.7.0.nar
    Searching for functions in /Volumes/other/apache-pulsar-2.8.0-SNAPSHOT/./functions
    ```

4. Create an AMQP1_0 sink.

    **Input**

    ```
    PULSAR_HOME/bin/pulsar-admin sinks create \
    --sink-config-file amqp-sink-config.yaml \
    --custom-schema-inputs '{"user-op-queue-topic": "org.apache.pulsar.client.impl.schema.ByteBufferSchema"}'
    ```

    **Output**

    ```
    "Created successfully"
    ```

    Verify whether the sink is created successfully or not.

    **Input**

    ```
    PULSAR_HOME/bin/pulsar-admin sinks list
    ```

    **Output**

    ```
    [
    "amqp1_0-sink"
    ]
    ```

    Check the sink status.

    **Input**

    ```
    PULSAR_HOME/bin/pulsar-admin sinks status --name amqp1_0-sink
    ```

    **Output**

    ```
      "numInstances" : 1,
      "numRunning" : 1,
      "instances" : [ {
        "instanceId" : 0,
        "status" : {
          "running" : true,
          "error" : "",
          "numRestarts" : 0,
          "numReadFromPulsar" : 0,
          "numSystemExceptions" : 0,
          "latestSystemExceptions" : [ ],
          "numSinkExceptions" : 0,
          "latestSinkExceptions" : [ ],
          "numWrittenToSink" : 0,
          "lastReceivedTime" : 0,
          "workerId" : "c-standalone-fw-localhost-8080"
        }
      } ]
    }
    ```

5. Send messages to Pulsar topics.

    ```
    @Test
    public void generateMessages() throws Exception {
        @Cleanup
        PulsarClient pulsarClient = PulsarClient.builder().serviceUrl("pulsar://localhost:6650").build();
        @Cleanup
        Producer<ByteBuffer> producer = pulsarClient.newProducer(Schema.BYTEBUFFER)
                .topic("user-op-queue-topic")
                .create();

        JmsConnectionFactory jmsConnectionFactory = new JmsConnectionFactory();
        @Cleanup
        JMSContext jmsContext = jmsConnectionFactory.createContext();

        for (int i = 0; i < 10; i++) {
            JmsTextMessage textMessage = (JmsTextMessage) jmsContext.createTextMessage("text message - " + i);
            ByteBuf byteBuf = (ByteBuf) textMessage.getFacade().encodeMessage();
            producer.send(byteBuf.nioBuffer());
        }
        System.out.println("finish send messages.");
    }
    ```

6. Consume messages from AMQP service using the `receiveMessages` method.

    **Input**

    ```
    @Test
    public void receiveMessages() throws Exception {
        ConnectionFactory connectionFactory = new JmsConnectionFactory("guest", "guest", "amqp://localhost:5672");
        @Cleanup
        Connection connection = connectionFactory.createConnection();
        connection.start();
        @Cleanup
        Session session = connection.createSession();
        @Cleanup
        MessageConsumer consumer = session.createConsumer(new JmsQueue("user-op-queue-pulsar"));
        for (int i = 0; i < 10; i++) {
            JmsTextMessage textMessage = (JmsTextMessage) consumer.receive();
            System.out.println("receive msg content: " + textMessage.getText());
            textMessage.acknowledge();
        }
    }
    ```

    Check the sink status.

    **Input**

    ```
    PULSAR_HOME/bin/pulsar-admin sinks status --name amqp1_0-sink
    ```

    **Output**

    ```
    {
      "numInstances" : 1,
      "numRunning" : 1,
      "instances" : [ {
        "instanceId" : 0,
        "status" : {
          "running" : true,
          "error" : "",
          "numRestarts" : 0,
          "numReadFromPulsar" : 10,
          "numSystemExceptions" : 0,
          "latestSystemExceptions" : [ ],
          "numSinkExceptions" : 0,
          "latestSinkExceptions" : [ ],
          "numWrittenToSink" : 10,
          "lastReceivedTime" :<PHONE_NUMBER>713,
          "workerId" : "c-standalone-fw-localhost-8080"
        }
      } ]
    }
    ```

### On-premises cluster

This example explains how to create an AMQP1_0 sink connector on an on-premises cluster.

1. Copy the NAR package of the AMQP1_0 connector to the Pulsar connectors directory.

    ```
    cp pulsar-io-amqp1_0-{version}.nar $PULSAR_HOME/connectors/pulsar-io-amqp1_0-{version}.nar
    ```

2. Reload all [built-in connectors](https://pulsar.apache.org/docs/en/next/io-connectors/).

    ```
    PULSAR_HOME/bin/pulsar-admin sinks reload
    ```

3. Check whether the AMQP1_0 sink connector is available on the list or not.

    ```
    PULSAR_HOME/bin/pulsar-admin sinks available-sinks
    ```

4. Create an AMQP1_0 sink connector on a Pulsar cluster using the [`pulsar-admin sinks create`](http://pulsar.apache.org/tools/pulsar-admin/2.8.0-SNAPSHOT/#-em-create-em--24) command.

    ```
    PULSAR_HOME/bin/pulsar-admin sinks create \
    --sink-type amqp1_0 \
    --sink-config-file amqp-sink-config.yaml \
    --name amqp1_0-sink
    ```

### K8S cluster

This example demonstrates how to create an AMQP1_0 sink connector on a K8S cluster.

1. Build a new image based on the Pulsar image with the AMQP1_0 sink connector and push the new image to your image registry.

    This example tags the new image as `streamnative/pulsar-amqp1_0:2.7.0`.

    ```Dockerfile
    FROM apachepulsar/pulsar-all:2.7.0
    RUN curl https://github.com/streamnative/pulsar-io-amqp1-0/releases/download/v{version}/pulsar-io-amqp1_0-{version}.nar -o /pulsar/connectors/pulsar-io-amqp1_0-{version}.nar
    ```

2. Extract the previous `--set` arguments from K8S to the `pulsar.yaml` file.

    ```
    helm get values <release-name> > pulsar.yaml
    ```

3. Replace the `images` section in the `pulsar.yaml` file with the `images` section of `streamnative/pulsar-amqp1_0:2.7.0`.

4. Upgrade the K8S cluster with the `pulsar.yaml` file.

    ```
    helm upgrade <release-name> streamnative/pulsar \
        --version <new version> \
        -f pulsar.yaml
    ```

    > **Tip**
    >
    > For more information about how to upgrade a Pulsar cluster with Helm, see [Upgrade Guide](https://docs.streamnative.io/platform/latest/install-and-upgrade/helm/install/upgrade).

5. Create an AMQP1_0 sink connector on a Pulsar cluster using the [`pulsar-admin sinks create`](http://pulsar.apache.org/tools/pulsar-admin/2.8.0-SNAPSHOT/#-em-create-em--24) command.

    ```
    PULSAR_HOME/bin/pulsar-admin sinks create \
    --sink-type amqp1_0 \
    --sink-config-file amqp-sink-config.yaml \
    --name amqp1_0-sink
    ```

repository: yunhorn/pulsar-hub

repo_name: yunhorn/pulsar-hub


text:
# Webrecorder Community

Welcome to the Webrecorder community info page!

This page is shared info for all Webrecorder projects.

### Community Discussions (via Gitter)

Feel free to use this Gitter channel for discussions about all Webrecorder tools and projects.

[![Chat about Webrecorder at https://gitter.im/webrecorder/community](https://badges.gitter.im/webrecorder/webrecorder.svg)](https://gitter.im/webrecorder/community)

### Code of Conduct.

Webrecorder Community is adopting the Contributor Covenenant [Code of Conduct](CODE_OF_CONDUCT.md)

repository: webrecorder/community

repo_name: webrecorder/community