text:
---
layout:     post
title:      "Parsing XML files made simple by PySpark"
date:       2019-07-14 12:00:00
author:     "Jason Feng"
header-img: "img/post-bg-2015.jpg"
excerpt_separator: <!--more-->
tags:
    - PySpark
    - XML
    - Data Processing
---

Imagine you are given a task to parse thousands of xml files to extract the information, write the records into table format with proper data types, the task must be done in a timely manner and is repeated every hour. What are you going to do? With Apache Spark, the embarrassingly parallel processing framework, it can be done with much less effort.

<!--more-->

### Introduction
In this post, we are going to use PySpark to process xml files to extract the required records, transform them into DataFrame, then write as csv files (or any other format) to the destination. The input and the output of this task looks like below.
![](/img/xml-2-dataframe-2019-07-20.png)

### XML files
[XML](https://www.w3schools.com/xml/xml_whatis.asp) is designed to store and transport data. XML is self-descriptive which makes it flexibile and extensible to store different kinds of data.

On the other hand, it makes difficult to convert into tabular data because of its nature of semi-structured. For example, in the below XML excerption, the `description` element can be expanded to multiple lines. The `price` element can be omitted because it is yet to be determined.

```xml
   <book id="bk119">
      <author>Feng, Jason</author>
      <title>Playground</title>
      <description>This is the place where Jason puts his fun stuff
      mainly related with Python, R and GCP.</description>
   </book>
```

### Solution
This is my scribble of the solution.
![](/img/parse-xml-scribble.PNG)

#### Step 1: Read XML files into RDD
We use `spark.read.text` to read all the xml files into a DataFrame. The DataFrame is with one column, and the value of each row is the whole content of each xml file. Then we convert it to RDD which we can utilise some low level API to perform the transformation.

```python
# read each xml file as one row, then convert to RDD
file_rdd = spark.read.text("./data/*.xml", wholetext=True).rdd
```

Here is the output of one row in the DataFrame.
```python

[Row(value='<?xml version="1.0"?>\r\n<catalog>\r\n   <book id="bk119">\r\n      <author>Feng, Jason</author>\r\n      <title>Playground</title>\r\n      <description>This is the place where Jason puts his fun stuff\r\n      mainly related with Python, R and GCP.</description>\r\n   </book>\r\n</catalog>')]
```

#### Step 2: Parse XML files, extract the records, and expand into multiple RDDs
Now it comes to the key part of the entire process. We need to parse each xml content into records according the pre-defined schema.

First, we define a function using Python standard library [`xml.etree.ElementTree`](https://docs.python.org/3/library/xml.etree.elementtree.html) to parse and extract the xml elements into a list of records. In this function, we cater for the scenario that some elements are missing which `None` is returned. It also casts `price` to float type and `publish_date` to date type.

```python
def parse_xml(rdd):
    """
    Read the xml string from rdd, parse and extract the elements,
    then return a list of list.
    """
    results = []
    root = ET.fromstring(rdd[0])

    for b in root.findall('book'):
        rec = []
        rec.append(b.attrib['id'])
        for e in ELEMENTS_TO_EXTRAT:
            if b.find(e) is None:
                rec.append(None)
                continue
            value = b.find(e).text
            if e == 'price':
                value = float(value)
            elif e == 'publish_date':
                value = datetime.strptime(value, '%Y-%m-%d')
            rec.append(value)
        results.append(rec)

    return results
```

Then we use [`flatMap`](https://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD.flatMap) function which each input item as the content of an XML file can be mapped to multiple items through the function `parse_xml`. `flatMap` is one of the functions made me "*WoW*" when I first used Spark a few years ago.

```python
# parse xml tree, extract the records and transform to new RDD
records_rdd = file_rdd.flatMap(parse_xml)
```

#### Step 3: Convert RDDs into DataFrame
We then convert the transformed RDDs to DataFrame with the pre-defined schema.
```python
# convert RDDs to DataFrame with the pre-defined schema
book_df = records_rdd.toDF(my_schema)
```
The DataFrame looks like below.
```
+-------+--------------------+--------------------+---------------+-----+------------+--------------------+
|book_id|              author|               title|          genre|price|publish_date|         description|
+-------+--------------------+--------------------+---------------+-----+------------+--------------------+
|  bk101|Gambardella, Matthew|XML Developer's G...|       Computer|44.95|  2000-10-01|An in-depth look ...|
|  bk102|          Ralls, Kim|       Midnight Rain|        Fantasy| 5.95|  2000-12-16|A former architec...|
|  bk103|         Corets, Eva|     Maeve Ascendant|        Fantasy| 5.95|  2000-11-17|After the collaps...|
|  bk104|         Corets, Eva|     Oberon's Legacy|        Fantasy| 5.95|  2001-03-10|In post-apocalyps...|
|  bk105|         Corets, Eva|  The Sundered Grail|        Fantasy| 5.95|  2001-09-10|The two daughters...|
|  bk106|    Randall, Cynthia|         Lover Birds|        Romance| 4.95|  2000-09-02|When Carla meets ...|
```

#### Step 4: Save DataFrame as csv files
Finally we can save the results as csv files. Spark provides rich set of destination formats, i.e. we can write to JSON, parquet, avro, or even to a table in a database.
```python
# write to csv
book_df.write.format("csv").mode("overwrite")\
    .save("./output")
```

### Conclusion
This is just one of the showcases of what Spark can help to simplify the data processing especially when dealing with large amount of data. Spark provides both high-level API (DataFrame / DataSet), and low-level API (RDD) which enables us with the flexibility to handle various types of data format. Spark also abstracts the  physical parallel computation on the cluster. We just need to focus our codes on the implementation of business logic.

Source codes are [here](https://github.com/q15928/python-snippets/blob/master/pyspark/parse-xml/xml-parse.py).


text:
# Sprint7lines v1.1
a Simple seven lines code game, made on Locomotive Basic for Amstrad CPC

2018 By Fabrizio Radica -<EMAIL_ADDRESS>

![sprint7lines](https://user-images.githubusercontent.com/1652242/46651489-83b9d500-cba0-11e8-86c1-4a83660342a4.PNG)

## Usage

Simply COPY/PASTE on your favourite CPC emulator (mine is JavaCPC)
Happy CODING!

license: MIT

branch_name: refs/heads/master

extension: md

license_type: permissive


text:
# OgrPackage

## Introduction

OgrPackage for Laravel provides a simple connection to Gravatar API to get all information you need to get from their service.

Once you have created a new Laravel application, you may install OgrPackage for Laravel using Composer:

`composer require okh/ogr-package`

You may publish this package config-file executing the next command:

`php artisan vendor:publish --tag=ogr-config`

`php artisan vendor:publish --tag=ogr-migrations`

Add next configuration to .env file

`GRAVATAR_API_HOST=`

`OGR_PACKAGE_FILE_PATH=`

`OGR_PACKAGE_FILE_NAME=`

## License
OgrPackage for Laravel is open-sourced software licensed under the [MIT license](LICENSE.md).

repository: KhOlga/ogr-package

repo_name: KhOlga/ogr-package


text:
# Josef
Josef Capek is the brother of Karel Capek. This Projekt is named after him.
Josef (this program) is a very basic FANUC KAREL document generator.
For the moment it reads in a karel file search for the routine key word and builds a documentation from the upper documentation of
this routine.
In furture more functions should be included. the current version only scans for the @return and @param tag.
I want to split the KAREL file into parts and then I want to write a perl module to work with each part of the KAREL and gernerat files using the perl Template2 module.

| TAG | Description |
| --- | --- |
| @file | The File name |
| @copyright | The copy right information |
| @todo | on todo task at a time. but this can used mutible time |
| @author | one author but this can used mutiple times |
| @date | date of the last change|
| @license | the license terms |
| @brief | one line description |
| @return | return value of the routine |
| @param | parameter for the routine. The syntax is TYPE DESCRIPTION|

repository: afeldman/Josef

repo_name: afeldman/Josef


text:
# fileUpload

@@@ div { .group-scala }

## Signature

@@signature [FileUploadDirectives.scala]($akka-http$/akka-http/src/main/scala/akka/http/scaladsl/server/directives/FileUploadDirectives.scala) { #fileUpload }

@@@

## Description

Simple access to the stream of bytes for a file uploaded as a multipart form together with metadata
about the upload.

If there is no field with the given name the request will be rejected. If there are multiple file parts
with the same name, the first one will be used and the subsequent ones ignored.

@@@ note
This directive will only upload one file with a given name. To upload multiple files with the same name
you should use the @ref[fileUploadAll](fileUploadAll.md#fileuploadall) directive, though all files will
be buffered to disk, even if there is only one.
@@@

## Example

Scala
:  @@snip [FileUploadDirectivesExamplesSpec.scala]($test$/scala/docs/http/scaladsl/server/directives/FileUploadDirectivesExamplesSpec.scala) { #fileUpload }

Java
:  @@snip [FileUploadDirectivesExamplesTest.java]($test$/java/docs/http/javadsl/server/directives/FileUploadDirectivesExamplesTest.java) { #fileUpload }

repository: yangbajing/akka-http

gha_license_id: NOASSERTION

repo_name: yangbajing/akka-http


text:
# ShipHullDesign

ShipHullDesign is a draft project, not final.
Using C# Splines and 3D capability

To run project:
- Download it
- Go to ShipHullDesign\bin\Release
- Run `HullDesign1.exe`
- Configuration Dialog will be opened
- Uncheck `Use DataBase` checkbox, if selected
- Go to: `Data Files->Select Folder`,
  Select `ShipHullDesign\SAVE_DATA_FILES`
  and press `OK` button. Application will be opened.
  Ignore "General" tab/view (TBD), and explore other tabs:
	* Secrions
	* Hull 3D
	* ShipHullSurface
	* Hull-Draft
	* Statica-Curves
	* Statica-Bonjean
	* Statica-Large Angles

repository: valerymo/ship-hull-design

branch_name: refs/heads/main

repo_name: valerymo/ship-hull-design


text:
# Jacobian calculation

```@meta
CurrentModule = NonlinearSolvers
```

Special mention of calcJacobianComplex.

One of the most involved parts of Newton's method is forming the Jacobian.
The NonlinearSolvers module contains the function [`physicsJac`](@ref) to
compute the Jacobian of the physics $\frac{\partial R(q)}{\partial q}$, where
$R$ is [`evalResidual`](@ref).  This function can be used by other methods
as a starting point for computing the residual of $g(R(q))$ described in
[ Newton's method](@ref) page.

```@docs
physicsJac
```

`physicsJac` calls one of several functions to compute the Jacobian.  Users
should not call these functions directly, they should use `physicsJac`.
There are two approaches to computing the Jacobian: using finite differences/complex step and calculating the entries of the matrix explicitly.
The latter approach is much faster than the former, however the former does
not require the physics module differentiate [`evalResidual`](@ref).

## Finite Differencing/Complex Step

The functions [`calcJacFD`](@ref) and [`calcJacobianComplex`](@ref) compute
the Jacobian as a dense matrix, one column at a time.  This is very slow
and only useful for debugging small cases.
A more efficient method is implemented in [`calcJacobianSparse`](@ref),
which uses a graph coloring approach to perturb the solution at several
nodes simultaneously.

```@docs
calcJacFD
calcJacobianComplex
calcJacobianSparse
applyPerturbation
assembleElement
calcJacCol
```

## Explicit Calculation

The most efficient method to compute the Jacobian is to explicitly compute
all its entries and assemble them into a matrix.
The [`evalJacobian`](@ref) function must be extended by each physics module
for this to work.
This function is passed all the same arguments as [`evalResidual`](@ref) plus an additional [`_AssembleElementData`](@ref) object which is used to assemble
the contribution of each element or interface into the matrix.

```@docs
_AssembleElementData
_AssembleElementData(::AbstractMatrix, ::Any, ::Any, ::Any, ::Any)
NullAssembleElementData
assembleElement(::_AssembleElementData, ::AbstractMesh, ::Integer, ::Array{Float64, 4})
assembleInterface
assembleSharedFace
assembleBoundary
```


text:
# 6.0.2 (2017-06-24)
* **Upgraded** to<EMAIL_ADDRESS>
# 6.0.1 (2017-03-23)

* **Upgraded** to eslint-plugin-ember-standard 0.0.22

# 6.0.0
## Breaking Changes
* **Converted** some warning rules to error rules (the `ember` and `ocd` ones that were recently introduced)
## Non-breaking Changes
* **Updated** `pr-bumper` and `travis` settings

# 5.3.2

* **Fixed** bug with fix functionality of `ember-standard/destructure` rule where a property could end up in the variable declarator twice.

# 5.3.1

* **Fixed** bug where `Ember.String` and `Ember.Object` were being destructured when they shouldn't be.

# 5.3.0

* **Added** warning to when using [object curly spacing](http://eslint.org/docs/rules/object-curly-spacing).

# 5.2.0

* **Fixed** `ocd/sort-imports` rule to separate internal and external modules.

# 5.1.0

* **Added** plugins [ember-standard](https://github.com/ciena-blueplanet/eslint-plugin-ember-standard) and [ocd](https://github.com/ciena-blueplanet/eslint-plugin-ocd) and configured their rules as warnings (will switch to errors in the next major release).

# 5.0.1
 * **Fixed** a typo in `README.md` (thanks @juwara0 😄)

# 5.0.0
* **Added** an additional rule: `no-unused-expressions` which will disallow `expect(foo).to.be.true` style assertions. See [this issue](https://github.com/chaijs/chai/issues/726) for more info.

# 4.0.1
 * **Updated** `eslint-plugin-mocha` to allow us to stop using the fork from `job13er`

# 4.0.0
 * **New Rule** `complexity` to enforce a maximum cyclomatic complexity of 5 for all functions.

# 3.0.0
## Breaking changes
 * **Switched** to<EMAIL_ADDRESS>
## New Rules

 * **Added** rule to error on a `describe.only` or `describeComponent.only` in tests
 * **Added** rule to warn on a `describe.skip` or `it.skip` (with `--fix` option 😄 )
 * **Added** rule to error on an `it('pending test')` with no test
 * **Added** rule to error on a global `it()`(not within a test suite)
 * **Added** rule to error if the `done` callback isn't called when it's defined.

# 2.1.0
* Added camelcase property for variables
* Added max-len property of 120 chars per line

repository: dafortin/eslint-config-frost-standard

repo_name: dafortin/eslint-config-frost-standard


text:
---
title: Halo "Choose the playlist" button
description: Create a Halo button to select a desired playlist
layout: pagetoc
---

This guide shows an example of how to control the BeoRemote Halo through macros. In this case, the Halo will be used to select a Deezer playlist from a predefined list and play it on a specific source. The macro also includes a multiroom setting in case the user wants to play the same music in multiple speakers.

We will use a BeoRemote Halo with the name “Halo” in an area called “My_Area” in a zone called “My_Zone”.

## Halo configuration
The Halo configuration page can be found in the Interfaces tab. Firstly, a button should be created in any page (if no pages exist one should be created too). After choosing a title, subtitle and icon, the behavior “Handle by custom macros” must be chosen. The button and page number should be remembered as they will be used later.
How to create a page and a button, as well as more information on behaviors and resources, can be found in the help document of the Halo.

## Macro configuration
The macro is triggered by the wheel or the button. When the wheel is turned, different playlists names (from a predefined list) should appear in the button title. Once you reach the one you want to play, you have to press the button for it to start. Supposing there is an area called “My_Area” with a zone “My_zone” that has a resource “Halo” of BeoRemote Halo type, we can create the macro.
The events for this macro would be:

When creating the event, the button to choose should be the one with the page and button number remembered before, this will automatically change to the button ID as shown in the image.

For this application, some virtual resources should also be created. In Resources->Virtual Resources create the following:
    • “Count”, of type INTEGER: This will be used for selecting the different playlists
    • “Source”, of type STRING: This will be used to save the selected playlist’s ID
    • “Fired”, of type BOOLEAN: While the macro is running, this variable will be TRUE. This is necessary to avoid the macro running multiple times at the same time and causing wrong values in the previous variables.

The code as LuaMacros begins as follows:

~~~lua
function(event, engine)
  ---- variables ------
  local HALO_ADDRESS = "My_Zone/My_Area/Halo remote/Beoremote Halo"
  local HALO_BUTTON_ADDRESS = "497f6eca-6276-4993-bfeb-000008101411"

  ---- Virtual resources
  local COUNT_ADDRESS = "My_Zone/My_Area/VARIABLE/count"
  local SOURCE_ADDRESS = "My_Zone/My_Area/VARIABLE/source"
  local FIRED_ADDRESS = "My_Zone/My_Area/VARIABLE/Fired"

  -- playlists
  local available_sources = {[1] = "TOP 100", [2] = "WINE & DINE", [3] = "FAVOURITES", [4] = "JAZZ"} -- Playlists' name for the button title
  local choose_source = {[1] = "1266971851", [2] = "10387252442", [3] = "253141911", [4] = "1914526462"} -- Playlists' ID
  local source_type = "deezer"
  local sources_quant = 4 -- number of available sources

  -- all the speakers you want to play in
  local speakers = {"My_Zone/My_Area/AV renderer/Cono"}
~~~

This section is for the user to define the resources’ address.
    • “speakers”: define the address of all speakers that should play the music you select. In the example, only one speaker named “Cono” is used. It’s necessary you define at least one speaker. If there are more, the multiroom function will activate so that all the speakers reproduce the same music.
    • “available_sources”:  define the playlists’ title in order to set it in the button title.
    • “choose_source”: define the playlists’ ID (corresponding to the title defined in “available_sources”)
    • “source_type”: source from where the playlists are selected (“deezer”, etc).
    • “sources_quant”: number of playlists you can choose from

Once all the variables and resources addresses are defined, you can go on to the actual macro instructions.

The code:
~~~lua
  local fired = engine.query(FIRED_ADDRESS)[1].get_boolean("VALUE")
  if not fired then
    engine.fire(FIRED_ADDRESS.."/SET?VALUE=true")
    -- Macro activated by wheel
    if event.parameters()["OFFSET"] then
     local count = engine.query(COUNT_ADDRESS)[1].get_number("VALUE")

     if tonumber(event.parameters()["OFFSET"]) >= 1 then
        if count == sources_quant then
          count = 1
        else
          count = count + 1
        end
     else
        if count == 1 then
          count = sources_quant
        else
          count = count - 1
        end
     end
     engine.fire(COUNT_ADDRESS.."/SET?VALUE="..tostring(count))
     engine.fire(HALO_ADDRESS.."/SET_TITLE?BUTTON="..HALO_BUTTON_ADDRESS.."&TITLE="..available_sources[count])
     engine.fire(SOURCE_ADDRESS.."/SET?VALUE="..choose_source[count])

    -- Macro activated by button
    else
      local multiroom_source =<EMAIL_ADDRESS>      local content = engine.query(SOURCE_ADDRESS)[1].get_string("VALUE")

      local bs1_state = engine.query(speakers[1])[1].get_string("state")
      local bs1_source = engine.query(speakers[1])[1].get_string("sourceUniqueId")

      engine.fire(speakers[1].."/Playqueue add Deezer playlist?Play now=true&Playlist id="..content)

      engine.delay(5,0)
      for i = 1,#speakers do
        state = engine.query(speakers[i])[1].get_string("state")
        if state ~= "Play" then
          engine.fire(speakers[i].."/Select source by id?sourceUniqueId="..multiroom_source)
        end
      end
    end
    engine.fire(FIRED_ADDRESS.."/SET?VALUE=false")
  end
end
~~~

We begin by checking if the macro is already running to avoid having problems. If the macro is not running, we continue with the instructions (and set de “fired” variable to “true”).

The code is divided into two parts: the first part has the instructions it has to follow if the macro was triggered by the wheel. The second one, are the instructions for when the macro is triggered by the button. This is checked in the “if” line that asks if the event parameters include “OFFSET”, which means that the wheel was turned.

We will start with the wheel section. The parameter “OFFSET” of the wheel is positive if the wheel is turning right and negative if it’s turning left. If the wheel is turning right, we want to go forward in the playlist list. If the wheel is turning left, we want to go backwards. That is set with the “count” variable, which increments its number if the wheel is turned clockwise and decreases if it is turned left.

Next we have a small correction of the number as we only want numbers form 1 to “sources_quant”, this means that after “sources_quant” the number will turn to 1 and so on. If we move left, after 1 the number will turn to “sources_quant”. After this, we set the count variable to the new number, the button title to the playslist’s name and define the sources address.

It’s important you notice that the playlist will not start playing by only turning the wheel because there is no “engine.fire()” command in this section.

If the macro is triggered by the button, we set the multiroom source, which content is going to be played (defined when the wheel was turned) and the state and source ID of our main speaker. After that, we fire the command that makes the playlist start playing in the main speaker. Finally, if multiple speakers were defined, the playlist will start playing with a five seconds delay in all the speakers that do not have music already playing.

Before leaving the macro, we change the “fired” variable to false and leave it ready for the next time you will use it.

This was just an example of how to select a playlist from a predefined list of deezer playlists  with a macro. You can select different playlists just by changing the ID in “choose_sources” and the button’s title in “available_sources”. You will also have to change the “source_type” and “source_quant” in case you have more (or less) than four choices.

gha_language: HTML


text:
### Checking Whether you have an Anagram or not

An anagram is a word or phrase that's formed by rearranging the letters of another word or phrase.

Anagram Checker will tell if the pair of inputs constitutes an anagram.

repository: ZoranPandovski/al-go-rithms

path: /math/Anagram Checker/README.md

gha_language: Jupyter Notebook

repo_name: ZoranPandovski/al-go-rithms


text:
# CCF Registration Competition

In collaboration with the [BRAIN Inititative Cell Census Network](biccn.org) (BICCN),
the [Brain Imaging Library](https://www.brainimagelibrary.org/) (BIL),
the [Penn Image Computing and Science Lab](http://picsl.upenn.edu/) (PICSL),
we are running the first [Allen Institute for Brain Science](https://portal.brain-map.org/)
CCF registration competition!

A brief description of the competition motivation, format, and next steps is below
Motivation

The [Allen Mouse Common Coordinate Framework](https://community.brain-map.org/t/allen-mouse-ccf-accessing-and-using-related-data-and-tools/359) (CCF) defines anatomical locations within an annotated population averaged mouse brain. This resource is already valuable to the field, providing a 3D reference framework for the mouse brain. Enhanced impact is possible if new 3D datasets can be routinely aligned into the CCF space. This enables comparison of results across experiments with a common and objective reference for mouse brain anatomy. Large-scale resources can be built systematically by leveraging automated alignment into a common spatial framework.

Achieving this vision requires robust alignment of new image data into the CCF space, regardless of acquisition modality. The field has produced many options for this task, but unbiased comparison of these techniques does not exist. To benchmark various approaches, and to encourage researchers to share available resources, we propose the Allen Institute for Brain Science organize a competition between CCF alignment tools sourced from the global neuroscience and image processing community.

## Competition Format

### About the data

The Allen Institute for Brain Science will provide the following image datasets:

#### Optical projection tomography datasets from the Neuropixels pipeline

<img src="images/opt_pipeline_overview.png" width="650" />

The Allen Institute has been using Neuropixels probes, a new type of silicon probe, to record spiking activity across the mouse visual system. In a typical experiment, we use 6 probes to simultaneously monitor hundreds of neurons, each of which needs to be registered to a specific point in the CCF. Our Neuropixels dataset is described at [brain-map.org](https://portal.brain-map.org/explore/circuits/visual-coding-neuropixels) and in [this preprint](https://www.biorxiv.org/content/10.1101/805010v1).

To determine the location of our recordings, we coat the probes in a fluorescent dye. After the experiment, we clear the brain and image it using the [AIBSOPT](https://github.com/alleninstitute/aibsopt) optical projection tomography system. This method has lower resolution than TissueCyte (around 10 microns), but provides isotropic sampling in all 3 dimensions. This is helpful for identifying the probe tracks, which are only 70 microns wide.

Each brain is currently being registered to the CCF through manual keypoint selection. This method works reasonably well, but is time consuming and subject to operator bias. An automated registration method that meets or exceeds human accuracy would greatly improve our overall workflow.

Some of the challenges of implementing a fully automated registration method for OPT volumes include:
* Unique tissue contrast profile for OPT, which means algorithms developed for serial 2-photon tomography do not automatically transfer
* Air bubbles in the sample create dark spots that can obscure important structures
* Probe tracks cause tissue damage that is visible in the transmitted light volume
* Common computed tomography artifacts, such as beam hardening, must be distinguished from the underlying anatomy

#### fMOST datasets from the whole-brain morphology project

(To-do: add details - Julie)

### Dataset access (through BIL)

(To-do: BIL to setup site in BIL )
(To-do: Rusty to upload data)
(To-do: instruction on how to download - BIL)

The data is in the following format:

(To-do: PICSL team to define format and convert the data)

### How will the competition be run?

For each  dataset, a set of landmarks covering the whole extent of the brain will be indentified by expert anatomists. Type of landmarks include:
* Pairs of corresponding points between the image dataset and the CCF
* Points that are within in specific structures in the CCF
* Points that are on the surface between two structures in the CCF

(To-do: Penn team to define the landmark format)
(To-do: Lydia/Julie to figure out fMOST data)
(To-do: Rusty/Josh to format the OPT data)

Example landmarks will be provided to participants as reference. The rest are withheld and used for scoring the registration quality.

To submit a result, a participant will
- start a pull request
- create a new submission page using the template
- provide information about their team and methods
- provide links to the registration code (source code or product page)
- For each imageset:
    - Scripts and parameters that was used to produce
    - Upload the deformation field that maps a image point to the CCF in the format described below
    - Add location of the deformation field
- Submit pull request

Once we received the pull request, we will compute registration score and post in on the scoreboard.

### Deformation field format

(To-do: Penn team to define the deformation field format)

### Registration scoring

(To-do: Penn team to code registration scoring method )
(To-do: Penn / BIL team to setup how to run the scoring at BIL)
(To-do: Rusty to update the score board)

Competition start is tentatively scheduled for summer 2020. We will release details regarding datasets, scoring, and entrance details as they become available.

## Save the date!

We will have a workshop at the [Neuroinformatics 2020 Congress in Seattle](https://neuroinformatics.incf.org/) in Aug 17-18, 2020


text:
# ICM20948_WE
An Arduino library for the ICM20948 9-axis accelerometer, gyroscope and magnetometer. It contains many example sketches with lots of comments to make it easy to use.

I have not implemented DMP features and most probably I won't do that in future. That would exceed the time I can invest.

You can find a documentation in my blog:

https://wolles-elektronikkiste.de/icm-20948-9-achsensensor-teil-i (German)

https://wolles-elektronikkiste.de/en/icm-20948-9-axis-sensor-part-i (English)

If you find bugs please inform me. If you like the library it would be great if you could give it a star.

If you are not familiar with the ICM20948 I recommend to work through the example sketches.

Known issue:
* If you upload sketches, the magnetometer occasionally does not respond. If you disconnect from power and then reconnect it will work. I experienced the issue only after uploads.


text:
## Preface

Hosting [Creating a Driftomaniac Software](https://jumanamp.github.io/driftomaniac/) blogs on projects done as part of [Self Driving Car Nanodegree](https://www.udacity.com/course/intro-to-self-driving-cars--nd113?utm_source=google&utm_medium=cpc&utm_campaign=google_search_alpha_nanodegree&gclid=Cj0KCQiA2Y_UBRCGARIsALglqQ1E8Esg4PaFs7C1_PvfF0fI5uLucr2F624vn5158WzhsDfL5EiJjggaAnm_EALw_wcB). Moreover, I would like to add additional information gathered from both academia and industry while following the course and executing the projects.

I hope you enjoy!

## Author

* Jumana Mundichipparakkal *"jumanamp"*

repository: jumanamp/driftomaniac

repo_name: jumanamp/driftomaniac


text:
# EngComputaria Logo

Este é o repositório que armazena o arquivo xcf da logo do podcast Engenhação da Computaria feito com o GIMP.

# Imagens

| Sombra+Trilhas | Sombra | Sombra+Trilhas+Borda Arredondada | Sombra+Borda Arredondada |
|:----:|:-------------------:|:----:|:-------------------:|
| ![](images/engcomputaria_flat_full_tracks.png) | ![](images/engcomputaria_flat_full.png) | ![](images/engcomputaria_flat_tracks.png) | ![](images/engcomputaria_flat.png) |

| Trilhas | Normal | Trilhas+Borda Arredondada | Borda Arredondada |
|:----:|:-------------------:|:----:|:-------------------:|
| ![](images/engcomputaria_full_tracks.png) | ![](images/engcomputaria_full.png) | ![](images/engcomputaria_tracks.png) | ![](images/engcomputaria.png) |

| Favicon |
|:-------:|
|   |

# Team

|  |
|:---------------------------------------------------------------------------------:|
| [Edimar Calebe Castanho (Calebe94)](https://github.com/Calebe94)                  |

# License

All software is covered under [MIT License](https://opensource.org/licenses/MIT).

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

 IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

repository: engcomputaria/logo

repo_name: engcomputaria/logo


text:
# The Story: The 'Historical Recipes' Project

## Preamble

I think it is best to have a relatable story as a backdrop for the workshop. In the Software Carpentry Git-lesson, [Wolfman, Dracula, and the Mummie explore where to build a Moonbase](https://swcarpentry.github.io/git-novice/03-create/index.html). I find this story a bit silly, which is not bad, but I think its usefulness is questionable, and learners cannot relate to the characters or the project. I want to reduce the 'irritation effect' of a made-up story while coming up with a more inclusive ensemble: researchers of all stages from different disciplines. If you think you can improve my story, I am happy to hear your suggestions! You can create an [Issue](https://github.com/arockenberger/github_collab_workshop/issues) for that.

## Historical Recipes Research Project

A small group of researchers share a common interest: They are fascinated by historical recipes and recreating all kinds of dishes that people in the past widely enjoyed.

The researchers have different backgrounds, a genuinely interdisciplinary group! There's a historian, a linguist, a cultural studies researcher, an anthropologist, a botanist, and a chemist. They have contacted a junior web developer to join their project and got some funding for hiring a student assistant to help with technical stuff. Inspired by the Open Research movement, they bravely decided to share their project from the initial idea throughout the research process and publish their results as openly as possible. Since they are not working with sensitive or personal data and have no commercial interests, they chose GitHub as a platform for collaboration.

They plan to work mostly remotely. The project team elects a project leader (the Historian, an amateur chef) whose task is to coordinate the project and facilitate collaboration. The researchers want the project to be dynamic: some members may want to take a break from working on the project, new members might want to join instead. They want to involve students to contribute as part of their course work. They want to allow both the scientific community and anyone else interested in historical recipes to contribute.

The project leader is hesitant to embark on such a task since she is used to working with word processing software and spreadsheets. She shares her work and writes with others using commercial cloud services and local copies of files on USB sticks. She knows that tracking who did what, when, and where she saved the latest file version quickly becomes messy. Thus, she wants to give privileges to some collaborators - and review quickly and effectively others' contributions. She also wants to have an option for collecting suggestions, questions and feedback.

For the finished project, the team wants to make the project data findable and accessible. The group decides that assigning a DOI - a Digital Object Identifier - would be best. Then they could add the project DOI to the papers they plan on publishing and add the project efficiently to their lists of publications.

## Point of Departure for the GitHub Workshop

The project is in its early stage: The members collect historical recipes from different sources and share them with the team. You are part of the *Historical Recipes* project team. You found an interesting recipe in an old Norwegian cookbook and try to create files with the ingredients and instructions, so your colleagues can review them.

## The Workshop Example Recipe

### Source

**Scrambled eggs (Æggerøre)**

* Taken from the Norwegian cookbook "Lærebog i de forskjellige Grene af Huusholdningen" (Manual for the different tasks of the household), by Hanna Winsnes, Christiania 1845. Available online and open access via the digital scholarly edition on [Bokselskap.no](https://www.bokselskap.no/boker/kogebog/tittelside) and as a [digital facsimile](https://urn.nb.no/URN:NBN:no-nb_digibok_2009111210001) from the National Library of Norway
* Read the [full text of the recipe](https://www.bokselskap.no/boker/kogebog/sild)

### Original wording in C19th Dano-Norwegian

**Æggerøre**
12 Æg med Hvider slaaes, deri kommes 6 Skeer sød Fløde, og lidt fiinhakket Græsløg. En Skee Smør kommes i Bunden af en lille Gryde, og Æggene heldes i; den sættes paa svage Gløder, og fra det begynder at blive varmt, maa man bestandig skrabe det, der lægger sig ved Bunden, fra med en Træskee. Naar man finder den passende fast, øses den op; den taaler ikke at blive staaende i Gryden. Man pleier at regne 2 Æg til hver Person.

### Translation into modern English

**Scrambled Eggs**

* Scramble 12 eggs (with the egg white) and whisk in 6 tablespoons of sweet cream and a few minced chives.
* Melt a tablespoon of butter in a small skillet and pour the eggs in
* Put on medium-low heat
* When the eggs start to stick to the skillet, scrape it loose with a wooden spoon
* Repeat until the desired consistency is reached
* Serve immediately
* Use two eggs per person

### Text for the Ingredients.txt file

Ingredients:

* 12 eggs
* 6 tbsp sweet cream
* 1 tbsp butter
* a bunch of chives

* skillet
* wooden spoon

### Text for the Instructions.txt file

* Scramble 12 eggs (with the egg white) and whisk in 6 tablespoons of sweet cream and a few minced chives.
* Melt a tablespoon of butter in a small skillet and pour the eggs in
* Put on medium-low heat
* When the eggs start to stick to the skillet, scrape it loose with a wooden spoon
* Repeat until the desired consistency is reached
* Serve immediately
* Use two eggs per person

license: Unlicense

gha_license_id: Unlicense


text:
---
title: "Sylvania 71831 Homebridge/HomeKit integration"
description: "Add HomeKit support to your Sylvania 71831, using Homebridge, Zigbee2MQTT and homebridge-z2m."
---
<!---
This file has been GENERATED using src/docgen/docgen.ts
DO NOT EDIT THIS FILE MANUALLY!
-->
# Sylvania 71831
> Smart Home adjustable white A19 LED bulb

# Services and characteristics
The following HomeKit Services and Characteristics are exposed by
the Sylvania 71831

* [Lightbulb](../../light.md)
  * Brightness
  * ColorTemperature
  * On

# Related
* [Other devices from Sylvania](../index.md#sylvania)
* [Zigbee2MQTT documentation for this device](https://www.zigbee2mqtt.io/devices/71831.html)


text:
# Proposal: Emit events from Jib Core

The tracking issue is at [#714](https://github.com/GoogleContainerTools/jib/issues/714).

## Motivation

Currently, Jib logs various log messages via injecting a [`JibLogger`](https://github.com/GoogleContainerTools/jib/blob/02f7f41874223e1e6acf2a40648b5b3695877397/jib-core/src/main/java/com/google/cloud/tools/jib/JibLogger.java) interface into the execution steps in the `builder` package. However, the data is not structured. The user (of Jib Core) needs to parse log messages in order to obtain useful information. The log messages are also catered towards the `jib-maven/gradle-plugin` execution output.

## Solution

Emit events with structured information.

## Goals

- Flexible to support different event payloads
- Extensible with different event types
- Minimal overhead if not used

## Proposal

An `EventHandlers` class holds handlers to pass into the execution steps. This is used by `ExecutionContext`.

```java
class EventHandlers {

  // Handles `E` event class to with `eventConsumer`.
  <E extends JibEvent> add(JibEventType<E> eventType, Consumer<E> eventConsumer);

  // Handles all events.
  add(Consumer<JibEvent> eventConsumer);
}
```

Emitted events will be matched to handlers by their exact type. `JibEvent`s should **not** inherit from each other. `JibEventType` defines constants for all the possible event types to add handlers for.

An example usage could look like this:

```java
// In Jib Core
class PushingBlobEvent implements JibEvent {

  DescriptorDigest getDigest();
  URL getUploadLocation();
}

// Called by user
ExecutionContext executionContext =
    ExecutionContext.newContext()
                    .addEventHandler(JibEventType.PUSHING_BLOB, event -> {
                      // Do some processing on event, like:
                      System.out.println(
                          "Pushing blob " + event.getDigest() + " to " +
                              event.getUploadLocation());
                    })
                    .addEventHandler(allJibEvents -> {
                      System.out.println("Some event happened");
                    })
                    // Some class that has pre-defined handlers that print log messages.
                    .addEventHandlers(new LoggingEventHandlers(jibLogger));
Jib.from(...)
   ...
   .containerize(
       Containerizers.withExecutionContext(executionContext)
                     .to(RegistryImage.named(targetImage)));
```

repository: glaforge/jib

repo_name: glaforge/jib


text:
# Usage

## Run dev server

### Mobile device

Android:

```sh
$ yarn android
```

iOS (on macOS only):

```sh
$ yarn ios
```

### Web

```sh
$ yarn web
```

That will open two browser tabs:

- http://localhost:19002/ - Expo debugger
- http://localhost:19006/ - Your web app

In Expo, I clicked the Android emulator option and got this prompt:

```
Couldn't start project on Android: No Android connected device found, and no emulators could be started automatically.
Please connect a device or create an emulator (https://docs.expo.io/workflow/android-studio-emulator).
Then follow the instructions here to enable USB debugging:
https://developer.android.com/studio/run/device.html#developer-device-options. If you are using Genymotion go to Settings -> ADB, select "Use custom Android SDK tools", and point it at your Android SDK directory.
```

If you want to use the

```sh
$ yarn start
```

> Make sure your app is running in the simulator or on a phone connected via USB.

## Run tests

Warning - the default app is not setup right so Jest gets a syntax error.

```sh
$ yarn test
```

repository: MichaelCurrin/react-native-quickstart

repo_name: MichaelCurrin/react-native-quickstart


text:
# Lambdapse forked
Lambdapse (aka dahon bot) is a Hentai and Discord Activity Bot

Current Features:
- Hentai
- Discord Activities

Use bot here: [Join Bot](https://bit.ly/dahonbot)
Special Thanks to:
lamdagit101 (and its contributors)

gha_license_id: MIT


text:
# EDA-Data-Tutorial
Exploratory Data Analysis Tutorial with code samples in Python

## Project Overview

This is a tutorial on how to do simple exploratory data analysis with Python. The dataset that has been used in this project is called Student Academics Performance Data Set and I took it from UCI Machine Learning Repository, here is the citation for it
```
Hussain S. (2018). Student Academics Performance Data Set [https://archive.ics.uci.edu/ml/datasets/Student+Academics+Performance]. Dibrugarh University, Dibrugarh, Assam, India.
```
and here is another citation from the same dataset
```
Hussain S, Dahan N.A, Ba-Alwi F.M, Ribata N. Educational Data Mining and Analysis of Studentsâ€™ Academic Performance Using WEKA. Indonesian Journal of Electrical Engineering and Computer Science. 2018; Vol. 9, No. 2. February. pp. 447~459
```
## Contents

* Convert & Load the dataset
* Get an Overview about the data e.g. samples, columns, mean and others
* data cleaning (if required), check for missing values or delete unwated columns
* Factorizing the data using Pandas
* Data Visualization, show correlation matrix or factorplot between multiple columns
* Train & Test split using SKlearn (Not Implemented Yet)
* Running bunch of ML models and observe the results (Not Implemented Yet)

Furthermore, I will keep updating this repo, when I've time, however, if you have any idea and would like to apply it, I will be happy to support that.

### Required Packages

I strongly encourage you to install Anaconda, if you want to work with Data Science, since Anaconda comes with a set of powerful Python packages for DS. You can download it from here - [Anaconda Distribution](https://www.anaconda.com/distribution/)

However, if you want to download packages individually you will have to install the following packages for this tutorial
* numpy
* pandas
* seaborn
* matplotlib

#### Importing packages
```
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
```

### Reading & Manipulating the Data

#### Reading CSV as a DataFrame
```
dataframe = pd.read_csv(dataPath, sep=',', header=0)
```
#### Printing first few rows
```
print('#### Header ####')
print(dataframe.head())

#### Header ####
  ge   cst   tnp   twp iap   esp arr         ms ls    as   ...     fq  mq        fo         mo       nf       sh       ss   me       tt      atd
0  F     G  Good  Good  Vg  Good   Y  Unmarried  V  Paid   ...     Um  10    Farmer  Housewife    Large     Poor     Govt  Asm    Small     Good
1  M   OBC    Vg    Vg  Vg    Vg   N  Unmarried  V  Paid   ...        Service  Housewife  Large  Average  Govt  Eng  Small  Good
freq     72   57    59    65   63    54   78        131  ...        38        115     58       59    91   62     78    56

[4 rows x 22 columns]

```

#### After getting info about the dataset, we can now perform some actions. First, lets drop the ms column since it has single value, we can do that by this line of code
```
data.drop('ms', axis=1, inplace=True)
```

#### Renaming Columns
This dataset originally provided with acronyms, you can find each column real name and how it calculated in the cited paper above, however, I will rename some columns to their original names for better understaing in visualization. SH column stands for 'Study Hours' and esp stands for 'End Semester Percentage'. This can be done via 'rename' function as follows
```
data = data.rename(columns={'sh': 'Study Hours', 'esp' : 'End Semester Percentage'})
```

### Data Visualization

#### Coefficient Correlation
First to get the big pitcure about the data you may visulaized it based on Coefficient Correlation, to do that seaborn packages provide us with heatmap
```
# get the correlation from the dataframe, then plug it to heatmap function and show it
corr = data.corr()
sns.heatmap(corr, annot=True, cmap=plt.cm.Reds)
plt.show()
```
output
![heatmap](https://raw.githubusercontent.com/arakhia/EDA-Data-Tutorial/master/images/heatmap.png)

#### Single Column Count
Here we show the count for each value in the 'Study Hours' column. Note: Study Hours values comes from this
``` >= 6 hours Good  >= 4 hours Average < 2 hours Poor ```

```
sns.countplot(x='Study Hours', data=data, hue='Study Hours')
```
output

![CounterPlot](https://raw.githubusercontent.com/arakhia/EDA-Data-Tutorial/master/images/counter_plot_sh.png)

#### Factor Count
With factorplot we can show the relationship between numerical and categorical value. Here I will show relation between [Study Hours] and [End Semester Percentage]. Percentage is calculated based on the following
```
if percentage >=80 then Best
If percentage >= 60 but less than 80 then Very Good
If percentage >= 45 but less than 60 then Good
If Percentage >= 30 but less than 45 then Pass
If Percentage < 30 then Fail
```
factorplot usage
```
sns.factorplot(x='End Semester Percentage', col='Study Hours', kind='count', data=data)
```
output

![FactorPlot](https://raw.githubusercontent.com/arakhia/EDA-Data-Tutorial/master/images/factor_plot_esp.png)

This plot shows relation in numbers between Study Hours & Final Percentage. If you go back to Coefficient Correlation you will find that Semester Percentage & Study Hours don't have high correlation in the first place. However, there is somthing interesting, the heighest grade ("Best") has strong relation with the time students spend on study. When the time Good the percentage was the Best, while nobody got over 80% (Best grade) when they spent less than 2 hour studying.

## Extra: How to get quick report about your Data
If you notice the majority of our previous work is about getting general overview from the dataset, for that I found a beautiful Python Package called [Pandas Profiling](https://github.com/pandas-profiling/pandas-profiling), the main advantage from it is to give an HTML report for your dataset which contain quick analysis. I have generated an HTML report for this dataset, please click on the image below to access it.

[![HTMLReport](https://raw.githubusercontent.com/arakhia/EDA-Data-Tutorial/master/images/html_report.png)](https://arakhia.github.io/resources/student-performance_report.html)

## License
For the dataset, please go to the dataset source "mentioned above" and check their license and/or requirements.
Regarding this project, it's under the MIT License.

repository: sushma-projects/EDA-Data-Tutorial

repo_name: sushma-projects/EDA-Data-Tutorial


text:
---
type: "tv-episode"
title: "TGI Kubernetes 020: Argo workflow system"
description: "TGI Kubernetes 020: Argo workflow system"
episode: "0020"
aliases: ["/0020"]
publishdate: "2020-08-10"
date: "2018-01-05T22:39:03Z"
episode_image: "/images/tv/episodes/tgik/0020.jpg"
images: ["/images/tv/episodes/tgik/0020.jpg"]
youtube: "M_rxPPLG8pU"
---

Come hang out with Joe Beda as he does a bit of hands on exploration of Kubernetes and related topics. Some of this will be Joe talking about the things he knows well. Some of this will be Joe exploring something new with the audience. Ask questions, comment and help decide where things go.

This week we will take a look at the Argo project.  This is a Kubernetes native declarative workflow system driven by CRDs.

Links:
* Argo github: https://github.com/argoproj/argo
* Meltdown/Spectre: https://meltdownattack.com/, https://googleprojectzero.blogspot.com/
* Brigade: https://github.com/azure/brigade
* Matt's post on Brigade/YAML: http://technosophos.com/2018/01/04/why-brigade-doesn-t-do-yaml.html
* Fission workflow: http://fission.io/workflows/
* AWS StepFunctions: https://aws.amazon.com/step-functions/
* kubeflow: https://github.com/google/kubeflow

repository: Weimankow/tanzu-dev-portal

repo_name: Weimankow/tanzu-dev-portal


text:
Alexey Radul
October 2, 2015

One may be interested in the stability of results from a probabilistic
program.

In the case of BayesDB, a typical program consists of analyzing some
data table under some population model for some amount of computation,
and then running some query, and possibly inspecting some consequence
of the result.  We are thus interested in stability under
- changing the seed entropy of this process
- changing the number of models constructed
- changing the amount of analysis performed

We may in general be interested in the stability of arbitrarily
complex objects, but the easy thing to start with is the independent
stability of a collection of boolean or 1-D numerical probes.

The baseline paradigm is therefore this:
- For each desired seed entropy
  - For each desired number of models to test
    - For each desired number of iterations of analysis
      - Perform that analysis
      - For each probe
        - Evaluate it on the resulting bdb
- And aggregate the results.

We can save work off the above in two ways, without losing any
generality of probes:
- It suffices to do just the maximum desired amount of analysis, and
  save checkpoints.
- It suffices to analyze just the maximum desired number of models,
  and evaluate the probes on subselections to assess stability with
  fewer models.

We can also save some complexity (but not computation) by noting that
analyzing k*n models with one seed is equivalent to analyzing n models
k times with different seeds (because the chains are actually
independent -- resampling, component swapping, or any such nonsense
would invalidate this observation).

Architecture
============

We break stability assessment into four conceptual phases, separated
by on-disk files:
- Analysis, which produces a collection of .bdb files with varying
  amounts of analysis done (on the same number of models, over the
  same dataset);
- Probing, which consumes such a collection of .bdb files and
  produces a file of serialized probe results;
- Visualization, which consumes such a file of probe results and
  draws plots that give visual indications of stability; and
- [Todo] Assessment, which consumes a file of probe results and
  produces a decision: "stable enough" or "not stable enough".

Why break into stages?  Because doing so gives room to memoize and
distribute the analysis and/or probe phases, for
- application of more compute
- iteration on downstream code

Why separate visualization and assessment?  Because assessment is what
you want for an automated build, and visualization is what you want
for developing the assessment's decision rules and for debugging build
failures.

I considered the alternative of fusing the analysis and probing phases
into one program.  My considerations and rationale for choosing to
separate are:
- Pro: can iterate on probe set without redoing analysis
- Pro: can distribute analysis separately from probing
- Con: .bdbs are actually quite big: 12MB per 60-model-satellites file
  makes 360MB per 30-point analysis series makes 36GB for 100 per-seed
  replications of a 30-point series.
  - Offset: sqlite's vacuum command helps, around 4x
  - Offset: zlib seems to compress vacuumed bdbs around 5x
- Con: The probes may take a long time to run too (either b/c of
  loading the files or b/c of compute cost per probe)
  - Offset: can still parallelize and distribute probing if needed
- Pro: The analysis-checkpointing program can be refactored relatively
  easily to fuse probing into it and avoid saving all the .bdbs.

Implementation on Satellites
============================

The analysis phase is carried out by the program
`examples/satellites/build_bdbs.py`
- This constructs a directory with a bunch of bdb files.
- The files are named according to a predictable pattern, and
- Metadata about them is saved in the directory as well.
- The program is configured by editing global variables in its
  source code.

The probing phase is carried out by the program
`examples/satellites/probe.py`
- The probe set and configuration is defined in the source of the
  program.

The visualization phase is carried out by the program
`examples/satellites/visualize.py`
- The configuration is defined in the source of the program.
- Visualization content is determined entirely by the probe results.

Generic probing and visualization machinery is available in the
modules `bdbcontrib.experiments.probe.py` and
`bdbcontrib.experiments.visualization.py`.

Possible Extensions
===================

The behavior of probes that are directly computed as Monte-Carlo
estimates over the models can be predicted using the Central Limit
Theorem.

To wit, suppose some probe F : bdb -> R is actually computed as the
average over all models in the bdb of an underlying f : model -> R.
Then, if f has finite mean and variance, the CLT predicts that as the
number of models increases, the distribution on F will become a
Gaussian, whose mean and variance can be predicted from the mean and
variance of f.  Further, if f has finite skew, the Berry-Esseen
theorem predicts the maximum Kolmogorov-Smirnoff distance of F from
the predicted Gaussian, in terms of the skew of f.

The mean, variance, and skew of f can be estimated from (the
computation performed during) a single evaluation of F on all
available models.  If the assumptions are believed, the stability of F
for any number of models can then be predicted without requiring
replications.
- In effect, we save n_replications work.
- If we are willing to extrapolate for more models than the original
  experiment, we save an unbounded amount of work.
- Assumption: The mean, variance, and skew of the sample are good
  estimates of the true mean, variance, and skew of f.

This could be implemented directly by adding a 'monte-carlo' result
type to the aggregation framework, which would maintain all the
desired statistics, and performing the above prediction in the
visualization or assessment phases.

This can also in principle be extended to predicting the behavior of
probes that evaluate a function G of several monte-carlo quantities
F_i.  If we either assume the underlying f_i are independent, or if we
use a multivariate version of the CLT, we can propagate the predicted
distributions on F_i through the computation of G to predict a
distribution on G.
- Again saving n_replications (or more) work.
- Doing this in general, however, requires a little probabilistic
  programming language for propagating Gaussian distributions through
  probe computations.
- Option: Implement said language by forward sampling.  That is, use
  the Gaussian approximation to generate many more inputs to the probe
  than we can afford with the underlying computation
  - Plus: Can study the probe arbitrarily finely by sampling repeatedly
  - Plus: Can project to higher true effort by lowering the variance
    of the Gaussians (but uncertainty about true means remains)
  - Minus: An exact analysis of the probe could be faster (by saving
    probe computation iterates) (but this is irrelevant if analysis
    time dominates probing time)
  - Minus: An exact analysis may be better able to use the
    Berry-Esseen theorem to bound the effect of non-Gaussian-ness
    - But there are still things one can say, e.g. refusing to do it
      if the projected K-S stat is too high.

The same machinery can also be used in BayesDB, in at least two ways:
- Predicting for our users how variable certain query results may be,
  at no additional work.
  - This applies to all queries that have numerical answers, that is,
    everything except simulate and predict_confidence.
- Teaching downstream functions to have a notion of confidence.
  - e.g. an ORDER BY WITH CONFIDENCE variant that, when the ordering
    column is an estimated property like PREDICTIVE PROBABILITY, does
    the Gaussian mass comparison and only puts one thing above another
    if the probability of them being ordered that way is above the
    confidence.

Note: There may also be a version of this whole story for
likelihood-weighted monte carlo estimation.
- c.f. Pareto-stabilized importance sampling

repository: probcomp/bdbcontrib

gha_language: Python

repo_name: probcomp/bdbcontrib


text:
# Melilotus sulcatus var. segetalis (Brot.) Rouy VARIETY

#### Status
SYNONYM

#### According to
GRIN Taxonomy for Plants

#### Published in
Fl. France 5:61. 1899

#### Original name
null

### Remarks
null

repository: mdoering/backbone

path: /life/Plantae/Magnoliophyta/Magnoliopsida/Fabales/Fabaceae/Melilotus/Melilotus segetalis/ Syn. Melilotus sulcatus segetalis/README.md

repo_name: mdoering/backbone


text:
# Vigenère Cipher
A simple text encryption program writen in Python 3 using the [Vigenère Cipher](https://en.wikipedia.org/wiki/Vigen%C3%A8re_cipher)<br>
**Curenty only Greek Letters are supported**


text:
---
layout: default
---

# 1996-09-12 - Re: Nonsense, absolute nonsense... [Fwd: HipCrime and Spam]

## Header Data

From: Andy Dustman<EMAIL_ADDRESS>To<EMAIL_ADDRESS>Message Hash: 0c5014ba522dad0589444e8170897ae8a8d3871eb309b9e45a66bc8e5575e940

Message ID<EMAIL_ADDRESS>Reply To<EMAIL_ADDRESS>UTC Datetime: 1996-09-12 04:11:24 UTC

Raw Date: Thu, 12 Sep 1996 12:11:24 +0800

## Raw message

```
{% raw  %}From: Andy Dustman<EMAIL_ADDRESS>Date: Thu, 12 Sep 1996 12:11:24 +0800
To<EMAIL_ADDRESS>Subject: Re: Nonsense, absolute nonsense... [Fwd: HipCrime and Spam]
In-Reply-To<EMAIL_ADDRESS>Message-ID<EMAIL_ADDRESS>MIME-Version: 1.0
Content-Type: text/plain

-----BEGIN PGP SIGNED MESSAGE-----

On Wed, 11 Sep 1996, Admin wrote:

> >This is an absurd and inaccurate analogy. The mailBot simply pushes
> >*mailto:* tags that people have willingly placed on their public websites.
> >They invite the mail so a more accurate analogy would be that City Hall puts
> >up a public suggestion box and invites comments. The bot then puts one, and
> >only one, unsigned anonymous suggestion into the box.  Then why invite them to your site (or even have a site)? It's
like if I sent out postcards to people with no return address saying,
"Please stop by 1313 Mockingbird Lane for snacks and refreshments. Signed,
A Friend". (Note, this is not my real address.) Only in this case the
postcards don't have any stamps so the people who receive them have to pay
the postage. Which is precisely what you're doing: Sending someone e-mail
costs them time and it often costs them money. Sending it through an
anonymous remailer accomplishes four things: 1) People can't complain to
you to stop sending them mail. 2) It costs us, the International Secret
Cabal of Anonymous Remailer Operators, time and money for no good reason,
because you are sending anonymous mail and not making yourself anonymous.
 4) It's giving us a bad
rap, man, and that's the *last* thing the remailer net needs now.

I could care less about your little applet, or what you have to say; in
the words of Thomas Jefferson, "it neither breaks my leg nor picks my
pocket." Until, that is, you start sending it through my remailer, and
*then* you start picking my pocket. The remailers of the world don't exist
to provide non-anonymous anonymous advertising, which you could do just as
well on your own sending forged e-mail headers from netcom.

Andy Dustman / Computational Center for Molecular Structure and Design / UGA
===== For PGP public key:  finger<EMAIL_ADDRESS>| pgp -fka =====
Sure, the Telecomm Act will create jobs: 100,000 new thought-cops on the net
http://charon.chem.uga.edu/~andy<EMAIL_ADDRESS>  <}+++<

-----BEGIN PGP SIGNATURE-----
Version: 2.6.2

iQCVAwUBMjdoky0jMb7JduJJAQGEhAQAt6t18GgMP3v6axTGtNiUHXwNN7UR/V+F
XAZtvTXgg9KFR+ZwnfJz3IMrry0aQCNMC2Ude7mldFyfq8FqVVrA6sE26rwYvWUS
U3C3SCiDXhp3rn9RDfbWV8mZcnC6IRHYz9o5qyuhZTfvGdlZr7/nDTOZZPO7icCN
MiYEGcjJzqY=
=l0Ek
-----END PGP SIGNATURE-----

{% endraw %}
```

## Thread

+ Return to [September 1996](/archive/1996/09)

+ Return to "[Admin <admin@superhot.com>](/authors/admin_admin_at_superhot_com_)"
+ Return to "[Andy Dustman <andy@CCMSD.chem.uga.edu>](/authors/andy_dustman_andy_at_ccmsd_chem_uga_edu_)"

+ 1996-09-12 (Thu, 12 Sep 1996 08:35:56 +0800) - [Nonsense, absolute nonsense... [Fwd: HipCrime and Spam]](/archive/1996/09/a4b3d517b4880bd422a74ae83eacf8c83a7afe11dd0a9c2856059826cdf33a08) - _Admin<EMAIL_ADDRESS>  + 1996-09-12 (Thu, 12 Sep 1996 12:11:24 +0800) - Re: Nonsense, absolute nonsense... [Fwd: HipCrime and Spam] - _Andy Dustman<EMAIL_ADDRESS>


text:
---
title: "The power of showing appreciation"
date: "2019-10-15T08:38:00.000Z"
category: "weekly-tips, speaking"
---

Yesterday's article brought up the subject of relationship building in communication.

On this topic, there's a very simple practice I've been working on for the last few years that I'd like to recommend.

Express appreciation. Directly.

<!-- more -->

Yesterday's article brought up the subject of [relationship building in communication](https://www.speakwritelisten.com/blog/10-14-relationship-layer-in-communication).

On this topic, there's a very simple practice I've been working on for the last few years that I'd like to recommend.

Express appreciation. Directly.

Did you see a speaker give a talk or presentation that you liked? Go and tell them so.

A coworker do some work that you thought was well done? Tell them.

A random stranger do something you think was kind? Tell them.

### Low Barrier Kindness

One reason I practice this and recommend it is that it is a low barrier form of kindness. It costs nothing to tell someone you liked what they did, or you appreciated their work. And yet it feels good to receive.

It's also remarkably rare for most people to hear.

You might think that experts frequently hear appreciation for their work, but in reality most experts are extremely focused on what they're doing wrong rather than what they're doing right. They watch their own presentations for mistakes, self-criticize, and devalue their own skills just like the rest of us!

### Relationship Building

The other thing about showing appreciation is that it starts to build a relationship. You're telling the other person "Hey, you are seen. I appreciate you."

Whether this is someone you work with every day, or someone you've never met before, that connection will make any future interactions you have with them easier.

It's also a great way to *start* a conversation. I don't know about you, but I'm naturally shy and it can be super hard to start a conversation. Noticing something you appreciate and calling it out is a great way to cross the invisible barrier and simply start talking with someone.

### It's simpler than you might think

When I first started working on this practice, I thought I had to have something clever to say. "Hey, I really liked how you drew out that deep parallel between X and Y".

Nope.

This isn't about showing your cleverness, it's simply about telling the person they are appreciated.

"That was a really good talk, I liked it"

"You did great on that project. Well done."

It doesn't need to be complicated because you're not communicating on the _mental_ or _logical_ level, you're communicating on the emotional level. You are appreciated. I see you as a human being, and I'm glad you're here.

And you know what? We need more of that.


text:
<!-- Do not edit this file. It is automatically generated by API Documenter. -->

[Home](./index.md) > [@anglr/select](./select.md) > [DynamicOptionsGathererOptions](./select.dynamicoptionsgathereroptions.md) > [delay](./select.dynamicoptionsgathereroptions.delay.md)

## DynamicOptionsGathererOptions.delay property

Number of miliseconds which represents minimal delay between calls of dynamicOptionsCallback

<b>Signature:</b>

```typescript
delay?: number;
```


text:
# ctxvfs [![Build Status](https://travis-ci.org/sourcegraph/ctxvfs.svg)](https://travis-ci.org/sourcegraph/ctxvfs)

Package ctxvfs defines a virtual file system interface whose methods accept a context.Context parameter. It is otherwise similar to vfs.FileSystem.

This package is **experimental** until further notice.

[**Open the code in Sourcegraph**](https://sourcegraph.com/github.com/sourcegraph/ctxvfs)

repository: sourcegraph/ctxvfs

gha_language: Go

repo_name: sourcegraph/ctxvfs


text:
# angularmusicstore

The MVC Music Store - but built on angular 2

**Note:** Angular 2.0 is not production ready yet! This seed project is perfect for playing around with the latest versions but do not start new projects with it since a lot of new changes are going to be introduced until the framework is officially released.

This project is based upon the awesome [Angular 2 seed project by Minko Gechev](https://github.com/mgechev/angular2-seed).
We have updated it to implement the functionality originally provided by the iconic [MVC Music Store] (http://www.asp.net/mvc/overview/older-versions/mvc-music-store/mvc-music-store-part-1).

# Quickstart

Open `PowerShell` or `CMD` and navigate to a suitable directory (e.g. `~\src`) and execute the following.

    git clone https://github.com/SSWConsulting/musicstore-ui-angular2-dotnet.git
    cd musicstore-ui-angular2-dotnet
    .\setup-environment.bat

# Detailed Building and deploying AngularMusicStore
Instructions for the developer

## Setting up the tools
Check that you have the required tools:

1.  Visual Studio 2015 Update 1
2.  ASP.Net 5
3.  Node
4.  NPM
5.  Gulp
6.  TypeScript
7.  WebEssentials - Visual Studio Extension

### Visual Studio 2015 Update 1
Download from here: [Download](http://go.microsoft.com/fwlink/?LinkId=691129)

Ensure you install it with Python tools.

### ASP.Net 5

Install [ASP.NET 5](https://go.microsoft.com/fwlink/?LinkId=627627)

Enable command line tools

    dnvm upgrade

See [more information](http://docs.asp.net/en/latest/getting-started/installing-on-windows.html)

### Node
You need to upgrade nodejs to the latest version.
Download and install from [here](https://nodejs.org/en/download/)

### NPM
The npm preinstalled with VS 2015 is outdated, so it needs to be updated.
You can check the current version doing:

    npm -v

You should at least have version 3.5.0, if not, in the command prompt, run the following command to install it globally

    npm install -g  npm

The result should be something like this, but version should be at least 3.5.0:
![README](README_images/README.png)

If Bower is not present, install globally using npm:

    npm install -g bower

### Gulp
Then check for gulp the same way, just do:

     gulp

 and if no such program, again use npm to install it globally

    npm install -g gulp

### Typescript
 Typescript comes with VS2015 too, but we install it globally together with the typescript definition manager.

     npm install -g  typescript
     npm install tsd -g

### WebEssentials

Install this from the gallery, or from Visual Studio Extension dialog, link to it is [here](https://visualstudiogallery.msdn.microsoft.com/ee6e6d8c-c837-41fb-886a-6b50ae2d06a2)

### Other tools

To edit markdown files, you can use the built-in WebEssentials tool, but it doesnt really handle images (copy/pasting etc) nicely.  A much better tool is [MarkPad](http://code52.org/DownmarkerWPF/).

## Setting up the project

* git clone https://github.com/SSWConsulting/angularmusicstore  angularmusicstore
* cd angularmusicstore
*# SSW Angular Music Store
### Set up your visual studio 2015
1. Install Visual Studio 2015
2. Install nodejs
      source: https://nodejs.org/en/. use version 4 or higher
	  reason: adds node globally so you can work with npm or tools like bower and gulp outside of Visual Studio

3. Install npm globally
        Run the following commands
		- npm install npm -g (This is included with a generic install of node so can normally be skipped)

3. Install typescript and gulp globally so we can use them anywhere
		Run the following commands
		- npm install typescript gulp -g

4. Install ASP.Net 5
	follow the instructions on http://docs.asp.net/en/latest/getting-started/installing-on-windows.html
	(includes installing ASP.Net beta, dnvm setup & dnvm upgrade)

### Getting and running the code
1. Clone the project
2. Run the command `dnu restore`
3. Run `npm install` from the command line for the SSW.MusicStore.Web project
4. Run `gulp` will create a dist folder
4. Run dnx web from the command line for the SSW.MusicStore.Web project and navigate to localhost:5000/dist/app to view the site in the browser


text:
How to avoid going insane with [Brook Jeynes](https://ssw.com.au/people/brook-jeynes/) (4 min)**

NPM packages often get new releases (adding new functionalities, fixing bugs or vulnerabilities). It is important to keep the packages updated as much as possible during the development of your application. The best way to do that is to update all the packages every time you add a new package to your application, and include the `npm outdated` and `npm audit` reports in the Sprint Review. These commands are also available with yarn with `yarn outdated` and `yarn audit`.

<!--endintro-->

Running `npm outdated` returns an overview of your packages versions showing:
- The **current** version your project is using
- The **wanted** version, i.e. the maximum version of the package that satisfies the semver range specified in package.json
- The **latest** version of the package

The packages output from this command will also show up in different colours:
-  **Red** indicates the package version is below the wanted version
-  **Yellow** indicates the package version is at the wanted version but below the latest version

![Figure: Use 'npm outdated'](npm_outdated.png)

`npm audit` returns an audit on your packages for vulnerabilities. It also provides information on how to resolve them.

![Figure: Use 'npm audit' to discover vulnerabilities in your application](npm_audit.png)

To add a new package, use `npm install package-name` or `yarn add package-name`.

To update your packages, use `npm update package-name` or `yarn upgrade package-name`. Yarn also has a useful tool called `yarn upgrade-interactive` that allows you to see which packages are outdated, and upgrade them all at once.

![Figure: Using yarn upgrade-interactive](upgrade-interactive.png)

::: greybox
**Note:** Use `yarn upgrade-interactive --latest` to see outdated packages with breaking changes.
:::

gha_language: TSQL


text:
# Grading Feedback for jalma146

## Functionality (136/140)

###   Simulator Basic Functionality
* Correct Disassembly (10/10)
* Flags panel (2/2)
* Registers panel (3/3)
* Step/Run Functionality (10/10)

###   Basic Instruction Tests
* Data Processing Tests (8/10)
* Load/Store Tests (9/10)
* Branch Tests (9/10)
* CMP Tests (10/10)
* Other Non-IO Tests (10/10)

###   I/O Tests
* B/C Output (0/8)
* C-Level Input (0/7)
* B-Level Input (0/27)
* A-Level Input/Output (65/65)

## Style (20/20)
* Comments (15/15)
* Code Organization / General Style (5/5)

## Submission (5/5)
* Correct Electronic Submission (5/5)

## Technical Report (42/45)
* Intro/Features/Prerequisites/Config (5/5)
* Build/Test (5/5)
* User Guide (10/10)
* Software Architecture (10/10)
* Bug Report (4/5)
  **Note: Bug report fails to note minor deviations in trace files**
* Appendices (5/5)
* Formatting (3/5)
  **Note: Effort shown, but lacks professional polish**

## Subtotal (203/200)

## Total (203/200)

repository: JoannaMadNat/ARMSim

repo_name: JoannaMadNat/ARMSim


text:
[![Switch](../docs/Pictures/Menu/Switch.png)](Home.md)[![Switch](../docs/Pictures/Menu/Gallery.png)](Gallery.md)[![Switch](../docs/Pictures/Menu/Examples.png)](Examples.md)[![Switch](../docs/Pictures/Menu/Downloads.png)](Downloads.md)[![Switch](../docs/Pictures/Menu/Documentation.png)](Documentation.md)[![Switch](../docs/Pictures/Menu/Project.png)](https://sourceforge.net/projects/switchpro)[![Switch](../docs/Pictures/Menu/Sources.png)](https://github.com/gammasoft71/switch)[![Switch](../docs/Pictures/Menu/Contact.png)](Contact.md)[![Switch](../docs/Pictures/Menu/Gammasoft.png)](https://gammasoft71.wixsite.com/gammasoft)

One common concern related to exceptions is that if exceptions are used for code that routinely fails, the performance of the implementation will be unacceptable. This is a valid concern. When a member throws an exception, its performance can be orders of magnitude slower. However, it is possible to achieve good performance while strictly adhering to the exception guidelines that disallow using error codes. Two patterns described in this section suggest ways to do this.

**X DO NOT** use error codes because of concerns that exceptions might affect performance negatively.

To improve performance, it is possible to use either the Tester-Doer Pattern or the Try-Parse Pattern, described in the next two sections.

# Tester-Doer Pattern

Sometimes performance of an exception-throwing member can be improved by breaking the member into two. Let’s look at the Add method of the System::Collections::Generic::ICollection<T> interface.

```c++
refptr<ICOllection<int>> numbers = ...
numbers->Add(1);
```

The method Add throws if the collection is read-only. This can be a performance problem in scenarios where the method call is expected to fail often. One of the ways to mitigate the problem is to test whether the collection is writable before trying to add a value.

```c++
refptr<ICOllection<int>> numbers = ...
...
if ((!numbers->IsReadOnly) {
  numbers->Add(1);
}
```

The member used to test a condition, which in our example is the property IsReadOnly, is referred to as the tester. The member used to perform a potentially throwing operation, the Add method in our example, is referred to as the doer.

**√ CONSIDER** the Tester-Doer Pattern for members that might throw exceptions in common scenarios to avoid performance problems related to exceptions.

# Try-parse Pattern

For extremely performance-sensitive APIs, an even faster pattern than the Tester-Doer Pattern described in the previous section should be used. The pattern calls for adjusting the member name to make a well-defined test case a part of the member semantics. For example, System::DateTime defines a Parse method that throws an exception if parsing of a string fails. It also defines a corresponding TryParse method that attempts to parse, but returns false if parsing is unsuccessful and returns the result of a successful parsing using an out parameter.

```c++
struct DateTime : public object {
public:
  static DateTime Parse(const string& dateTime) {
    ...
  }

  static bool TryParse(const string& dateTime, DateTime& result) {
    ...
  }
};
```

When using this pattern, it is important to define the try functionality in strict terms. If the member fails for any reason other than the well-defined try, the member must still throw a corresponding exception.

**√ CONSIDER** the Try-Parse Pattern for members that might throw exceptions in common scenarios to avoid performance problems related to exceptions.

**√ DO** use the prefix "Try" and Boolean return type for methods implementing this pattern.

**√ DO** provide an exception-throwing member for each member using the Try-Parse Pattern.

# See also

Other Resources

* [Framework Design Guidelines](FrameworkDesignGuidelines.md)
* [Design Guidelines for Exception](DesignGuidelinesForException.md)

______________________________________________________________________________________________

© 2018 Gammasoft.

repository: victor-timoshin/Switch

repo_name: victor-timoshin/Switch


text:
# Android Instant Apps - Feature Module sample

This sample introduces feature modules to the Hello World sample.

Code and resources that are only required within a single feature module
can be found in the corresponding feature modules.

* `features/base` -> Shared
* `features/hello` -> Hello feature
* `features/bye` -> Bye feature

## Launch URLs

```
https://hello-feature.instantappsample.com/hello -> HelloActivity
https://hello-feature.instantappsample.com/goodbye -> GoodbyeActivity
```

## License

```
Copyright 2017 Google Inc.

Licensed to the Apache Software Foundation (ASF) under one or more contributor
license agreements. See the NOTICE file distributed with this work for
additional information regarding copyright ownership. The ASF licenses this
file to you under the Apache License, Version 2.0 (the "License"); you may not
use this file except in compliance with the License. You may obtain a copy of
the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations under
the License.
```

repository: DalavanCloud/android-instant-apps

repo_name: DalavanCloud/android-instant-apps


text:
---
title: "bookdown: Authoring Books and Technical Documents with R Markdown"
author: "Yihui Xie"
date: "2023-05-25T14:39:43Z"
tags: [R Markdown, Guide, Shiny]
link: "https://bookdown.org/yihui/bookdown/"
length_weight: "8.9%"
cover: "https://bookdown.org/yihui/bookdown/images/cover.jpg"
repo: "rstudio/bookdown"
pinned: true
---

A guide to authoring books with R Markdown, including how to generate figures and tables, and insert cross-references, citations, HTML widgets, and Shiny apps in R Markdown. The book can be exported to HTML, PDF, and e-books (e.g. EPUB). The book style is customizable. You can easily write and preview the book in RStudio IDE or other editors, and host the book wherever you want (e.g. bookdown.org).

gha_language: R


text:
# DZThemeKit

[![Carthage compatible](https://img.shields.io/badge/Carthage-compatible-4BC51D.svg?style=flat)](https://github.com/Carthage/Carthage)

DZThemeKit is a simple implementation for managing multiple User Interface themes in your app. For the moment, only UIKit is being targeted. Categories for WatchOS and AppKit will be added soon.

### Installation
You can add the project's source to your Project or Workspace directly. However, carthage is the preferred method:
```
github "dzns/DZThemeKit"
```

### Usage
To start using ThemeKit, all you have to do is simply add the following line, preferrably in your AppDelegate file.
```objc
#import <DZThemeKit/DZThemeKit.h>
```

This loads up ThemeKit and makes the `MyThemeKit` singleton available for you. ThemeKit comes with a few sane defaults with the exception of  `customColor`.  That's just a sample to demonstrate you can add your own colors easily.
This however does not make everything automatic for you.

Here is a sample implementation of how a basic ViewController implementation may look like:

```objc
- (void)themeView {
    self.titleBlock.backgroundColor   = [MyThemeKit valueForKey:@"titleColor"];
    self.textBlock.backgroundColor    = [MyThemeKit valueForKey:@"textColor"];
    self.subtextBlock.backgroundColor = [MyThemeKit valueForKey:@"subtextColor"];
    self.borderBlock.backgroundColor  = [MyThemeKit valueForKey:@"borderColor"];
    self.tintBlock.backgroundColor    = [MyThemeKit valueForKey:@"tintColor"];
    self.customBlock.backgroundColor  = [MyThemeKit valueForKey:@"customColor"];
}
```

You can call the above method from your `-[viewDidLoad]` in your view controller or `-[awakeFromNib]` or similar from inside your view.

### Theme Updates

DZThemeKit comes with Dark UI support out of the box. If you're loading a theme called `aqua.json`, DZThemeKit will automatically look for a `aqua-dark.json` in the same directory. If the file is found, it is loaded automatically for you.

From there, you can set `autoUpdatingTheme` to `YES` on `MyThemeKit` or your own instance and it'll fire a `ThemeNeedsUpdateNotification` notification when the display update's it's brightness for accomodating ambient light.

The `ThemeNeedsUpdateNotification` is fired once when the instance is first loaded into the memory and loads the default `colours.json`. If you load your own theme later, it'll be fired again.

### Categories
A few categories are available. These are basic at the moment and more advanced implementations will be made available soon. If you have any ideas, please feel free to open an issue with them.

```objc
- [UIView tk_updateBackgroundColor];
- [UILabel tk_updateTitleColorForTheme];
- [UILabel tk_updateTextColorForTheme];
- [UILabel tk_updateSubtextColorForTheme];
- [UITextField tk_setTextColor];
- [UITextField tk_setPlaceholderColor];
- [UITextView tk_setTextColor];
```

Whenever querying any of the color values from `ThemeKit`
- you have either set the value of `useDark` or the framework has updated it for you
- a dark theme configuration was found and loaded successfully
the framework will automatically return a value from either the normal or dark variant based on the boolean value of `isDark`.

### Colours
You can setup your colours using a simple JSON structure. You can view the default file here.

Notable features include:
- All colours are implemented using the sRGB colour space.
- You can optionally tell the framework to auto-extract the **DCI-P3** value for a colour by creating a dictionary value like so:
```json
{
  "rgb": "#2979FF",
  "p3": true
}
```

If you don't want a DCI-P3 value, you can set it to false which is the default behavior.

- Or you can provide the P3 value yourself if you have fine tuned it in your graphics program like so:
```json
{
  "rgb": "#2979FF",
  "p3": "#0631FF"
}
```

The framework also supports colour values like the following:
```
#33350    // 333333 at 0.5 alpha
#E00      // EE0000 at 1.0 alpha
#2979FF50 // 2979FF at 0.5 alpha
```

### LICENSE
DZThemeKit is available under the MIT License. Please see the LICENSE file for more information.

repository: DZNS/DZThemeKit

gha_language: Objective-C

repo_name: DZNS/DZThemeKit


text:
---
layout: post
title: Notes for anwsering a piazza question
date: 2017-11-8
author: Kaiyuan Chen
catalog: true
tags:
    - Machine Learning
    - Study Notes
---

# Notes for anwsering a piazza question
What's difference between multi-class log linear also known as logistic regression model and a maximum entropy classifier?

### Professor's Anwser:
> Maximum entropy and log-linear model apply different modeling ideas. But, the final model they got is the same.

> the optimization problem solved when learning a maximum entropy model is the dual problem of maximizing log-likelihood of a log-linear model.

### Explaination
Slides Lec3 73 / Binder 239
Log linear model: learns directly from w^Tx
softmax: learns a score for output y, which can be w^Tx or others like Kesler's construction

or

                                                  |     generative model      |       discriminative model
     ---------------------------------------------------------------------------------------------
     directed graph model           |      HMM                        |         MEMM
     ---------------------------------------------------------------------------------------------
     undirected graph model       |          neural network                       |         MaxEnt, CRF, MRF

Maximum Entropy models the conditional dependency between the observation and outputs. It uses the maximum entrophy function, while not use the pure conditional probablity to measure the output dependence given the observation(Compared to Bayes).

### maximum entropy modelling
Maximum entropy and log-linear model apply different modeling ideas. But, the final model they got is the same.

It is a way of choosing the maximum entropy distribution P(x). It satisfies the input constraints and maximizes the uncertainty.

### random topics
1 vs k
    w_i^Tx > w_j^Tx \forall j
Loss function: Perceptron loss, hinge loss(for SVM, very straight one), exponential loss, logistic loss
    Hinge : incorrect for linear penalty and no penalty for far away from 1
    logistic/sigmoid
        can view probabilistically, decision boundary is log(P=1 / P=-1) = w^T + b (which is very interesting)
Kesler's construction: allows for constructing N features based on K classes. We can select both from input and output space.
HMM assumption: P(x1,x2 ... y1, y2 ...)

### TODO
viterbi algorithm


text:
#Recipes

Although Cloudbreak lets you provision Hadoop clusters in the cloud from Ambari blueprints, Cloudbreak built-in provisioning doesn't contain all possible use cases. For that reason, we introduced recipes.

A **recipe** is a script extension to a cluster that runs on all nodes before or after the Ambari cluster installation. For example, you can use a recipe to put a JAR file on the Hadoop classpath. Recipes are uploaded and stored in Cloudbreak through the web UI or shell.

The easiest way to create a custom recipe is to:

  * Create your own pre and/or post scripts
  * Upload them using the web UI or shell

##Adding Recipes

To add recipe via the web UI, in the **manage recipes** section, choose **create new recipe**. Next, select either SCRIPT or FILE type plugin and fill in required fields.

To add recipe via shell, use the following command:

```
recipe create --name [recipe-name] --type [PRE|POST] --scriptFile /path/of/the/recipe-script
```

This command has optional parameters:

`--description` "string" description of the recipe

`--scriptUrl` "string" remote location of the recipe

`--publicInAccount` "flag" flags if the recipe is public in the account

## Sample Recipe for Yum Proxy Setting

We've created a sample recipe that can be used to set proxy for yum:

```
#!/bin/bash
cat >> /etc/yum.conf <<ENDOF
proxy=http://<IP_ADDRESS>:3128
ENDOF
```

![](/images/create-recipe.png)

You can add this recipe to hostgroups:

![](/images/select-recipe.png)

repository: sequenceiq/cloudbreak-docs

repo_name: sequenceiq/cloudbreak-docs


text:
# Grow <!-- omit in toc -->

- [Getting Started](#getting-started)
  - [Requirements](#requirements)
    - [Python 3 & pip](#python-3--pip)
    - [Enabling i2c and spi](#enabling-i2c-and-spi)
  - [Installing the library](#installing-the-library)
- [Reference](#reference)
  - [Moisture](#moisture)
    - [Calibrating the moisture sensor](#calibrating-the-moisture-sensor)
    - [Moisture Reference](#moisture-reference)
      - [moisture](#moisture-1)
      - [saturation](#saturation)
      - [range](#range)
      - [new_data](#new_data)
      - [active](#active)
  - [Pump](#pump)
    - [Calibrating The Pump](#calibrating-the-pump)
    - [Pump Reference](#pump-reference)
      - [dose](#dose)
      - [set_speed](#set_speed)
      - [get_speed](#get_speed)
      - [stop](#stop)
  - [Light Sensor](#light-sensor)
  - [Display](#display)

## Getting Started

You'll need to install the LTP305 software library and enable i2c on your Raspberry Pi.

### Requirements

#### Python 3 & pip

You should use Python 3, which may need installing on your Pi:

```
sudo apt update
sudo apt install python3 python3-pip python3-setuptools
```

#### Enabling i2c and spi

You can use `sudo raspi-config` on the command line, the GUI Raspberry Pi Configuration app from the Pi desktop menu, or use the following commands to enable i2c and spi:

```
sudo raspi-config nonint do_i2c 0
sudo raspi-config nonint do_spi 0
```

### Installing Dependencies

The following dependencies are required:

```
sudo apt install python3-yaml python3-smbus python3-pil python3-spidev python3-rpi.gpio
```

### Installing the library

```python
python3 -m pip install growhat
```

This will also install the display driver - ST7735 - and a driver for the light sensor - LTR559.

## Reference

The Grow library includes several modules for monitoring, watering and conveying status information.

### Moisture

The moisture module is responsible for reading the moisture sensor.

Grow moisture sensors output pulses that correspond to the moisture content of the soil.

Wet soil will produce low frequency pulses, which will gradually become more frequent as the soil dries.

The range runs from about 27-28 pulses per second in air, to <=1 pulse per second in fully saturated soil.

To set up a moisture sensor you must import and initialise the class with the channel number you want to monitor:

```python
from grow.moisture import Moisture

moisture1 = Moisture(1)  # Monitor channel 1
```

Moisture channels are labelled on the PCB from left to right as S1, S2 and S3.

#### Calibrating the moisture sensor

Since every soil substrate and environment is different, you must tell Grow what Wet and Dry soil look like. You can do this using the methods `set_wet_point` and `set_dry_point`.

The wet and dry points are expressed as raw counts. You can read the raw moisture level of the soil at any time by using:

```python
moisture1.moisture()
```

These methods will either use the current read value or, optionally, a value that you supply so that you can save your calibration and load it again if you need to restart your script. A simple calibration process might look like the following:

```python
# Leave the soil dry
moisture1.set_dry_point()
# Water the plant
moisture1.set_wet_point()
```

Or you could have variables that you wish to set as your wet/dry points:

```python
DRY_POINT = 27
WET_POINT = 1
moisture1.set_dry_point(DRY_POINT)
moisture1.set_wet_point(WET_POINT)
```

#### Moisture Reference

##### moisture

```python
value = moisture1.moisture()
```

Reads the raw moisture level of the sensor.

This is expressed in pulses-per-second and is inversely proportional to the moisture content of the soil.

The moisture sensors are deliberately scaled to a very low frequency and range from around 27-28Hz in dry air down to around 1Hz in saturated soil.

##### saturation

```python
value = moisture1.saturation()
```

Returns a value from 0.0 to 1.0 that corresponds to how moist the soil is.

The saturation level is calculated using the wet and dry points and is useful for populating visual indicators, logs or graphs.

##### range

```python
moisture1.range()
```

Returns the distance, in counts per second, between the wet and dry points. Used predominately in the calculation of saturation level, but may be useful.

##### new_data

```python
moisture1.new_data()
```

Returns `True` if the internal moisture reading has been updated since the last time `moisture()` or `saturation()` was called.

This allows you to log or run averaging over each new reading as it is collected or process readings in your program loop.

For example, this snippet will loop ~60 times a second but only append new moisture readings to `log` once every second:

```python
import time
from grow.moisture import Moisture

moisture1 = Moisture(1)  # Monitor channel 1
log = []

while True:
    if moisture1.new_data():
        log.append(moisture1.saturation())

    time.sleep(1.0 / 60)  # 60 updates/sec
```

##### active

```python
moisture1.active()
```

Returns `True` if the moisture sensor is connected and returning valid readings.

Checks if a pulse has happened within the last second, and that the reading is within a sensible range.

### Pump

The Pump module is responsible for driving a pump. It uses PWM to run the pump at variable speeds.

To set up a pump you must import and initialise the class with the channel number you want to control:

```python
from grow.pump import Pump

pump1 = Pump(1)  # Control channel 1
```

Pump channels are labelled on the PCB from left to right as P1, P2 and P3.

#### Calibrating The Pump

In order for your pump/hose setup to reliably deliver a sensible amount of water you will have to calibrate both the speed at which it runs, and the time that it runs for. The speed/time settings are collectively known as a "dose" and we've produced an example - `dose.py` - to help find the right values.

Higher pump speeds are useful for defeating gravity; in case where you water tank might be much lower than your plants. Longer pump times will deliver more water.

Refer to the instructions at the top of `dose.py` for its usage. Generally you should dial in the lowest pump speed that will get water to traverse your hose, and adjust the time to deliver the right amount of water.

These values are then passed into `pump.dose()` when watering.

#### Pump Reference

##### dose

```python
pump1.dose(0.5, 0.5, blocking=False)
```

Delivers a dose of water by running the pump at a specified speed for a specified time. Can be run in blocking (method takes time to return) or non-blocking (returns instantly and pump runs in the background) modes.

This is the recommended way to water plants, since it delivers a controlled, short dose and is less likely to result in unwanted floods.

##### set_speed

```python
pump1.set_speed(0.5)
```

Turns the pump on at the given speed. To stop the pump call `stop()` or `set_speed(0)`.

Unless your setup (vertical hydroponics for example) requires continuous pumping of water then you should not use this function and use the `dose` instead.

##### get_speed

```python
speed = pump1.get_speed()
```

Returns the current pump speed.

##### stop

```python
pump1.stop()
```

Stops the pump by setting the speed to 0.

### Light Sensor

Grow is equipped with an LTR-559 light and proximity sensor that you can use to limit waterings to daytime, or monitor the level of light your plant is receiving.

The LTR-559 has its own library, and should be initialised like so:

```python
from ltr559 import LTR559
light = LTR559()
```

You can then update the sensor and retrieve the `lux` and `proximity` values like so:

```python
while True:
    ltr559.update_sensor()
    lux = ltr559.get_lux()
    prox = ltr559.get_proximity()
```

See the LTR559 library for a full reference: https://github.com/pimoroni/ltr559-python

### Display

The ST7735 display on Grow is 160x80 pixels and a great way to convey current watering status or build an interface for controlling your watering station.

The display must be set up like so:

```python
display = ST7735.ST7735(
    port=0,
    cs=1,               # Chip select 1 (BCM )
    dc=9,               # BCM 9 is the data/command pin
    backlight=12,       # BCM 12 is the backlight
    rotation=270,
    spi_speed_hz=80000000
)
display.begin()
```

You should use the Python Image Library to build up what you want to display, and then display the finished image with:

```python
display.display(image)
```

See the examples for demonstrations of this. See the ST7735 library for a full reference: https://github.com/pimoroni/st7735-python/

repository: NewGardenFarms/grow-python

repo_name: NewGardenFarms/grow-python