text:
# cs-hidden-markov-models

Hidden Markov Models using C#. Hidden Markov Models for sequence generation, event sequence prediction, supervised learning on temporal sequence. The library is built on .NET 4.5.2.

# Install

```bash
Install-Package cs-hidden-markov-models -Version 1.0.1
```

# Usage

## Generate / forecast sequence

The sample code below shows how to use HMM to generate a sequence based on historical data:

```cs
MathHelper.SetupGenerator(42);

// Consider some phrases:
//
string[][] phrases =
 {
	 new[] { "those", "are", "sample", "words", "from", "a", "dictionary" },
	 new[] { "those", "are", "sample", "words" },
	 new[] { "sample", "words", "are", "words" },
	 new[] { "those", "words" },
	 new[] { "those", "are", "words" },
	 new[] { "words", "from", "a", "dictionary" },
	 new[] { "those", "are", "words", "from", "a", "dictionary" }
 };

// Let's begin by transforming them to sequence of
// integer labels using a codification codebook:
var codebook = new Codification(phrases);

// Now we can create the training data for the models:
int[][] sequence = codebook.Translate(phrases);

// To create the models, we will specify a forward topology,
// as the sequences have definite start and ending points.
//
var topology = new Forward(state_count: 4);
int symbols = codebook.SymbolCount; // We have 7 different words
Console.WriteLine("Symbol Count: {0}", symbols);

// Create the hidden Markov model
HiddenMarkovModel hmm = new HiddenMarkovModel(topology, symbols);

// Create the learning algorithm
BaumWelchLearning teacher = new BaumWelchLearning(hmm);

// Teach the model about the phrases
double error = teacher.Run(sequence);

// Now, we can ask the model to generate new samples
// from the word distributions it has just learned:
//
int[] sample = hmm.Generate(3);

// And the result will be: "those", "are", "words".
string[] result = codebook.Translate(sample);

foreach(string result_word in result)
{
	Console.WriteLine(result_word);
}
```

## Evaluate the likelihood of discrete event sequence

The sample codes below evaluate the likelihood of discrete event sequences based on historical data.

```cs
int[][] sequences = new int[][]
   {
	   new int[] { 0,1,1,1,1,0,1,1,1,1 },
	   new int[] { 0,1,1,1,0,1,1,1,1,1 },
	   new int[] { 0,1,1,1,1,1,1,1,1,1 },
	   new int[] { 0,1,1,1,1,1         },
	   new int[] { 0,1,1,1,1,1,1       },
	   new int[] { 0,1,1,1,1,1,1,1,1,1 },
	   new int[] { 0,1,1,1,1,1,1,1,1,1 },
   };

// Creates a new Hidden Markov Model with 3 states for
//  an output alphabet of two characters (zero and one)
HiddenMarkovModel hmm = new HiddenMarkovModel(state_count: 3, symbol_count: 2);

// Try to fit the model to the data until the difference in
//  the average log-likelihood changes only by as little as 0.0001
var teacher = new BaumWelchLearning(hmm) { Tolerance = 0.0001, Iterations = 0 };
double ll = teacher.Run(sequences);

// Calculate the probability that the given
//  sequences originated from the model
double l1 = hmm.Evaluate(new int[] { 0, 1 });       // 0.999
double l2 = hmm.Evaluate(new int[] { 0, 1, 1, 1 }); // 0.916
Console.WriteLine("l1: {0}", System.Math.Exp(l1));
Console.WriteLine("l2: {0}", System.Math.Exp(l2));

// Sequences which do not start with zero have much lesser probability.
double l3 = hmm.Evaluate(new int[] { 1, 1 });       // 0.000
double l4 = hmm.Evaluate(new int[] { 1, 0, 0, 0 }); // 0.000
Console.WriteLine("l3: {0}", System.Math.Exp(l3));
Console.WriteLine("l4: {0}", System.Math.Exp(l4));

// Sequences which contains few errors have higher probability
//  than the ones which do not start with zero. This shows some
//  of the temporal elasticity and error tolerance of the HMMs.
double l5 = hmm.Evaluate(new int[] { 0, 1, 0, 1, 1, 1, 1, 1, 1 }); // 0.034
double l6 = hmm.Evaluate(new int[] { 0, 1, 1, 1, 1, 1, 1, 0, 1 }); // 0.034
Console.WriteLine("l5: {0}", System.Math.Exp(l5));
Console.WriteLine("l6: {0}", System.Math.Exp(l6));

```

## Evaluate the likelihood of continuous value sequence

The sample codes below evaluate the likelihood of continuous value sequence based on historical data.

```cs
// Create continuous sequences. In the sequences below, there
//  seems to be two states, one for values between 0 and 1 and
//  another for values between 5 and 7. The states seems to be
//  switched on every observation.
double[][] sequences = new double[][]
{
	new double[] { 0.1, 5.2, 0.3, 6.7, 0.1, 6.0 },
	new double[] { 0.2, 6.2, 0.3, 6.3, 0.1, 5.0 },
	new double[] { 0.1, 7.0, 0.1, 7.0, 0.2, 5.6 },
};

// Specify a initial normal distribution for the samples.
Gaussian density = new Gaussian();

// Creates a continuous hidden Markov Model with two states organized in a forward
//  topology and an underlying univariate Normal distribution as probability density.
var model = new HiddenMarkovModel(new Ergodic(2), density);

// Configure the learning algorithms to train the sequence classifier until the
// difference in the average log-likelihood changes only by as little as 0.0001
var teacher = new BaumWelchLearning(model)
{
	Tolerance = 0.0001,
	Iterations = 0,
};

// Fit the model
double logLikelihood = teacher.Run(sequences);

// See the log-probability of the sequences learned
double a1 = model.Evaluate(new double[] { 0.1, 5.2, 0.3, 6.7, 0.1, 6.0 }); // -0.12799388666109757
double a2 = model.Evaluate(new double[] { 0.2, 6.2, 0.3, 6.3, 0.1, 5.0 }); // 0.01171157434400194
Console.WriteLine("a1 = {0}", a1);
Console.WriteLine("a2 = {0}", a2);

// See the log-probability of an unrelated sequence
double a3 = model.Evaluate(new[] { 1.1, 2.2, 1.3, 3.2, 4.2, 1.0 }); // -298.7465244473417
Console.WriteLine("a3 = {0}", a3);

// We can transform the log-probabilities to actual probabilities:
double likelihood = System.Math.Exp(logLikelihood);
a1 = System.Math.Exp(a1); // 0.879
a2 = System.Math.Exp(a2); // 1.011
a3 = System.Math.Exp(a3); // 0.000
Console.WriteLine("a1 = {0}", a1);
Console.WriteLine("a2 = {0}", a2);
Console.WriteLine("a3 = {0}", a3);

// We can also ask the model to decode one of the sequences. After
// this step the state variable will contain: { 0, 1, 0, 1, 0, 1 }

double lll;
int[] states = model.Decode(new double[] { 0.1, 5.2, 0.3, 6.7, 0.1, 6.0 }, out lll);
Console.WriteLine("states: {{{0}}}", string.Join(", ", states));
Console.WriteLine("lll: {0}", lll);
```

## HMM Classifier

The following sample codes shows how to train and predict temporal sequence using HMM as a supervised classifier:

```cs
public static void Learn()
{
	int[][] inputs = new int[][]
	   {
		   new int[] { 0,1,1,0 },   // Class 0
		   new int[] { 0,0,1,0 },   // Class 0
		   new int[] { 0,1,1,1,0 }, // Class 0
		   new int[] { 0,1,0 },     // Class 0

		   new int[] { 1,0,0,1 },   // Class 1
		   new int[] { 1,1,0,1 },   // Class 1
		   new int[] { 1,0,0,0,1 }, // Class 1
		   new int[] { 1,0,1 },     // Class 1
	   };

	int[] outputs = new int[]
	   {
		   0,0,0,0, // First four sequences are of class 0
		   1,1,1,1, // Last four sequences are of class 1
	   };

	// We are trying to predict two different classes
	int classes = 2;

	// Each sequence may have up to two symbols (0 or 1)
	int symbols = 2;

	// Nested models will have two states each
	int[] states = new int[] { 2, 2 };

	// Creates a new Hidden Markov Model Sequence Classifier with the given parameters
	HiddenMarkovClassifier classifier = new HiddenMarkovClassifier(classes, states, symbols);

	// Create a new learning algorithm to train the sequence classifier
	var teacher = new HiddenMarkovClassifierLearning(classifier,

		// Train each model until the log-likelihood changes less than 0.001
		modelIndex => new BaumWelchLearning(classifier.Models[modelIndex])
		{
			Tolerance = 0.001,
			Iterations = 0
		}
	);

	// Train the sequence classifier using the algorithm
	double likelihood = teacher.Run(inputs, outputs);

	Console.WriteLine("likelihood: {0}", likelihood);
}

public static void LearnAndPredict()
{
	// Suppose we would like to learn how to classify the
	// following set of sequences among three class labels:

	int[][] inputSequences =
	{
		// First class of sequences: starts and
		// ends with zeros, ones in the middle:
		new[] { 0, 1, 1, 1, 0 },
		new[] { 0, 0, 1, 1, 0, 0 },
		new[] { 0, 1, 1, 1, 1, 0 },

		// Second class of sequences: starts with
		// twos and switches to ones until the end.
		new[] { 2, 2, 2, 2, 1, 1, 1, 1, 1 },
		new[] { 2, 2, 1, 2, 1, 1, 1, 1, 1 },
		new[] { 2, 2, 2, 2, 2, 1, 1, 1, 1 },

		// Third class of sequences: can start
		// with any symbols, but ends with three.
		new[] { 0, 0, 1, 1, 3, 3, 3, 3 },
		new[] { 0, 0, 0, 3, 3, 3, 3 },
		new[] { 1, 0, 1, 2, 2, 2, 3, 3 },
		new[] { 1, 1, 2, 3, 3, 3, 3 },
		new[] { 0, 0, 1, 1, 3, 3, 3, 3 },
		new[] { 2, 2, 0, 3, 3, 3, 3 },
		new[] { 1, 0, 1, 2, 3, 3, 3, 3 },
		new[] { 1, 1, 2, 3, 3, 3, 3 },
	};

				// Now consider their respective class labels
				int[] outputLabels =
	{
		/* Sequences  1-3 are from class 0: */ 0, 0, 0,
		/* Sequences  4-6 are from class 1: */ 1, 1, 1,
		/* Sequences 7-14 are from class 2: */ 2, 2, 2, 2, 2, 2, 2, 2
	};

	// We will use a single topology for all inner models, but we
	// could also have explicit led different topologies for each:

	ITopology forward = new Forward(state_count: 3);

	// Now we create a hidden Markov classifier with the given topology
	HiddenMarkovClassifier classifier = new HiddenMarkovClassifier(class_count: 3, topology: forward, symbol_count: 4);

	// And create a algorithms to teach each of the inner models
	var teacher = new HiddenMarkovClassifierLearning(classifier,

		// We can specify individual training options for each inner model:
		modelIndex => new BaumWelchLearning(classifier.Models[modelIndex])
		{
			Tolerance = 0.001, // iterate until log-likelihood changes less than 0.001
			Iterations = 0     // don't place an upper limit on the number of iterations
		});

	// Then let's call its Run method to start learning
	double error = teacher.Run(inputSequences, outputLabels);

	// After training has finished, we can check the
	// output classificaton label for some sequences.

	int y1 = classifier.Compute(new[] { 0, 1, 1, 1, 0 });    // output is y1 = 0
	int y2 = classifier.Compute(new[] { 0, 0, 1, 1, 0, 0 }); // output is y1 = 0

	int y3 = classifier.Compute(new[] { 2, 2, 2, 2, 1, 1 }); // output is y2 = 1
	int y4 = classifier.Compute(new[] { 2, 2, 1, 1 });       // output is y2 = 1

	int y5 = classifier.Compute(new[] { 0, 0, 1, 3, 3, 3 }); // output is y3 = 2
	int y6 = classifier.Compute(new[] { 2, 0, 2, 2, 3, 3 }); // output is y3 = 2

	Console.WriteLine("y1 = {0}", y1);
	Console.WriteLine("y2 = {0}", y2);
	Console.WriteLine("y3 = {0}", y3);
	Console.WriteLine("y4 = {0}", y4);
	Console.WriteLine("y5 = {0}", y5);
	Console.WriteLine("y6 = {0}", y6);
}

public static void LearnAndPredictContinuous()
{
	// Create a Continuous density Hidden Markov Model Sequence Classifier
	// to detect a univariate sequence and the same sequence backwards.
	double[][] sequences = new double[][]
	{
		new double[] { 0,1,2,3,4 }, // This is the first  sequence with label = 0
		new double[] { 4,3,2,1,0 }, // This is the second sequence with label = 1
	};

	// Labels for the sequences
	int[] labels = { 0, 1 };

	// Creates a new Continuous-density Hidden Markov Model Sequence Classifier
	//  containing 2 hidden Markov Models with 2 states and an underlying Normal
	//  distribution as the continuous probability density.
	Gaussian density = new Gaussian();
	var classifier = new HiddenMarkovClassifier(2, new Ergodic(2), density);

	// Create a new learning algorithm to train the sequence classifier
	var teacher = new HiddenMarkovClassifierLearning(classifier,

		// Train each model until the log-likelihood changes less than 0.001
		modelIndex => new BaumWelchLearning(classifier.Models[modelIndex])
		{
			Tolerance = 0.0001,
			Iterations = 0
		}
	);

	// Train the sequence classifier using the algorithm
	teacher.Run(sequences, labels);

	// Calculate the probability that the given
	//  sequences originated from the model
	double likelihood;

	// Try to classify the first sequence (output should be 0)
	int c1 = classifier.Compute(sequences[0], out likelihood);
	Console.WriteLine("c1: {0}", c1);

	// Try to classify the second sequence (output should be 1)
	int c2 = classifier.Compute(sequences[1], out likelihood);
	Console.WriteLine("c2: {0}", c2);
}
```

repository: mwbaassiri/cs-hidden-markov-models

repo_name: mwbaassiri/cs-hidden-markov-models


text:
---
published: true
title: Parsing Markdown
tags: markdown compiler
---
> [kramdown](https://kramdown.gettalong.org/) - fast, pure-Ruby Markdown-superset converter

ChatGPT is able to mostly code that parser itself while guided.

{% highlight ruby %}
# mostly written by ChatGPT
require 'kramdown'

def extract_urls(el, urls)
    if el.type == :a
        urls << "#{el.attr['href']} - (#{  el.children[0].value})"
    end

    el.children.each { |child| extract_urls(child, urls) }
end

  def extract_markdown_urls(markdown_text)
    doc = Kramdown::Document.new(markdown_text)
    urls = []
    extract_urls(doc.root, urls)
    urls
  end

markdown_text = File.read("input.md")
urls = extract_markdown_urls(markdown_text)

puts urls
{% endhighlight %}

Alternatives
- [ redcarpet](https://github.com/vmg/redcarpet) - a Ruby library for Markdown processing

see also
- [What is Markdown?](https://www.markdownguide.org/getting-started) - Created by John Gruber in 2004, Markdown is now one of the world’s most popular markup languages.

license: MIT

branch_name: refs/heads/master

extension: md

license_type: permissive


text:
So, in chapter 7.1 we saw how to download multiple images from Google in a single go, using a few JavaScript functions. But what if we had a few duplicate images amongst them? Why are duplicate images not favourable while creating a dataset for Deep Learning? Well, let's find out.

Having duplicate images in your dataset creates a problem for two reasons:

    It introduces bias into your dataset, giving your deep neural network additional opportunities to learn patterns specific to the duplicates
    It hurts the ability of your model to generalize to new images outside of what it was trained on

While we often assume that data points in a dataset are independent and identically distributed, that’s rarely (if ever) the case when working with a real-world dataset. When training a Convolutional Neural Network, we typically want to remove those duplicate images before training the model.

Secondly, trying to manually detect duplicate images in a dataset is extremely time-consuming and error-prone — it also doesn’t scale to large image datasets. We therefore need a method to automatically detect and remove duplicate images from our deep learning dataset.

Is such a method possible?

It certainly is — and I’ll be covering it in the remainder of this sub-chapter.

# Detect and remove duplicate images from a dataset for deep learning

In the first part of this sub-chapter, you’ll learn why detecting and removing duplicate images from your dataset is typically a requirement before you attempt to train a deep neural network on top of your data.

From there, we’ll review the example dataset created so we can practice detecting duplicate images in a dataset.

We’ll then implement our image duplicate detector using a method called image hashing.

Finally, we’ll review the results of our work and:

    Perform a dry run to validate that our image duplicate detector is working properly
    Run our duplicate detector a second time, this time removing the actual duplicates from our dataset

# Why bother removing duplicate images from a dataset when training a deep neural network?

Typically, you end up with duplicate images in your dataset by:

    Scraping images from multiple sources (e.g., Google, Bing, etc.)
    Combining existing datasets (ex., combining ImageNet with Sun397 and Indoor Scenes)

When this happens you need a way to:

    Detect that there are duplicate images in your dataset
    Remove the duplicates

But that raises the question — why bother caring about duplicates in the first place?

The usual assumption for supervised machine learning methods is that:

    Data points are independent
    They are identically distributed
    Training and testing data are sampled from the same distribution

The problem is that these assumptions rarely (if ever) hold in practice.

What you really need to be afraid of is your model’s ability to generalize.

If you include multiple identical images in your dataset, your neural network is allowed to see and learn patterns from that image multiple times per epoch.

Your network could become biased toward patterns in those duplicate images, making it less likely to generalize to new images.

Bias and ability to generalize are a big deal in machine learning — they can be hard enough to combat when working with an “ideal” dataset.

Take the time to remove duplicates from your image dataset so you don’t accidentally introduce bias or hurt the ability of your model to generalize.

# Project structure

I’ve included the duplicate image dataset along with the code in this sub-chapter.

Once you extract the ```.zip```, you’ll be presented with the following directory structure:
```
$ tree --dirsfirst --filelimit 10
.
├── dataset [1000 entries]
└── detect_and_remove.py
1 directory, 1 file
```
As you can see, our project structure is quite simple. We have a ```dataset/``` of 1,000 images (duplicates included). Additionally, we have our ```detect_and_remove.py``` Python script, which is the basis of today’s tutorial.

# Implementing our image duplicate detector

We are now ready to implement our image duplicate detector.

Open up the ```detect_and_remove.py``` script in your project directory, and let’s get to work.

Running our image duplicate detector for our deep learning dataset

Let’s put our image duplicate detector to work.

Start by making sure you have downloaded the source code and example dataset.

From there, open up a terminal, and execute the following command just to verify there are 1,000 images in our ```dataset/``` directory:
```
$ ls -l dataset/*.jpg | wc -l
    1000
```

Let’s now perform a dry run, which will allow us to visualize the duplicates in our dataset:
```
$ python detect_and_remove.py --dataset dataset
[INFO] computing image hashes...
[INFO] hash:<PHONE_NUMBER>732718398
[INFO] hash:<PHONE_NUMBER>5133582635
[INFO] hash:<PHONE_NUMBER>5636363614
```
To actually remove the duplicates from our system, we need to execute ```detect_and_remove.py``` again, this time supplying the ```--remove 1``` command line argument:
```
$ python detect_and_remove.py --dataset dataset --remove 1
[INFO] computing image hashes...
```

We can verify that the duplicates have been removed by counting the number of JPEG Images in the dataset directory:
```
$ ls -l dataset/*.jpg | wc -l
     993
```

Originally, there were 1,000 images in dataset, but now there are 993, implying that we removed the 7 duplicate images.

At this point, you could proceed to train a deep neural network on this dataset.

gha_license_id: MIT


text:
# Change Log
All notable changes to this project will be documented in this file. This change log follows the conventions of [keepachangelog.com](http://keepachangelog.com/).

## [Unreleased]

Lots of quality of life fixes

### Fixed

* Lower count of items in sequence before expanding icon shows [#126](https://github.com/Day8/re-frame-trace/issues/126)
* Only allow one time based filter at a time [#125](https://github.com/Day8/re-frame-trace/issues/125)
* Persist trace expansions when changing tab [#105](https://github.com/Day8/re-frame-trace/issues/105)
* Hide :sub/create traces for cached subscriptions [#93](https://github.com/Day8/re-frame-trace/issues/93)
* Persist filter category settings to localstorage [#121](https://github.com/Day8/re-frame-trace/issues/121)
* Highlight events in trace window [#106](https://github.com/Day8/re-frame-trace/issues/106)

## [0.1.14] - 2017-12-19

### Added

* New state snapshotting feature: Click the camera to snapshot app-db, and click the load button to restore to your snapshot. Snapshots only persist for the browser session, they aren't available after reloading the browser.
* X button to remove app-db paths

### Changed

* Switched from LESS to Garden styles. Now interactive development and debugging of re-frame-trace is even faster.
* Reopen/reattach external popup windows when reloading host application
* Reorganise namespace layout and remove (hidden) subviz panel and D3 dependency. This may return in the future though.
* Only sort subscriptions by the subscription key, not the whole vector
* Visual tweaks

### Fixed

* Set panel width correctly so you can see the whole panel at all times. Previously the right edge was just off screen.

## [0.1.13] - 2017-11-23

**N.B.** To make the best use of the new subscriptions panel, you also need to update to re-frame v0.10.3-alpha1.

### Added

* Bright yellow text when re-frame trace is preloaded, but tracing is not enabled.
* Add preliminary subscriptions panel. Still a bit buggy and not very pretty, but it should be useful even in this state.

## [0.1.12] - 2017-11-16

### Added

* Preliminary support for React 16 component paths. [#89](https://github.com/Day8/re-frame-trace/pull/89)
* External popout window. [#92](https://github.com/Day8/re-frame-trace/issues/92)
* Namespace aware truncation in the traces panel. [#100](https://github.com/Day8/re-frame-trace/pull/100)
* App DB expansions and contractions now persist across reloads. Note the [limitations](https://github.com/Day8/re-frame-trace/blob/master/docs/README.md#app-db-path-expansions) with this feature.

### Changed

* Hide index spans (the blocks next to each element counting the number of elements in a collection)

## [0.1.11] - 2017-10-27

### Changed

* Rename app-state panel to app-db
* Visual improvements to data viewer

## [0.1.10] - 2017-10-27

### Removed

* Subvis panel has been commented out while it is in a broken state.

## [0.1.9] - 2017-10-27

### Added

* Limit captured traces to a maximum of 4,000 to prevent performance slowdowns.

### Changed

* Speed up animations for autoscrolling to bottom of list
* Print full keyword for operation name

### Fixed

* Bad merge for handling window width changing.

## [0.1.8] - 2017-10-25

### Added

* Add a new App State panel. This lets you visualise your application state and inspect it. You can add multiple selections to view a subset of your application state.

### Fixed

* Handle window viewport size changing and other dragging bugs. This _should_ be the last of them, but open an issue if you still find glitches when resizing.

## [0.1.7] - 2017-09-25

### Changed

* Improve style resetting
* Visual improvements

### Fixed

* Fix panel resizing behaviour under edge cases

## [0.1.6] - 2017-09-05

### Added

* Log trace data to the console when clicking on it.

## [0.1.5] - 2017-08-31

### Changed

* Put CSS in resources directory

## [0.1.4] - 2017-08-31

### Added

* Save settings to localstorage to persist across reloads
* Autoscroll to the bottom of the traces panel when at the bottom
* Use LESS for CSS

## [0.1.3] - 2017-08-24

### Added

* Show/hide traces

## [0.1.2] - 2017-08-16

### Changed

* Move styles and components to trace folder

## [0.1.1] - 2017-08-16

### Added

* Add event filtering
* Pin the filter input to the top of the panel

### Changed

* Improve installation instructions

### Fixed

* Fix panel resizing

## [0.1.0] - 2017-05-02

### Changed

* Artifact coordinates changed from `day8.re-frame/abra` to `day8.re-frame/trace`.

## [0.0.9] - 2017-05-02

### Changed

* The preloads namespace now adds the tracing panel to the DOM automatically, so you don't need to change any of your app code to bring it in. [#14](https://github.com/Day8/re-frame-trace/pull/14) via [Dexter Gramfors](https://github.com/Dexterminator).
  **Migration steps:** Remove any explicit rendering instructions for `day8.re-frame.trace/devtools` in your app, as this is automatically added now.

## [0.0.8] - 2017-04-13

### Added

* There is now a preload namespace you can use to configure re-frame-trace. Install details are in the README.md. [#13](https://github.com/Day8/re-frame-trace/pull/13) via [Dexter Gramfors](https://github.com/Dexterminator).

## [0.0.7] - 2017-04-13

This was a botched deploy. See 0.0.8.

## [0.0.6] - 2017-03-28

### Added

* Added getting started instructions to README.

repository: hkrishnan/re-frame-trace

repo_name: hkrishnan/re-frame-trace


text:
# Changelog

## 1.4.0-dev

## 1.3.0

* Expose `path` and `method` on endpoints
* Fix issue where references that matched a substring of the path were not being resolved
  (e.g. "/definitions/PersonInfo" in "/definitions/PersonInfoResponse/schema")

## 1.2.3

* Handle circular references during resolution

## 1.2.2

* Use `json-schema` to validate meta schema

## 1.2.1

* Use `Addressable::URI.unencode` instead of obsoleted `URI.decode`

## 1.2.0

* Make all response headers optional

## 1.1.2

* Match path to correct endpoint when many similar paths exist

## 1.1.1

* Handle invalid URLs in `endpoint`

## 1.1.0

* Bump JsonSchema version

## 1.0.1

* Correctly handle known responses with empty schemas

## 1.0.0

* Initial release


text:
# INDEC-Export-Import-Data-Extract
Selenium Web Scrapper to extract export and import information from [INDEC query](https://comex.indec.gob.ar/search)

gha_language: Python


text:
[![Cylon.js](https://cdn.rawgit.com/hybridgroup/cylon-site/master/source/images/elements/cylon.png)](http://cylonjs.com)

Cylon.js is a JavaScript framework for robotics and physical computing built on
top of Node.js.

It provides a simple, but powerful way to create solutions that incorporate
multiple, different hardware devices concurrently.

Want to use Ruby on robots? Check out our sister project, [Artoo][].

Want to use Golang to power your robots? Check out our sister project,
[Gobot][].

[Artoo]: http://artoo.io
[Gobot]: http://gobot.io

## Build Status:

[![Build Status](https://secure.travis-ci.org/hybridgroup/cylon.png?branch=master)](http://travis-ci.org/hybridgroup/cylon) [![Code Climate](https://codeclimate.com/github/hybridgroup/cylon.png)](https://codeclimate.com/github/hybridgroup/cylon) [![Code Climate](https://codeclimate.com/github/hybridgroup/cylon/coverage.png)](https://codeclimate.com/github/hybridgroup/cylon)

## Examples

### Arduino + LED

The below example connects to an Arduino over a serial connection, and blinks an
LED once per second.

The example requires that the Arduino have the Firmata sketch installed; which
can be obtained either through the Ardunio IDE or the `cylon arduino upload
firmata` command available in [cylon-cli][].

```javascript
var Cylon = require('cylon');

// define the robot
var robot = Cylon.robot({
  // change the port to the correct one for your Arduino
  connection: { name: 'arduino', adaptor: 'firmata', port: '/dev/ttyACM0' },
  device: { name: 'led', driver: 'led', pin: 13 },

  work: function(my) {
    every((1).second(), my.led.toggle);
  }
});

// connect to the Arduino and start working
robot.start();
```

### Parrot ARDrone 2.0

```javascript
var Cylon = require('cylon');

Cylon.robot({
  connection: { name: 'ardrone', adaptor: 'ardrone', port: '<IP_ADDRESS>' },
  device: { name: 'drone', driver: 'ardrone' },

  work: function(my) {
    my.drone.takeoff();
    after((10).seconds(), my.drone.land);
    after((15).seconds(), my.drone.stop);
  }
}).start();
```

### Cat Toy (Leap Motion + Digispark + Servos)

```javascript
var Cylon = require('cylon');

Cylon.robot({
  connections: [
    { name: 'digispark', adaptor: 'digispark'},
    { name: 'leapmotion', adaptor: 'leapmotion', port: '<IP_ADDRESS>:6437' }
  ],

  devices: [
    {name: 'servo1', driver: 'servo', pin: 0, connection: 'digispark'},
    {name: 'servo2', driver: 'servo', pin: 1, connection: 'digispark'},
    {name: 'leapmotion', driver: 'leapmotion', connection: 'leapmotion'}
  ],

  work: function(my) {
    my['x'] = 90;
    my['z'] = 90;

    my.leapmotion.on('hand', function(hand) {
      my['x'] = hand.palmX.fromScale(-300, 300).toScale(30, 150);
      my['z'] = hand.palmZ.fromScale(-300, 300).toScale(30, 150);
    });

    every(100, function() {
      my.servo1.angle(my['x']);
      my.servo2.angle(my['z']);

      console.log("Current Angle: " + my.servo1.currentAngle() + ", " + my.servo2.currentAngle());
    });
  }
}).start();
```

### Multiple Spheros + API Server

```javascript
var Cylon = require('cylon');

// tell the API server to listen for requests at
// https://localhost:4000
Cylon.api({ port: 4000 });

var bots = [
  { port: '/dev/rfcomm0', name: 'Thelma' },
  { port: '/dev/rfcomm1', name: 'Louise' }
];

var SpheroBot = function() {};

SpheroBot.prototype.connection = { name: "sphero", adaptor: "sphero" };
SpheroBot.prototype.device = { name: "sphero", driver: "sphero" };

SpheroBot.prototype.work = function(my) {
  every((1).second(), function() {
    console.log(my.name);
    my.sphero.setRandomColor();
    my.sphero.roll(60, Math.floor(Math.random() * 360));
  });
};

for (var i = 0; i < bots.length; i++) {
  var bot = bots[i];
  var robot = new SpheroBot();

  robot.connection.port = bot.port;
  robot.name = bot.name;

  Cylon.robot(robot);
}

// start up all robots at once
Cylon.start();
```

## Hardware Support

Cylon.js has an extensible syntax for connecting to multiple, different hardware
devices. The following 21 platforms are currently supported:

- [Ardrone](http://ardrone2.parrot.com/) <==> [Adaptor/Drivers](https://github.com/hybridgroup/cylon-ardrone)
- [Arduino](http://www.arduino.cc/) <==> [Adaptor](https://github.com/hybridgroup/cylon-firmata)
- [Arduino YUN](http://arduino.cc/en/Main/ArduinoBoardYun?from=Products.ArduinoYUN) <==> [Adaptor](https://github.com/hybridgroup/cylon-firmata)
- [Beaglebone Black](http://beagleboard.org/Products/BeagleBone+Black/) <==> [Adaptor](https://github.com/hybridgroup/cylon-beaglebone)
- [Crazyflie](http://www.bitcraze.se/) <==> [Adaptor/Driver](https://github.com/hybridgroup/cylon-crazyflie)
- [Digispark](http://digistump.com/products/1) <==> [Adaptor](https://github.com/hybridgroup/cylon-digispark)
- [Joystick](http://en.wikipedia.org/wiki/Joystick) <==> [Adaptor/Driver](https://github.com/hybridgroup/cylon-joystick)
- [Keyboard](http://en.wikipedia.org/wiki/Computer_keyboard) <==> [Adaptor/Driver](https://github.com/hybridgroup/cylon-keyboard)
- [Leap Motion](https://www.leapmotion.com/) <==> [Adaptor/Driver](https://github.com/hybridgroup/cylon-leapmotion)
- [Nest](http://nest.com/) <==> [Adaptor/Drivers](https://github.com/hybridgroup/cylon-nest)
- [Neurosky](http://store.neurosky.com/products/mindwave-mobile) <==> [Adaptor/Driver](https://github.com/hybridgroup/cylon-neurosky)
- [OpenCV](http://opencv.org/) <==> [Adaptor/Drivers](https://github.com/hybridgroup/cylon-opencv)
- [Pebble](http://www.getpebble.com/) <==> [Adaptor/Driver](https://github.com/hybridgroup/cylon-pebble)
- [Pinoccio](https://pinocc.io) <==> [Adaptor/Driver](https://github.com/hybridgroup/cylon-pinoccio)
- [Rapiro](http://www.rapiro.com/) <==> [Adaptor/Driver](https://github.com/hybridgroup/cylon-rapiro)
- [Raspberry Pi](http://www.raspberrypi.org/) <==> [Adaptor](https://github.com/hybridgroup/cylon-raspi)
- [Salesforce](http://www.force.com/) <==> [Adaptor/Driver](https://github.com/hybridgroup/cylon-force)
- [Skynet](http://skynet.im/) <==> [Adaptor](https://github.com/hybridgroup/cylon-skynet)
- [Spark](http://www.spark.io/) <==> [Adaptor](https://github.com/hybridgroup/cylon-spark)
- [Sphero](http://www.gosphero.com/) <==> [Adaptor/Driver](https://github.com/hybridgroup/cylon-sphero)
- [Tessel](https://tessel.io/) <==> [Adaptor/Driver](https://github.com/hybridgroup/cylon-tessel)

Our implementation of GPIO (General Purpose Input/Output) allows for a shared
set of drivers supporting a number of devices:

  - [GPIO](https://en.wikipedia.org/wiki/General_Purpose_Input/Output) <=> [Drivers](https://github.com/hybridgroup/cylon-gpio)
    - Analog Sensor
    - Button
    - Continuous Servo
    - Direct Pin
    - IR Rangefinder
    - LED
    - MakeyButton
    - Motor
    - Maxbotix Ultrasonic Range Finder
    - Servo

Additionally, we also support a number of I2C (Inter-Integrated Circuit) devices
through a shared `cylon-i2c` module:

  - [I2C](https://en.wikipedia.org/wiki/I%C2%B2C) <=> [Drivers](https://github.com/hybridgroup/cylon-i2c)
    - BlinkM
    - BMP180
    - HMC6352 Digital Compass
    - LCD Display
    - MPL115A2 Barometer/Thermometer
    - MPU6050

We'll also have many more platforms and drivers coming soon, [follow us on
Twitter][Twitter] for updates.

[Twitter]: https://twitter.com/cylonjs

## Getting Started

### Installation

All you need to get started on a new robot is the `cylon` module:

    npm install cylon

With the core module installed, now install the modules for whatever hardware
support you need. For the Arduino + LED blink example, we'll need the 'firmata',
'gpio', and 'i2c' modules:

    npm install cylon-firmata cylon-gpio cylon-i2c

## CLI

Cylon uses the Gort [http://gort.io](http://gort.io) Command Line Interface (CLI) so you can access important features right from the command line. We call it "RobotOps", aka "DevOps For Robotics". You can scan, connect, update device firmware, and more!

Cylon also has its own CLI to generate new robots, adaptors, and drivers. You can check it out at [https://github.com/hybridgroup/cylon-cli](https://github.com/hybridgroup/cylon-cli).

## Documentation

We're busy adding documentation to our website, check it out at
[cylonjs.com/documentation][docs].

If you want to help with documentation, you can find the code for our website at
on the [https://github.com/hybridgroup/cylon-site](https://github.com/hybridgroup/cylon-site).

[docs]: http://cylonjs.com/documentation
[docs site]: https://github.com/hybridgroup/cylon-site

## Contributing

* All patches must be provided under the Apache 2.0 License
* Please use the -s option in git to "sign off" that the commit is your work and you are providing it under the Apache 2.0 License
* Submit a Github Pull Request to the appropriate branch and ideally discuss the changes with us in IRC.
* We will look at the patch, test it out, and give you feedback.
* Avoid doing minor whitespace changes, renamings, etc. along with merged content. These will be done by the maintainers from time to time but they can complicate merges and should be done seperately.
* Take care to maintain the existing coding style.
* Add unit tests for any new or changed functionality & lint and test your code using `make test` and `make lint`.
* All pull requests should be "fast forward"
  * If there are commits after yours use “git rebase -i <new_head_branch>”
  * If you have local changes you may need to use “git stash”
  * For git help see [progit](http://git-scm.com/book) which is an awesome (and free) book on git

## Release History

Version 0.18.0 - Updates Robot and Driver commands structure

Version 0.17.0 - Updates to API to match CPPP-IO spec

Version 0.16.0 - New IO Utils, removal of Utils#bind, add Adaptor#_noop method.

Version 0.15.1 - Fixed issue with the API on Tessel

Version 0.15.0 - Better halting, cleaner startup, removed 'connect' and 'start'
                 events, and misc other cleanups/refactors.

Version 0.14.0 - Removal of node-namespace and misc. cleanup

Version 0.13.3 - Fixes bug with disconnect functions not being called.

Version 0.13.2 - Use pure Express, adds server-sent-events, upd API.

Version 0.13.1 - Add API authentication and HTTPS support

Version 0.13.0 - Set minimum Node version to 0.10.20, add utils to global namespace and improve initialization routines

Version 0.12.0 - Extraction of CLI tooling

Version 0.11.2 - bugfixes

Version 0.11.0 - Refactor into pure JavaScript

Version 0.10.4 - Add JS helper functions

Version 0.10.3 - Fix dependency issue

Version 0.10.2 - Create connections convenience vars, refactor config loading

Version 0.10.1 - Updates required for test driven robotics, update Robeaux version, bugfixes

Version 0.10.0 - Use Robeaux UX, add CLI commands for helping connect to devices, bugfixes

Version 0.9.0 - Add AngularJS web interface to API, extensible commands for CLI

Version 0.8.0 - Refactored Adaptor and Driver into proper base classes for easier authoring of new modules

Version 0.7.0 - cylon command for generating new adaptors, support code for better GPIO support, literate examples

Version 0.6.0 - API exposes robot commands, fixes issues in driver/adaptor init

Version 0.5.0 - Improve API, add GPIO support for reuse in adaptors

Version 0.4.0 - Refactor proxy in Cylon.Basestar, improve API

Version 0.3.0 - Improved Cylon.Basestar, and added API

Version 0.2.0 - Cylon.Basestar to help develop external adaptors/drivers

Version 0.1.0 - Initial release for ongoing development

## License

Copyright (c) 2013-2014 The Hybrid Group. Licensed under the Apache 2.0 license.

extension: markdown


text:
# Graphics API Server (GAPIS)

The Graphics API Server is the central component of the GAPID tool suite, providing the interface between the client and the replay systems.

GAPIS has been designed to support multiple graphics APIs.
The client does not need to have any knowledge of any graphics APIs, making implementing new clients relatively trivial.

## API deep-knowledge

GAPIS uses the graphics API packages declared in [gapis/api](api) to have a deep understanding of each command in the API and how it interacts with the state.

Most of the code in these graphics API packages is generated from a corresponding graphics API file, which describes the internal state of a graphics driver, each command exposed by the API, and how each command interacts with the internal driver state. When these API files are combined with the API compiler and template files, we produce Go code that:

* Describes the classes, enums, and bitfields that are used in the graphics API. Each type declared in the API file produces a corresponding Go type via the api.go.tmpl template file.

* Describes the structure of the graphics library's State object. These are formed from the global fields in the API file.

* Describes each of the commands declared in the API. Each command in the API file produces a Go struct that implements the `api.Cmd` interface.

* Implements the `Cmd.Mutate()` method on each command, which performs the necessary mutations to the state object provided in the call.

## State mutation

The generated `Cmd.Mutate()` methods simulate the execution of the command by mutating the provided State object, and applying any memory Observations to the memory pools.

This state mutation is essential for much of the GAPIS functionality - it performs the work to get a default-initialized state to a particular point in the capture.

For example, the client can ask for a Report describing all the issues and warnings found in a capture by performing state mutation over the capture’s command list. The `Mutate()` function will log any issues it finds, and the service function keeps track of the command that raised the issue. The full collection of issues is handed back to the client in the Report structure.

The client may also ask for the state object at a particular point in the stream to display to the user. Again, GAPIS uses a default initialized State object, then calls `Mutate()` on each of the commands in turn to bring the State to the requested point in the stream, before encoding the State object and sending to the client.

Because the state mutation is so frequently used, and because captures typically contain hundreds of thousands of commands, state mutation needs to be fast in order for GAPIS to be responsive.

One of the ways GAPIS tries to reduce the amount of work needed to simulate a command is to store records of memory writes on buffers instead of allocating and copying real buffer data. Many of the commands found in a graphics API involve copying N bytes from one memory pool to another. We find that in most of the common uses of the state mutator, we may require a read of this data far less frequently than we perform copy operations.

In the rare cases we need to get at the actual binary data “written” to a particular buffer, we scan backwards through the records of the memory pool of interest, finding records that overlap the queried memory range, and applying the buffer records by copying into a real, target byte array.

## Replay

Some of the most important (and complicated) work performed by GAPIS is the building of the replay opcode stream for a capture.

Replay opcode generation also performed using the `Cmd.Mutate()` method, being handed the state object and replay Builder object.

The `Builder` contains target device memory layout information and exposes a number of high-level functions that generate a stream of `asm.Instructions`.

Each `Instruction` may result in one or more `Opcodes`, which are packaged into the payload which the replay VM executes.

Stage |                     | Description
----- | ------------------- | -----------
`1`   | `Replay()`          | `Replay()` is called on each command in the final steam, which can make a number of builder method calls.
`2`   | `Builder.foo()`     | Each builder method can produce a number of instructions.
`3`   | Instructions        | After all instructions have been generated, each instruction is processed to create a number of opcodes.
`4`   | Opcodes             | The opcodes are written directly to a byte stream in the payload.
`5`   | Payload             | And then the payload is sent to the replay system.

## Remappings

The most challenging aspect of generating a replay opcode stream is dealing with non-deterministic values and pointers. We have 3 subtly different remapping problems:

### Non-deterministic object identifier generation.

Commands like `glGenTextures` generate texture "names". The OpenGL ES specification does not state how these identifiers should be generated, so the "names" returned cannot be predicted at opcode generation time. To deal with this, we need to generate instructions to store replay- generated identifiers into a table that can be fetched for later commands that use that identifier. We call this process “*id-remapping*” and is explicitly handled on specific types.

For example, consider the following captured commands:

```
glCreateShader(GL_VERTEX_SHADER) ⇒ 5
...
glCompileShader(5)
```

The generated VM opcodes would look something like:

```
PUSH_I<s32>(GL_VERTEX_SHADER)
CALL(glCreateShader)
STORE_V<s32>(0x1234) // table address to store ID of shader 5
...
LOAD_V<s32>(0x1234)
CALL(glCompileShader)
```

### Application owned memory addresses

Application-allocated memory is often handed to the graphics driver. Commands like `glTexImage2D` has a pointer to an application-allocated buffer as the data parameter. In order to replay these commands correctly, the replay system needs to allocate chunks of memory big enough to hold the buffers, populate them with observed memory, and finally, hand allocation-relative pointers to the replay target’s driver.

In order to reduce the number of small allocations required in a replay, a pass is performed over all the commands, and all the overlapping observed memory ranges in the capture are merged into the fewest number of allocations required to encapsulate all observations. Replay pointers are then adjusted to be relative to the start address of the merged block.

![Memory Observations](docs/observations.svg)

### Memory Observations

Most of this logic is handled by the replay Builder via the `volatileMemoryLayout` structure.

### Driver-owned memory addresses

Some graphics API functions return pointers to graphics driver memory (for example glMapBufferRange). Unlike the pointer remappings performed for application-allocated memory, the replay system needs to remap the observed memory to addresses relative to pointers returned by the driver.

In this situation, the pointer returned by the function mapping the memory is stored to table, and when there are observations made within the “mapped” memory range, the replay logic performs a table load and then ADDs the relative offset.

For example, consider the following captured commands:

```
glMapBufferRange(GL_ARRAY_BUFFER, 0, 0x1000, GL_MAP_WRITE_BIT) ⇒ 0x400000
...
glUnmapBuffer(GL_ARRAY_BUFFER)
  Read-Observation{ Address: 0x400800, ResourceID: 0x10 }
```

The generated VM opcodes would look something like:

```
PUSH_I<s32>(GL_MAP_WRITE_BIT) // access
PUSH_I<s32>(0x1000)           // size
PUSH_I<s32>(0)                // offset
PUSH_I<s32>(GL_ARRAY_BUFFER)  // target
CALL(glMapBufferRange)
STORE_V<s32>(0x1234)  // table address to store remapped pointer 0x400000
...
LOAD_V<s32>(0x1234)
ADD<abs-ptr>(0x800)  // 0x400800 - 0x400000
RESOURCE(0x10)       // read-observation
PUSH_I<s32>(GL_ARRAY_BUFFER)
CALL(glUnmapBuffer)
```

Again, most of this logic is handled by `Builder`

### Compatibility

One of GAPID's core goals is to support replays on targets different to those used to create the capture. For example, we may want to capture an OpenGL ES 2.0 application, and replay the capture using a desktop OpenGL 4.0 context.

OpenGL has the concept of core and compatibility profiles - as functions are marked as deprecated between OpenGL versions, the compatibility profile will still permit the deprecated functionality, the core profile will error. Most drivers and devices will offer compatibility profiles, but certain targets, including macOS, only supports core profiles for OpenGL 3+.

In order to support replay of older profiles on core profile devices, we need to perform command translations when building the opcodes for replay on the target device. These translations are performed by the compatibility layer.

### Unsupported commands

Our compatibility layer will compare the version of OpenGL used when taking the capture to the version of OpenGL we are building the replay for. Commands that are not supported the target’s OpenGL profile are substituted with functionally equivalent commands that are compatible with the replay device.

Take for example:

```
void glVertexAttribPointer(GLuint        index,
                           GLint         size,
                           GLenum        type,
                           GLboolean     normalized,
                           GLsizei       stride,
                           const GLvoid* pointer)
```

In OpenGL ES 2, (and OpenGL 2) the pointer argument can either be a client- side pointer to the vertex data or an offset on a `GL_ARRAY_BUFFER_BINDING`.

However, in OpenGL 4 core profile the pointer argument is always treated as an offset on a mandatory array buffer.

In this particular case, the compatibility layer will create a temporary array buffer filled with the client memory each time the vertex attribute is used.

### Texture support

Many Android devices support compression formats exposed via extensions that are only supported on particular GPUs. In order to replay captures using texture formats unsupported on the replay device, our compatibility layers can convert textures on-the-fly to formats accepted by the target hardware.

For example if the target device did not support the GL_ETC1_RGB8_OES texture format, then:

```
glCompressedTexImage2D(GL_TEXTURE_2D,       // target
                       0,                   // level
                       GL_ETC1_RGB8_OES,    // internal format
                       64, 64,              // width, height
                       0,                   // border
                       0x800,               // size
                       0x10000)             // data
```

May be substituted with:

```
glTexImage2D(GL_TEXTURE_2D,       // target
             0,                   // level
             GL_RGBA,             // internal format
             64, 64,              // width, height
             0,                   // border
             GL_RGBA,             // format
             GL_UNSIGNED_BYTE,    // type
             0x40000)             // data
```

Where decompressed texture data is stored at 0x40000.

### GLSL shader source conversion

Much like the OpenGL and OpenGL ES commands, there are many different and often subtle differences in the shader language between versions. Our compatibility layers will reformat the shader source code embedded in the capture streams to satisfy the target device.

Example transformations include:

* Replacing the `attribute` keyword with `out` for vertex shaders and in for fragment shaders.
* Replacing the `texture2D` keyword with texture`.
* Stripping `highp`, `mediump`, `lowp` precision specifiers.
* Adding or updating `#version` headers.

## Client service

GAPIS offers simultaneous connections from any number of clients. All client-server communication is performed through the [gRPC service interface](service/service.proto).

The service relies heavily on resource identifiers and paths to refer to computed data. This means the list of RPC functions is relatively small when considering the number of different queries the client can make.

## Paths

Paths are well defined set of encodable objects that can be chained to describe a reference to some object or data held by the server. The list of path object types can be found in [gapis/service/path/path.proto](service/path/path.proto).

A path is resolved using the RPC `Get()`, and the object returned is guaranteed to be the same with subsequent calls.

A path can be used in the RPC `Set()` call to change a value or object. As a path refers to an immutable object, the `Set()` RPC returns a new path to the changed data - typically this involves forking the capture.

Example paths:

To access the state after the 50th command of the capture with ID 0x1234ABCD, we’d use the path:

`Capture(0x1234ABCD).Commands[50].State`

To access the observed memory range 0x10000 to 0x20000 after the 30th command in capture with ID 0x1234ABCD, we’d use the path:

`Capture(0x1234ABCD).Commands[30].MemoryRange<0x10000-0x20000>`

## Resource Identifiers

A resource identifier is a sequence of 20 bytes, usually calculated as a SHA1 of the data it represents. Resource identifiers are used by Blob paths to access a chunk of binary data.

repository: google/gapid

gha_language: Go

repo_name: google/gapid


text:
Copyright 2016 Redsaz

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

! Endpoints

!! Design

All endpoints which are accessed via GET can also be accessed by browser. This is in contrast with other designs which split the API away from the browser endpoints (e.g. REST: "api/v1/logs" vs. Browser: "logs"). Additionally, there is no versioning in the endpoints either (e.g. "api/v1").

Instead, these are controlled by the content-type of the request. The current content type is application/x-embeddedrest-v1+json. (Yes, content-types do allow versioning behind it, for example application/x-embeddedrest-v1+json;version=1, which would allow the version to be optional, but this seems a bit too much and the version would be better as required.) https://en.wikipedia.org/wiki/Internet_media_type explains the rules behind it.

Watch the video for reasons why: https://yow.evener.com/yow-2011-1004/domain-driven-design-for-restful-systems-by-jim-webber-1047

HATEOAS is achieved with link headers, rather than in the message body.

!! List

GET /logs - Lists brief summaries of all of the logs contained in the system.

repository: redsaz/lognition

gha_language: Java

repo_name: redsaz/lognition


text:
---
date: '2007-07-24 12:10:07'
layout: post
comments: true
slug: verdammt-slimserver-anyway
status: publish
title: Verdammt slimserver anyway
wordpress_id: '402'
categories:
- Audio, sound and music
- Debian
- Networking
---

I've had a lot of people trying to use my wireless network of late

>
Sidebar: The Airport Extreme can send log messages to your Unix box via the syslog mechanism. Check out 'Base station options'/ 'Logging/NTP' and 'Send base station logging to'

I also set the LED light to blink for traffic, as I find that more useful than always-on.

See [this page](http://www.phfactor.net/wp/2006/01/15/syslog-net-wide-finally/) for syslog setup on Debian.

and I've been trying to close down the hatches. I've always used MAC filtering, so they couldn't get on, but someone keeps trying. Given the rapid repeat, its certainly automated or just OS stupidity, but it annoys me.

Plan of attack:

* Switch from B/G WiFi to G only.

* Enable encryption, WPA2 preferred.

* Disable SSID broadcast.

So. First problem is my version 1 [Squeezebox.](http://slimdevices.com/)

![](http://www.phfactor.net/wp-pics/slim_devices_squeezebox.png)

It's [been trouble before](http://www.phfactor.net/wp/2006/01/26/more-squeezebox-hacking/), but I don't have the dosh to upgrade to the much-nicer v2 or v3. It's WiFi is B-only. Fortunately, I had a Linksys WRT54GS spare, and had [previously setup one as a bridge.](http://www.phfactor.net/wp/2005/12/11/client-mode-on-linksys-wrt54gs/)

Since I wanted to put images in this post, here's a pic of one borrowed from [the Wikipedia page](http://en.wikipedia.org/wiki/WRT54G) on it:

![](http://www.phfactor.net/wp-pics/800px-linksys_wrt54g-wp.jpg)

I found a [slightly better set of instructions](http://forums.anandtech.com/messageview.aspx?catid=36&threadid=1513386&frmKeyword=&STARTPAGE=1&FTVAR_FORUMVIEWTMP=Linear) for bridge mode on AnandTech, and managed to get it working. I found that the newer Talisman firmware worked better than the Alchemy release.

>
Sidenote on firmware and Sveasoft: I am a former subscriber to Sveasoft, and have paid for a year of access. This time, subscription having expired, I downloaded the 'Freeman' version from [this page instead](http://wrt54g.thermoman.de/), which lacks the ultra-stupid MAC-based copy protection. I am fairly certain that Sveasoft has bent or broken GPL on this, and FWIW the Freeman version is working well. Rant over.

After you do all the setup (subnet, DHCP, client-mode wireless) there's just one real gotcha - to get it to work, you have to enable ARP proxying on the Linksys. There's no way to do this from the web interface, and **the setting is erased if you reboot the router.** So you have to

* Enable SSH on the router.

* Upload your SSH key via the web interface.

* SSH in as root, and run

    echo 1 > /proc/sys/net/ipv4/conf/`route | grep default | awk '{print $NF}'`/proxy_arp

Messy, eh?

Once I had that figured out, I wanted to enable encryption. [WEP](http://en.wikipedia.org/wiki/Wired_Equivalent_Privacy) is no good, so I wanted [WPA](http://en.wikipedia.org/wiki/Wi-Fi_Protected_Access) or [WPA2.](http://en.wikipedia.org/wiki/IEEE_802.11i) I'm using an [Airport Express](http://www.apple.com/airportexpress/) which supports all of the above as my base station. Gratuitous picture:

![](http://www.phfactor.net/wp-pics/airport-express.jpg)

However, the Freeman firmware pages use different terminology than the Apple admin program, so I had to google a bit. [This page](http://www.punknix.com/?q=node/62) and [this one](http://www.networkworld.com/net.worker/columnists/2005/0110shaw.html) got me going. You have to

* Select 'Wireless security'

* Select WPA2 personal.

* Select Encryption type as 'WPA only'

* Only now will 'WPA personal' show up as an option; choose it.

* Select 'Pre-shared key' above, and enter your 64 hex secret.

Now, while it reboots, go to the Freeman firmware and select 'WPA-TKIP'. Enter same key, reboot, do the SSH ARP voodoo above, and you're good to go.

(You also have to go around to all your laptops and enter the key into their setups, but that's much easier.)

Once I did this I hit the next problem: Slimserver is not working. I spent hours thinking it was the bridge setup, which was reasonable given the number of places you can make mistakes, but it isn't. For some reason, the current release in debian unstable is borked, and the symptom is that the web interface never loads. You can connect to the port ok but you get no content. I erased the contents of /var/cache/slimserver/MySQL and restarted it, but got the same result.

Oddly, the web interface was working while it rebuilt the database, but somewhere in that process it Just Stopped. No error log that I can find, no errors in /var/cache/slimserver/mysql-error-log.txt either. I'm at a loss.

The other problem that I have is probably my error - I can't load the Linksys web interface unless I'm cabled into it. I enabled the remote admin, but it may be subnet-related or such.

I **really** wish that the ARP proxy setup was a) permanent and b) web-interface-settable. Yeesh.

I also am a bit peeved at Slimserver. It's the program single most likely to break on a Debian dist-upgrade.

Ahh well, maybe with G-only and WPA it'll cut down on the neighbors trying to associate with my base station...

**Update 8/19/07:** Squeezebox got updated and appears to stay up now, so I'm reopening this one. Fixed the missing anandtech URL, sorry. I should also note that the WPA+G change **did** remove the slowdowns and associations -- hooray! Now tackling the ARP proxy and such. Post to follow!

repository: phubbard/fnord

repo_name: phubbard/fnord


text:
  An extension of Graphene which allows you to secure your GraphQL API against malicious queries.

Features included:
- Limit Query Depth

Other features (limiting query complexity, etc) are under development.

### Prerequisites

- [graphql-core](https://github.com/graphql-python/graphql-core)

### Installation

-  `pip install secure-graphene`

### Detailed Example

Given the following schema:

```python
import graphene
from secure_graphene import depth

class Episode(graphene.Enum):
    NEWHOPE = 4
    EMPIRE = 5
    JEDI = 6

class Character(graphene.Interface):
    id = graphene.ID()
    name = graphene.String()
    friends = graphene.List(lambda: Character)
    appears_in = graphene.List(Episode)

    def resolve_friends(self, info):
        # The character friends is a list of strings
        return [get_character(f) for f in self.friends]

class Human(graphene.ObjectType):
    class Meta:
        interfaces = (Character,)

    home_planet = graphene.String()

class Droid(graphene.ObjectType):
    class Meta:
        interfaces = (Character,)

    primary_function = graphene.String()

class Query(graphene.ObjectType):
    hero = graphene.Field(Character, episode=Episode())
    human = graphene.Field(Human, id=graphene.String())
    droid = graphene.Field(Droid, id=graphene.String())

    def resolve_hero(self, info, episode=None):
        return get_hero(episode)

    def resolve_human(self, info, id):
        return get_human(id)

    def resolve_droid(self, info, id):
        return get_droid(id)

```

And the resolver functions and data:

```python
human_data = {}
droid_data = {}

luke = Human(
    id="1000",
    name="Luke Skywalker",
    friends=["1002", "1003", "2000", "2001"],
    appears_in=[4, 5, 6],
    home_planet="Tatooine",
)

vader = Human(
    id="1001",
    name="Darth Vader",
    friends=["1004"],
    appears_in=[4, 5, 6],
    home_planet="Tatooine",
)

han = Human(
    id="1002",
    name="Han Solo",
    friends=["1000", "1003", "2001"],
    appears_in=[4, 5, 6],
    home_planet=None,
)

leia = Human(
    id="1003",
    name="Leia Organa",
    friends=["1000", "1002", "2000", "2001"],
    appears_in=[4, 5, 6],
    home_planet="Alderaan",
)

tarkin = Human(
    id="1004",
    name="Wilhuff Tarkin",
    friends=["1001"],
    appears_in=[4],
    home_planet=None,
)

human_data = {
    "1000": luke,
    "1001": vader,
    "1002": han,
    "1003": leia,
    "1004": tarkin,
}

c3po = Droid(
    id="2000",
    name="C-3PO",
    friends=["1000", "1002", "1003", "2001"],
    appears_in=[4, 5, 6],
    primary_function="Protocol",
)

r2d2 = Droid(
    id="2001",
    name="R2-D2",
    friends=["1000", "1002", "1003"],
    appears_in=[4, 5, 6],
    primary_function="Astromech",
)

droid_data = {"2000": c3po, "2001": r2d2}

def get_character(id):
    return human_data.get(id) or droid_data.get(id)

def get_friends(character):
    return map(get_character, character.friends)

def get_hero(episode):
    if episode == 5:
        return human_data["1000"]
    return droid_data["2001"]

def get_human(id):
    return human_data.get(id)

def get_droid(id):
    return droid_data.get(id)
```
Let's run this query

```python
query_string = """
            query HeroNameAndFriendsQuery {
              hero {
                id
                name
                friends {
                  name
                }
              }
            }
        """
```

By doing the following:

```python
backend = depth.DepthAnalysisBackend(max_depth=2, execute_params={'executor': None})
schema = graphene.Schema(query=Query)
result = schema.execute(query_string, backend=backend)
result.errors
```
Which returns the following:
```python
[Exception('Query is too deep - your depth is 3 and the max depth is 2')]
```

repository: bogdan-zs/secure-graphene

repo_name: bogdan-zs/secure-graphene


text:
# VM scale set based on the Azure Data Science VM
This template creates a VM scale set based on the Azure Marketplace [Windows 2016](https://azuremarketplace.microsoft.com/marketplace/apps/microsoft-ads.windows-data-science-vm?tab=Overview) and [Ubuntu 16.04-LTS](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/microsoft-ads.linux-data-science-vm-ubuntu?tab=Overview) Data Science VMs.

The template creates a scale set, a load balancer and a public IP address. Inbound NAT pools are defined to route incoming connections to the VMs in the set.

Inbound NAT rules are configured as follows:
```
SSH/RDP to the ip/dns address port 50000 to connect to VM#0
SSH/RDP to the ip/dns address port 50001 to connect to VM#1
etc..
```

![Windows 2016 screenshot](../201-vmss-datascience/img/datasciencewin2016.PNG)

## Deploy Windows 2016 Data Science VM to Azure

## Deploy Ubuntu 16.04 Data Science VM to Azure

## Additional information

[Introduction to the Azure Data Science VM](https://docs.microsoft.com/azure/machine-learning/machine-learning-data-science-virtual-machine-overview)

repository: MSBrett/azure-quickstart-templates

gha_language: Shell

repo_name: MSBrett/azure-quickstart-templates


text:
---
layout: post
title:  9. Thinking outside the box - Chromium's defense in "depth"
date: 2020-03-28 21:01:00
description: What is brilliant about Chrome?
---
Ever wondered why there is so much stress on using the right browser?

What do browsers really do? They take code (HTML, JavaScript) written by other people and execute it on your laptop to render the webpage. Sounds scary when you put it that way from security point of view since a user written code can execute anything it wants on your system.

Chromium handles this by separating the rendering engine from the system accessors. Code in chrome has minimum permissions which does not let the rendering code direct access to the file system. It can do so only through APIs (e.g. while adding attachments in email). For a breach in Chromium, an attacker needs to inflitrate both the rendering engine and the system handlers (increasing depth). This method stops innumerable attacks and is almost unbreakable.

Next time a shady page asks you to download a browser, think again.

Here's the paper discussing the architecture -

https://lnkd.in/fNerx-j

branch_name: refs/heads/source

gha_language: JavaScript


text:
| Navigation |||
| --- | --- | ---: |
| [Previous](../../CrossToolChain.md) | [HOME](../../README.md) | [Next](../LinuxHeaders/) |

# File - A tool for discovering the type of a file

The `file` command is used for determining the type of a file or collection of files. More details are covered in the later build for the basic toolchain.

To build it, run the following command as the `builder` user:

```bash
pushd /source
    tar xvf file-5.38.tar.gz
    pushd file-5.38
        ./configure --prefix=/cross-tools
        make
        make install
    popd
popd
```

- [Build JSON](build.json)
- [Build Script](build.sh)

**NOTE:** We will not build the test suits of various packages during the cross-compile phase as they will not accurately run in many cases.

| Navigation |||
| --- | --- | ---: |
| [Previous](../../CrossToolChain.md) | [HOME](../../README.md) | [Next](../LinuxHeaders/) |


text:
# Simple Locale

## Install

    composer require a3gz/simple-locale

## Setup
Create an associative array with the settings having:

1. `languages`: An array of language codes. The codes used here are completely arbitrary.
2. `env`: Several environment settings for each language so the localizer knows how to format things like currency and dates.
3. `dictionaries`: Here we provide the dictionary files where string keys are associated to translations.

A typical settings array would look like this

    $settings = [
      'languages' => ['en'],
      'env' => [
        'en' => [
          /* Numbers */
          'decimals' => 2,
          'decimalPoint' => '.',
          'thousandsSeparator' => ' ',
          /* Currency */
          'symbolBefore' => '$ ',
          'symbolAfter' => '',
          /* Dates */
          'timestamp' => 'Y-m-d H:i:s',
          'short' => 'Y/m/d',
          'long' => 'M j, Y',
          'longer' => 'M j, Y \a\t H:i:s',
        ],
        'fr' => false,
        'es' => false,
      ],
      'dictionaries' => [
        function($languageCode) {
          $base = dirname(__DIR__) . '/domain/locale';
          $dn = "{$base}/{$languageCode}/dictionary.php";
          if (is_readable($dn)) {
            return @include($dn);
          }
        },
      ],
    ];

### About dictionaries
A dictionary is a PHP file that returns an associative array having string keys and translations as shown below:

    return [
      'hello world' => 'Hola mundo!',
    ];

An application may have one or more dictionaries and this is why the `dictionaries` section is expected to be an array. The demo above shows the simplest form where all translations are located in one file.

Although it isn't required, it is suggested that dictionaries are located inside a directory named as the corresponding language code defined under the `languages` section. Note the part where the full path to the dictionary file is built:

    $dn = "{$base}/{$code}/dictionary.php";

## Usage

    $locale = new Locale($settings);

    echo $locale->dictionary->say('hello world');
    echo $locale->currency->format12.50);

gha_language: PHP


text:
# AngularJS OAuth2 Sample App

This project is the sample app for the [angularjs-oauth2 bower package](https://github.com/JamesRandall/AngularJS-OAuth2).

This [blog post](http://www.azurefromthetrenches.com/?p=2241) gives an overview.


text:
# Change Log

## 0.24.0 ([#47](https://git.mobcastdev.com/Platform/common-spray/pull/47) 2015-01-02 11:30:51)

Default rejection handler

### New Feature

- `RejectionHandler` that converts Spray rejections into responses with JSON error messages

## 0.23.0 ([#46](https://git.mobcastdev.com/Platform/common-spray/pull/46) 2014-12-01 10:57:11)

v2 API Image representation

### New feature

- v2 API `Image` case class.

## 0.22.2 ([#45](https://git.mobcastdev.com/Platform/common-spray/pull/45) 2014-11-27 11:54:11)

Fixed a bug and updated metrics library

### Bug fixes

- The error body for `RejectedExecutionException` being thrown is now
correctly reported as service unavailable rather than internal server
error.

### Improvements

- Updated the version of the metrics library, and a couple of other
shared libraries.

## 0.22.1 ([#44](https://git.mobcastdev.com/Platform/common-spray/pull/44) 2014-11-26 18:02:35)

Bumped versions

### Improvements

- Bumped versions

## 0.22.0 ([#43](https://git.mobcastdev.com/Platform/common-spray/pull/43) 2014-11-26 17:36:02)

Returns right status for full thread pools

### New features

- Now returns `503 Service Unavailable` if the service code throws a
`RejectedExecutionException`, which occurs if an item cannot be added
to a bounded thread pool.

## 0.21.0 ([#42](https://git.mobcastdev.com/Platform/common-spray/pull/42) 2014-11-24 14:44:18)

Changed logging to use scala-logging (again...)

### Breaking Changes

- Logging now uses the Typesafe scala-logging library rather than the
SLF4J loggers. This _should_ work correctly this time…
- Now only supports Scala 2.11 as the logging library does not work
with 2.10

### Notes

- The json4s version is now 3.2.11 which adds a new method in
`DefaultFormats`; if you have references to older versions then this
can result in `AbstractMethodException` so make sure all your versions
are consistent. `common-json` has already been updated.

## 0.20.0 ([#41](https://git.mobcastdev.com/Platform/common-spray/pull/41) 2014-11-19 11:49:08)

Json error responses

### Breaking Change

- Added support for v2 API error messages in exception handler used by monitor directive. For example, `500 Internal Server Error` responses now contain JSON body:
```json
{"code":"InternalServerError","developerMessage":"There was an internal server error."}
```

## 0.19.1 ([#39](https://git.mobcastdev.com/Platform/common-spray/pull/39) 2014-11-13 17:27:25)

Monitoring directive logger fix

### Bugfix

Typesafe Logging library has a bizarre bug that results in attempts to log with a default implementation of `com.typesafe.scalalogging.Logger` that has no implementation when the logger is passed as a parameter to other methods.  :see_no_evil:

## 0.19.0 ([#37](https://git.mobcastdev.com/Platform/common-spray/pull/37) 2014-11-13 12:02:12)

V2 links

### New feature

- Added `Link` class

## 0.18.0 ([#36](https://git.mobcastdev.com/Platform/common-spray/pull/36) 2014-11-10 15:42:02)

Monitoring directive now uses typesafe Logger

### Breaking changes

- The `monitor` directive now expects a
`com.typesafe.scalalogging.Logger` rather than an `org.slf4j.Logger`
instance.

## 0.17.5 ([#35](https://git.mobcastdev.com/Platform/common-spray/pull/35) 2014-11-04 15:31:12)

Use strict mode option parsing by default

patch

- Use strict mode for parsing by default.

## 0.17.4 ([#34](https://git.mobcastdev.com/Platform/common-spray/pull/34) 2014-11-04 09:30:30)

Support Validation of Optional fields

patch

- Optionally throw a MappingException when a value for a key of type `Option[T]` can't be converted to a `T` when parsing json.

## 0.17.3 ([#33](https://git.mobcastdev.com/Platform/common-spray/pull/33) 2014-10-15 15:10:14)

V2 homogenous ListPage

patch

- Added v2 homogenous ListPage model, as discussed on #books-api-design on slack.

## 0.17.2 ([#32](https://git.mobcastdev.com/Platform/common-spray/pull/32) 2014-10-10 14:46:40)

Upgrade common-json dependency for better error messages

patch

- Upgraded common-json dependency to get the better error messages from version 0.2.3
- Added test for this behaviour

## 0.17.1 ([#30](https://git.mobcastdev.com/Platform/common-spray/pull/30) 2014-09-30 10:50:56)

Bumping versions

### Improvements

* Bumped common-lang, common-config and common-json

## 0.17.0 ([#29](https://git.mobcastdev.com/Platform/common-spray/pull/29) 2014-09-22 11:32:41)

Placeholder for v2 API json support

### New feature

- Marshaller and unmarshaller for `application/vnd.blinkbox.books.v2+json` media type.

This PR contains bare minimum to get going. We still need to decide what common model classes for paging, error messages, etc. will be in v2 API.

## 0.16.2 ([#28](https://git.mobcastdev.com/Platform/common-spray/pull/28) 2014-09-08 12:57:38)

Dependency updates

Patch to update dependency versions

## 0.16.1 ([#27](https://git.mobcastdev.com/Platform/common-spray/pull/27) 2014-09-05 14:47:41)

Added app version to monitoring/health checks

### Improvements

- Now returns an `X-App-Version` header from health endpoints with the
current version of the service.
- Logs an extra `facilityVersion` into the logging of requests and
responses for the same reason.

## 0.16.0 ([#26](https://git.mobcastdev.com/Platform/common-spray/pull/26) 2014-09-03 17:20:17)

Cross compiles to Scala 2.11, and no Swagger

### Breaking changes

- Removed Swagger v1 dependency and annotations as we’re going to dump
support for this documentation format (we never really implemented it
anyway) and use a YAML-based format for our v2 APIs.

### Improvements

- Now cross-compiles to Scala 2.11

### Deprecations

- `BigDecimalDeserializer` and `ISODateTimeDeserializer` in
`com.blinkbox.books.spray.JsonFormats` are now deprecated and point to
`com.blinkbox.books.spray.unmarshalling` package as these classes have
nothing to do with JSON.

## 0.15.0 ([#25](https://git.mobcastdev.com/Platform/common-spray/pull/25) 2014-08-12 09:25:00)

Added health checks and http bind checking

### New features

- Added a minimal `HttpServer` class which checks that the HTTP bind
succeeded.
- Updated the `HealthCheckHttpService` to have `/health/ping` and
`/health/threads` endpoints like the Codahale metrics ones.

### Improvements

- The health check service now uses its own execution context so it
shouldn’t interfere with the main service.

## 0.14.1 ([#24](https://git.mobcastdev.com/Platform/common-spray/pull/24) 2014-08-07 16:12:21)

Split out the logging of HTTP headers

### Improvements

Interesting HTTP headers are now logged under their own key, e.g.
`httpUserAgent` or `httpXForwardedFor` rather than being lumped
together under `httpRequestHeaders` and `httpResponseHeaders`. This
will enable searches to be done on them more easily.

## 0.14.0 ([#23](https://git.mobcastdev.com/Platform/common-spray/pull/23) 2014-08-06 13:45:43)

Replaced some code with common-json

### Breaking changes

- `JsonFormats`.`ExplicitTypeHints` and
`JsonFormats.ISODateTimeSerializer` have been removed; use the
corresponding classes in `common-json`.

### Bug fixes

- DateTime query parameters are now parsed correctly if they include a
milliseconds component.
- DateTime query parameters are always converted to UTC even if they
were sent in a different time zone.

## 0.13.1 ([#21](https://git.mobcastdev.com/Platform/common-spray/pull/21) 2014-07-25 17:40:19)

Log more info from HTTP requests

### Improvements

- Now captures interesting HTTP request and response headers in the log
output.
- Logs `401 Unauthorized` calls as INFO rather than WARN because this
is a common BAU code for us.

## 0.13.0 ([#17](https://git.mobcastdev.com/Platform/common-spray/pull/17) 2014-06-30 18:58:17)

Added a health check service trait

### New features

- Added a `HealthCheckHttpService` trait to serve Codahale health check results using Spray.
- Added a `rootPath` directive for easier and more reliable hosting of services at configurable paths.

## 0.12.0 ([#20](https://git.mobcastdev.com/Platform/common-spray/pull/20) 2014-07-03 17:28:41)

Quick fix for asQueryParams

### Breaking change

* Changed `asQueryParams` to return Seq[(String, String)]

## 0.11.0 ([#19](https://git.mobcastdev.com/Platform/common-spray/pull/19) 2014-07-03 16:33:23)

Pagination links improvements

### Breaking changes

* Changed the signature of paginated links generation: accepts a sequence of tuples instead of a map, to support repeated parameters

### Improvements

* Added `asQueryParams` for each `SortOrder` object so that you can combine it with pagination.

## 0.10.0 ([#18](https://git.mobcastdev.com/Platform/common-spray/pull/18) 2014-07-02 16:33:10)

Sorting and paging improvements

### Breaking changes

* Added ordered directive
* Added orderedAndPaged directive
* Paging links now support an optional query param
* Added an implicit conversion from `PageLink` to `Link` so that it can be used with `ListPage`

## 0.9.1 ([#16](https://git.mobcastdev.com/Platform/common-spray/pull/16) 2014-06-27 12:47:47)

monitor directive now handles exceptions

### Improvements

- The `monitor` directive now handles any unhandled exceptions using
the default logic so that all requests are logged irrespective or
whether they finish successfully, with a rejection, or with an
exception.

## 0.9.0 ([#15](https://git.mobcastdev.com/Platform/common-spray/pull/15) 2014-06-26 15:49:50)

Diagnostics

### New features

- Added a new `monitor()` directive which adds MDC context about the request and logs basic details of the request and response.
- Added a `clientIP` function to `HttpRequest`.

## 0.8.1 ([#14](https://git.mobcastdev.com/Platform/common-spray/pull/14) 2014-06-20 11:07:06)

ISODateTimeDeserializer

### Improvements

* Added ISODateTimeDeserializer for Joda DateTime query parameters
* Refactored BigDecimalDeserializer tests

## 0.8.0 ([#13](https://git.mobcastdev.com/Platform/common-spray/pull/13) 2014-06-09 10:12:15)

Convert from Java URI/URL to Spray Uri

### New features

- Defines implicit conversions from the Java `URI` and `URL` classes to
a Spray `Uri` instance, which is nicer to work when you’re using the
Spray framework.

## 0.7.2 ([#11](https://git.mobcastdev.com/Platform/common-spray/pull/11) 2014-06-05 12:13:38)

Tiny change to force rebuild, should be published to Artifactory

Patch to update docs, to force a version number increase and rebuild. This is the first version that will be published to Artifactory.

## 0.7.1 ([#9](https://git.mobcastdev.com/Platform/common-spray/pull/9) 2014-06-04 12:22:21)

Removed whitespace

### Patch

Merged the previous pull request too early :eyes:

## 0.7.0 ([#8](https://git.mobcastdev.com/Platform/common-spray/pull/8) 2014-06-04 12:14:00)

Added BigDecimal deserialiser

### New features

- Added `BigDecimalDeserializer` to allow `BigDecimal` query parameters

## 0.6.0 ([#7](https://git.mobcastdev.com/Platform/common-spray/pull/7) 2014-05-30 12:30:00)

Moved v1 JSON support into its own package

### Breaking changes

- The v1-specific JSON support is now in a `v1` sub-package.

### Improvements

- Includes resource models for v1 paged lists, links, images and errors.
- You can specify `responseTypeHints` on `Version1JsonSupport` for type
hints that are used only on responses (i.e. if you want to ignore the
type hint on requests, which you typically do).

## 0.5.0 ([#6](https://git.mobcastdev.com/Platform/common-spray/pull/6) 2014-05-28 17:38:46)

Added 'do not cache' directives

### New features

- Added `neverCache` and `uncacheable` directives to allow standard ‘do
not cache’ headers to be set.

## 0.4.0 ([#5](https://git.mobcastdev.com/Platform/common-spray/pull/5) 2014-05-23 16:45:47)

Added directive for setting standard cache headers on cached resources

### New features

* Added "cacheable" directive that allows standard cache headers to be set, with a configurable cache expiry time.

## 0.3.0 ([#4](https://git.mobcastdev.com/Platform/common-spray/pull/4) 2014-05-22 14:01:12)

Switched from native JSON library to Jackson

#### New features

- Parses JSON with Jackson instead of native support, as this is generally seen to be faster, hence should be the default choice for our services.

## 0.2.0 ([#3](https://git.mobcastdev.com/Platform/common-spray/pull/3) 2014-05-20 14:55:05)

JSON support + versionable by proteus

#### Breaking changes

- Removed the `version1ResponseHeaders` directive as this is obsoleted
by the `Version1JsonSupport` trait.

#### New features

- Added an `ISODateTimeSerializer` which serialises JodaTime dates
to/from JSON without losing time zone information, which the Json4s
default serialiser does.
- Added a `Version1JsonSupport` trait which can be mixed in to
configure both input and output for the blinkbox books v1 JSON format.

#### Improvements

- Changed the way the version is loaded so it can now be versioned by proteus.


text:
# Weather-App-Android-
Android app to show weather conditions of any city in world.
This app will be able to show details of the place such as Latitude, Longitude, Wind Direction, Temperature and much more.
# API and dependencies:

The Api used is Open Weather Map API which returns a data in JSON or XML format as needed by the user.The API_Key can be accesed by
signing in and then selecting the type of service as weather services.

Here in this project the HTTP request in sent through volley Library with the dependency:

implementation 'com.android.volley:volley:1.1.1'


text:
# population_rollup

population_rollup computes aggregate statistics for Core Based Statistical Areas (CBSA) using input data at the level of census tracts from the 2000 and 2010 censuses of the United States.

## Installation

From the directory in which you want the project to live, clone the repo.

```
git clone https://github.com/fox-daniel/population-rollup population_rollup
```
This will reproduce the project within the directory "population_rollup".

## Usage

Copy the file "censustract-00-10.csv" to /population_rollup/input.

From within the directory, population_rollup, run the following bash command.
```
./run.sh
```
The file "report.csv" will be written to /population_rollup/output.

The "run.sh" file runs the following command.
```
python3.7 ./src/pipeline.py ./input/censustract-00-10.csv ./output/report.csv
```
The variables for the script pipeline.py should be the paths (relative to population_rollup) to the input csv file and the output csv file.

### PPCH = '(X)'
When PP00 is 0 the PPCH is reported as infinite using the symbol '(X)'. This is done even if PP10 is also 0 even though in that case one could argue to report the PPCH as 0. When a CBSA contains a census tract for which PPCH is reported as '(X)', then the average PPCH over the CBSA is reported as '(X)' under the convention that infinity plus a finite number (or infinity) is infinity. Other reporting options might prove more useful. For example, the census tracts with PPCH = '(X)' could be dropped from the computation of the average. Another possibility would be to set PPCH = 0 if PP00 = PP10 = 0; this would apply to some CBSAs but still require a choice to be made when PP00 = 0 and PP10 > 0.

## Tests
With the exception of test_1, all tests can be run as unit tests by executing
```
$ python3.7 test_name.py
```
in the test directory. Each test has short csv files so that the results can easily be checked either in the output/report.csv or in the test_log.csv that is generated by the test. The test_name.py uses as few functions as possible to test the pipeline.

With the exception of test_row_length, each csv file in the input folder of the test folder can also be moved to the top-level input directory and then run.sh can be run for an end-to-end test. This does not work for test_row_length because the csv for that test has a row with too few columns. This currently prevents the data from proceding through the pipeline. Future versions could remove that row from the data csv (continuing to log it) and allow the data to continue flowing through the pipeline.

### test_1
Move test_1/input/censustract-00-10.csv to top-level input folder and run the run.sh file on it to check that the basic aggregate statistics are computed correctly.

### test_type_test
This checks whether values in the csv can be converted to the appropriate data type. If an exception is raised this is written to a log.

### test_no_cbsa
This test runs an abbreviated version of the pipeline on a test csv in which the CBSA09 is missing for one of the two rows. The output correctly only reflects the single row with the CBSA09.

### test_row_length
This tests whether the row lengths remains constant.

### test_groupby_cbsa
This tests the groupby function which computes aggregate statistics.

### test_geoid
This tests whether the GEOID is the concatenation of ST10, COU10,and TRACT10.

### test_scaling_file_size

This is the beginning of a test. Given more time, to develop it I would plot run time of pipeline.py against the number of rows in the csv of synthetic data. This would produce a threshold for when the current data structures and algorithms become impractical. If there were relevant use cases that exceeded the threshold then I would revise the pipeline to 1) write and read from csv less frequently and 2) use parquet files.

### Further Work
This code could be improved to scale better in the following ways.
- If large csv files are expected as input, then converting them to parquet files to store data during the inner stages of the pipeline should improve the speed.

## Contributing
This is a private repo.

## License

MIT License

Copyright (c) 2020 Daniel Fox

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

 IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

repository: fox-daniel/population-rollup

branch_name: refs/heads/main

repo_name: fox-daniel/population-rollup


text:
---
title: "qraptor Technology Stack Part Two: Applications in September 2019"
subtitle: "The application we use for our reinforcement trading platform"
output: html_document
image: "https://res.cloudinary.com/jenslaufer/image/upload/c_scale,q_74,w_800/v1568015884/matt-artz-pH6wLT6TVFc-unsplash.jpg"
image_caption: "Photo by Matt Artz on Unsplash"
tags: docker, programming languages, technology stacks, applications
categories: qraptor
layout: post
language: en
---

In [Part One: Architecture and Programming Languages](/qraptor/qraptors-technology-stack-programming-languages.html) we showed you that we using several programming
languages to get things done. Today we want to go more into details what applications/micro services for our reinforcement trading platform. It's devided into
several parts. First, we show you the shared technical services we use and second all the services to run our core business: algorithmic trdaing with reinforcement learning.

## Shared Technical Services ##

## MongoDB ##

![@TODO MongoDB Logo]()

MongoDb is our persistence container for a variety of artefacts, some of them are saved as MongoDB documents, which is basically simple JSON. We use also the GridFS-Filesystem for persisting
file-like structures. We love MongoDB, because of it's simplicity. You don't need schemas to persist data. However, we know that having no schema is in some situations risky.

The artefacts we persist in MongoDB:

- Training Session Meta Data
@TODO example

We save the complete the Meta Data for each training session with all the parmeters in the MongoDB. The trainer fetches new training sessions and dynamically initializes the train

- Trained Models from Trainer in hdf5 file

What we haven't done so far is to persist the source data files into the MongoDB, although they should be there to be fully straighforward. To be honest we are bit to scared to save hundreds of GB in the GridFS.
filesystem.

- Training results as CSV

## Portainer ##

Every application/service we use is packed as a Docker container. We use docker-compose and docker-machine for the management of the application. We use [Portainer](https://www.portainer.io/) for web based Docker management. Portainer is a Docker container by itself and is dead-simple to install and makes life easy.

![Portainer Screenshot for qrapotor Applications](@TODO)

## Exploratory Data Analysis ##

For the exploratory data analysis we use RMarkdown a lot. As the data files are very big we apply some data reducing before. We seperate the core logic and visualisations from the Rmarkdown code to reuse it in other notebooks and in Shiny Apps. We use Shiny Dashboards for on the fly data analysis or for Dashboarding for our clients.

## Data Wrangling ##

### Data Reducing ###

For our one client we are having data for each instruments for very second for 9 years with about 30 datapoints. It adds about to about 15 GB of data each instrument. Importing this amount of data into a dataframe with pandas in Python or dplyr in R is challenge. So we perform a kind of reducing on the data for data analysis or our first training sessions. You can reduce the data to a certain unit and interval e.g. five minutes or 30 seconds. The Libraries from [tidyverse](https://www.tidyverse.org/) ecosystem in R lets us do this redcution. The code is pretty self-explaining with a bare minimum lines of code. We added REST-like interface to the reducer microservice with [plumber](https://www.rplumber.io/).

### Data Preprocessing ###

We created a data preprocessing microservice also with a REST-like API with plumber. We can select what features we want to include into the datasets and which of these fields need to be preprocessed. The services splits in a first step the training and test data. The data are time series data, so we detrended the data by differencing.

## Training Environment ##

### Training Framework ###

Our Training frameworks loads training sessions from the MongoDB and initializes dynamically the Python objects for the training. We do the Then a training is performed and the model is tested against the test set. The training and test process, the model and test metrics are persisted in the MongoDB. We keep our  training process reproducable this way. The heart of the training framework are [OpenAI Gym Environment](https://github.com/openai/gym/blob/master/docs/creating-environments.md) and the [Stable Baseline](https://stable-baselines.readthedocs.io/en/master/) implementations of reinforcement learning algorithms, although the framework should also be able to use other libraries like [Keras-RL](https://github.com/keras-rl/keras-rl).

### Training Session Tool ###

The JSON for training session is very complicated, as it holds all attributes for a training session. Adding new training sessions into the DB takes therefore a lot of time and is error prone. Parameter tuning requires strategies: You change e.g. one parameter and observe if the performance on the test set is getting better or not. We created a tool that lets us easily create multiple training sessions for different parameter set ups. The tool is written with [Vue.js](https://vuejs.org/) and [Quasar](https://quasar.dev/) with a [Spring Boot](https://spring.io/projects/spring-boot) Java Backend.

### Tensorboard ###

Tensorboard lets us track session experiment metrics, visualise the training process and model graphs. We use Tensorboard hand in hand with out Training Analysis Dasboard to analyse our training sessions.

[@Todo Screenshot]()

## Training Result Analysis ##

We created a dashboard for analysis of the training results. It's a shiny app with very lines of code. ggplot2 is our visualisation library, as it is the vis library we are the most familiar with. The ggplot visualisations by itself are not per se interactive, but we might use plot.ly in future to accomplish this.
Another option is to integrate our own datapoints into Tensorboard visualisations.

[@Todo Screenshot ]()


text:
# An example Snakemake workflow using ppx and ANN-SoLo

[ppx](https://github.com/wfondrie/ppx) provides programmatic access to files
stored in proteomics data repositories, such as PRIDE and MassIVE. In doing
so, ppx enhances the reproducibility of proteomics analyses. Here, we provide
an example [Snakemake](https://snakemake.readthedocs.io/en/stable/) workflow,
which in part reproduces the open modification search results from
[ANN-SoLo](https://github.com/bittremieux/ANN-SoLo) presented by [Bittremieux
et al.](https://doi.org/10.1021/acs.jproteome.8b00359)

This workflow uses ppx to download the MGF mass spectrometry data files and the
spectral library from the original publication's project on PRIDE Archive,
[PXD009861](https://www.ebi.ac.uk/pride/archive/projects/PXD009861). These MGF
files are then searched using ANN-SoLo. Additionally, the workflow downloads
the original ANN-SoLo result files described in the original publication
(`*.mztab`). Finally, the workflow creates figures displaying the modification
mass shifts detected by the different ANN-SoLo versions.

## Prerequisites

You can run this workflow on MacOS and Linux systems, due to the requirements
for ANN-SoLo. You'll also need the following software installed:

- **A conda-based Python distribution** - We recommend
  [miniconda](https://docs.conda.io/en/latest/miniconda.html). This provides
  the conda package manager.
- **git** - [git](https://github.com/git-guides/install-git) is a tool for
  managing versions of code. You'll need git to grab this workflow from here on
  GitHub.

You can confirm that both requirements are met by opening your terminal and
trying the following commands:

**conda**
``` sh
$ conda --version
conda 4.10.1
```

**git**
``` sh
$ git --version
git version 2.31.1
```

## Running the workflow
### Downloading the workflow

We'll use git to download this workflow from GitHub:

``` sh
$ git clone https://github.com/Noble-Lab/ppx-workflow.git
$ cd ppx-workflow
```

Alternatively, you can use click `Code -> Download ZIP` in the [GitHub
repository](https://github.com/Noble-Lab/ppx-workflow), unpack the zip archive,
and navigate to the new `ppx-workflow` directory in your terminal.

### Create a conda environment

We'll create a conda environment, which will automatically install Snakemake,
ppx, and their dependencies:

``` sh
$ conda env create -f environment.yaml
$ conda activate ppx-workflow
```

### Execute the workflow

Now we'll use Snakemake to execute our workflow. Snakemake will manage the
installation of ANN-SoLo and other dependencies, then run all of the steps
required for our analyses.

In this example, we'll use `--configfile config/one.yaml`, which will restrict
our analysis to a single MGF file:

``` sh
$ snakemake -j -d workflow --use-conda --configfile config/one.yaml
```

Here is what each parameter does:

- `-j`: Use the maximum number of cores. You can also specify a number, such
  as `-j4` to use 4 cores.
- `-d workflow`: Tells Snakemake that our working directory should be the
  `workflow` directory. This is where our `Snakefile` is.
- `--use-conda`: Tells Snakemake that we want it to handle setting up a conda
  environment, such as for ANN-SoLo.
- `--configfile config/one.yaml`: Specifies a config file, other than the
  default one. The `config/one.yaml` file changes the workflow to only download
  and search a single MGF file, instead of all of them.

Alternatively, you could analyze all of the MGF files as we did in the paper.
Note that this could take a while depending on your system:

``` sh
$ snakemake -j -d workflow --use-conda
```

### Using a GPU

If you have an NVIDIA GPU available with at least 8 Gb of memory, you can use
it during the ANN-SoLo searches in this workflow. By default, the workflow will
only use the CPU. Enabling the GPU requires an additional parameter:

``` sh
$ snakemake -j -d workflow --use-conda --config GPU=True
```

Note that we've setup the work flow to use a CUDA Toolkit version of 11.2. If
this is not suitable for your system, you should adjust it in the [ANN-SoLo GPU
conda environment](workflow/envs/ann_solo-gpu.yaml).

## Conclusion

We've described how to run this workflow that uses ppx, ANN-SoLo, and Snakemake
to reproduce a previously published analysis with data shared in PRIDE Archive.
Our intention with this workflow is that it would be an illustration of how ppx
can help create reproducible and shareable proteomics analyses.

To find out more about ppx, see our documentation: https://ppx.readthedocs.io

repository: Noble-Lab/ppx-workflow

gha_language: Jupyter Notebook

repo_name: Noble-Lab/ppx-workflow


text:
---
title: Call-In Guide
layout: assignment
permalink: /:year/:month/:title
district_number: 25
year: 2018
month: September
talking_points:
  - theme: Energy
    gratitude:
      text: Thank you Rep. Williams for his August trip throughout the district, including his stop at Comanche Peak Nuclear Power Plant.
      link: http://google.com
    education:
      text: U.S. Wind Power Is ‘Going All Out’ with Bigger Tech, Falling Prices, Reports Show
      link: https://insideclimatenews.org/news/23082018/wind-energy-prices-market-growth-offshore-tax-credits-turbines-technology
    request:
      text: Rep. Williams’ district is home to tremendous energy resources, from Comanche Peak to the Barnett Shale natural gas reserves. Does the Representative know that large players in the nuclear and natural gas industries support carbon fee and dividend legislation to address climate change? This legislative approach to countering climate change provides energy companies a predictable business cost for carbon dioxide pollution rather than constantly changing and unpredictable regulations.
      link: https://clcouncil.org/founding-members
  - theme: Public Health
    gratitude:
      text: Thank you Rep. Williams for visiting us in Austin at the Dell Medical School and for your work on the partnership between the medical school and Central Texas Veterans Health Care System.
    education:
      text: Climate Change is Making Texas Summers Worse. Here’s Who That Hurts the Most.
      link: https://www.texasobserver.org/climate-change-is-making-texas-summers-worse-heres-who-that-hurts-the-most/
    request:
      text: Climate change is a hazard to people with disabilities, including disabled vets, which Rep. Williams works hard to serve. A report by the EPA noted that the extreme weather events caused by climate change, such as heat waves, flooding, and droughts, make people with disabilities more susceptible to illness and death. They may face additional physical challenges associated with evacuations, and extreme weather can cause power outages that affect electrically powered medical equipment. We ask Rep. Williams to advance legislation to slow climate change and lessen its impacts.
      link: https://19january2017snapshot.epa.gov/sites/production/files/2016-10/documents/disabilities-health-climate-change.pdf
  - theme: National Security
    gratitude:
      text: Thank you Rep. Williams for visiting us in Austin in August for the activation ceremony of the new Army Futures Command in downtown.
    education:
      text: According to the Department of Defense (DoD), “Each service agency within DoD incorporates the potential impact of global change into their long-range strategic plans,” and leaders in the military have called climate change a “threat multiplier.”
      link: http://globalchange.gov
    request:
      text: We ask Rep. Williams to continue his support for our military and reduce the burden on our military by advancing legislation to counter climate change.
      link: http://globalchange.gov
---

repository: boralben/grandcanyonlanding

repo_name: boralben/grandcanyonlanding