text:
---
title: "Traditions vs Statistics: Sechseläuten"
categories: [ statistics ]
tags: [ Zurich, statistics, fsharp, deedle, R ]
image:
  feature: sechselaeuten/header.jpg
  credit: Böögg (zuerich.com)
  creditlink: https://www.zuerich.com/en/visit/sechselaeuten
date: 2015-04-25
---
What do you expect from this summer? Should you worry about _20:39_ or not? Sunglasses or umbrella? Let's find out! [^1]

### Traditions

Each country has its own fun traditions and festivals, and Switzerland is not an exception. One of the most exciting events is the spring festival, [Sechseläuten](https://www.zuerich.com/en/visit/history-tradition), which is celebrated in Zürich in April. And it's also a day when you realize, that there're some people in town except the tourists walking along the Bahnhofstrasse on Saturdays! The trees become green, the flags appear here and there, the city becomes lively and colourful. And this year the day was just perfectly sunny. Not the best one for the drivers though because of the [parade of Guilds](https://www.youtube.com/watch?v=VHGKHn-Om-o) (everyone I showed the video especially likes the bear ~1:57 :).

{{% figure class="center" src="/images/sechselaeuten/sechselaeuten.jpg" title="Böögg (www.zuerich.com)" alt="Böögg" %}}

The culmination of Sechseläuten is the burning of [Böögg](https://www.zuerich.com/en/visit/who-or-what-is-the-boeoegg), a figure of snowman - the winter symbol. There's a belief, that the faster the head explodes, the nicer summer is going to be: say, below 10 min is for sunny and warm weather, above 15 min - for rains and cold... This year it took 20 min 39 s! Doesn't look very nice, huh?

### Statistics

So is there any correlation between the Böögg's forecast and actual weather? I couldn't resist checking the facts and downloaded the daily data for the Zürich weather station from the [European Climate Assessment & Dataset](http://eca.knmi.nl/) - it's free and contains a lot of information for the 20th century, including the period we're interested in, I was lucky to find it rather quickly! One can find the yearly Sechseläuten statistics, including the burning times, on the official [Sechseläuten website](http://www.sechselaeuten.ch/sechselaeuten/statistik.asp) (in German).

 Let's assume the _cold summer_ means lower temperatures, or more clouds, or less sunshine, or all of these together.

Now when we know more or less what we'd like to look at, we need to extract that information from the data - a bunch of simple csv files.

The weather datasets uses _-9999_ as indication of missing data, and another relevant column contains quality codes. As there's no requirement for each day's data to be present, we'll select only the valid records.

{{< highlight fsharp >}}
type QualityCode =
  | Valid   = 0
  | Suspect = 1
  | Missing = 9

// read the dataset from a file, index by DATE and remove unnecessary columns
let readData (fileName: string) =
    Frame.ReadCsv fileName
    |> Frame.indexRowsInt "DATE"
    |> Frame.filterRowValues (fun r -> r?Q = float QualityCode.Valid)
    |> Frame.dropCol "SOUID"
    |> Frame.dropCol "Q"

let dataFrames =
    List.map readData
        [
            "mean_temperature.txt" // TG, in 0.1 C
            "min_temperature.txt"  // TN, in 0.1 C
            "max_temperature.txt"  // TX, in 0.1 C
            "precipitation.txt"    // RR, in 0.1 mm
            "sunshine.txt"         // SS, in 0.1 hrs
            "humidity.txt"         // HU, in 1%
            "clouds.txt"           // CC, in oktas
        ]
{{< /highlight >}}

Cloud cover is in [oktas](http://en.wikipedia.org/wiki/Okta), where 0 means perfectly clear sky and 8 - completely cloudy.

Now we merge all these datasets, select only the summer months, and transform the temperatures to be in degrees C, sunshine - in hours (per day) and so on:

{{< highlight fsharp >}}
let weatherData =
    Frame.mergeAll dataFrames
    |> Frame.filterRows (fun date _ -> let m = date / 100 % 100 in 6 <= m && m <= 8)
    |> Frame.mapCols (fun col vs ->
        let series = vs.As()
        match col with
        | "HU" -> series / 100.0
        | "CC" -> series
        | _    -> series / 10.0)
{{< /highlight >}}

Finally, we combine the burning times and weather averages for corresponding years:

{{< highlight fsharp >}}
let weatherDataByYear =
    weatherData
    |> Frame.groupRowsByIndex (fun date -> date / 10000)
    |> Frame.applyLevel fst Stats.mean

let fullStats = Frame.join JoinKind.Inner boeoeggStats weatherDataByYear
fullStats.SaveCsv("fullStats.csv", keyNames = ["year"])
{{< /highlight >}}

Now that we have something to look at, let's start with the mean temperatures. We will use R for creating the charts:

{{< highlight fsharp >}}
meanTempPlot = qplot(duration,
                     TG,
                     data   = stats,
                     geom   = c("point", "smooth"),
                     method = "lm",
                     color  = TG,
                     main   = "Mean Temperature ~ Duration",
                     xlab   = "Duration, min",
                     ylab   = "Temperature, C")

meanTempBoxplot = qplot(duration,
                        TG,
                        data = stats,
                        geom = c("boxplot", "jitter"),
                        outlier.colour = "red",
                        fill = I("#3366FFAF"),
                        main = "",
                        xlab = "Duration, min",
                        ylab = "Temperature, C")

{{< /highlight >}}

{{% center %}}
![Mean Temperature](/images/sechselaeuten/mean_temperature.png)
{{% /center %}}

It's hard not to notice that there's no dependency between the mean temperature and duration. However, there're some outliers (the red dots): for example, the point in the top-left with the highest temperature is too far from the clustered group. Ironically, that was the most "accurate" prediction, in the year 2003, when it took less than 6 min for the head to explode and the summer was very warm and sunny.

The next two "best" years are in the rows with 20+ min. Even though in 2013 the weather was not that bad despite of 35 min for burning. The data in the table below is sorted by mean temperatures:

{{< highlight fsharp >}}
statsByTG = stats[order(stats$TG),]
{{< /highlight >}}

| year | time, min	| avg T, C	| min T, C	| max T, C	|precipitation, mm|sunshine, hrs|humidity|cloud cover, oktas|
|------|--------|-------|-------|-------|-------------|--------|--------|-----------|
| 1956 | 4.00	| 15.07	| 11.12	| 21.13	| 5.70	      |  5.64  | 0.74   | 5.66	    |
| 1978 | 12.00	| 15.38	| 11.42	| 20.27	| 4.52	      |  5.53  | 0.73   | 5.02	    |
| 1980 | 17.00	| 15.63	| 12.10	| 20.24	| 3.91	      |  4.69  | 0.75   | 5.30	    |
| 1972 | 8.00	| 15.77	| 11.86	| 20.70	| 4.84	      |  6.28  | 0.78   | 5.36	    |
| 1965 | 20.00	| 15.84	| 11.77	| 22.21	| 5.17	      |  6.15  | 0.77   | 5.32	    |
| 1966 | 16.00	| 15.84	| 11.62	| 22.12	| 5.24	      |  6.36  | 0.74   | 5.21	    |
| 1968 | 5.00	| 16.03	| 11.72	| 22.48	| 3.91	      |  6.94  | 0.71   | 5.20	    |
| ...  |		| 		|      	|      	|     	      |	       |		|           |
| 2013 | 35.18	| 18.53	| NA   	| NA   	| NA  	      |  8.14  | 0.70   | 4.21	    |
| 1952 | 6.00	| 18.75	| 13.68	| 25.73	| 3.05	      |  8.79  | 0.65   | 4.14	    |
| 1950 | 50.00	| 18.76	| 13.57	| 25.87	| 3.43	      |  8.72  | 0.65   | 4.08	    |
| 1983 | 24.33	| 19.06	| 14.15	| 24.63	| 1.72	      |  6.77  | 0.67   | 4.02	    |
| 1994 | 21.92	| 19.15	| 14.36	| 24.77	| 2.77	      |  7.09  | 0.72   | 4.34	    |
| 2003 | 5.70	| 21.66	| 16.53	| 28.02	| 2.29	      |  9.20  | 0.63   | 3.68	    |

As we see, the weather tends to be average - very dry, or rainy, or hot summers are exceptions. If you try the random forests (regression) with this data, _% Var explained_ is negative.

{{% center %}}
![Weather](/images/sechselaeuten/weather.png)
{{% /center %}}

One can argue, that for a number of reasons (the quality of weather measures, different structure of the Böögg itself or anything else) the years 1950 and 2014 are not comparable. I tried to run the same experiment with the data for the last 15 years, removing outliers and adding the labels for bad/ok/nice weather to make it a classification problem instead of regression - but the main conclusion is the same. Take a look at the historical averages for each of the summer days (the dark-blue line): the normal range (steel-blue) is quite narrow, even though the lowest and highest points (light-blue) may vary a lot[^2]:

{{% center %}}
![Historical Data](/images/sechselaeuten/historical.png)
{{% /center %}}

### Expectations

The results are pretty reassuring: we can be optimistic and hope that this summer will be great, or at least average - which is also quite nice. The weather is not an easy thing to predict months in advance!

### Reality

Update: The summer 2015 happened to be [extremely](http://allaboutgeneva.com/2015/07/15/swiss-heatwave-2015-new-danger-alert/) [hot](http://www.meteoschweiz.admin.ch/home/aktuell/meteoschweiz-blog.subpage.html/de/data/blogs/2015/8/sehr-warmer-august-extrem-heisser-sommer.html) and beat some temperature [records](https://www.washingtonpost.com/news/capital-weather-gang/wp/2015/09/15/the-summer-of-2015-was-earths-hottest-on-record-nasa-data-show/), not only in Europe. Let's hope we'll have more of a regular summer kind in the future.

And I've recently found out there's also an official study (in German): [Böögg Prognose](http://www.meteoswiss.admin.ch/home/climate/past/climate-of-switzerland/reports-throughout-the-year/boeoegg-prognose.html).

[^1]: I realized that I forgot to publish this post on time (the date of event) only several days ago, then I decided to convert the data processing part and use Deedle... Maybe that's because of sunny Milan, but it took some time to figure out which functions to use for relatively simple groupings and aggregations.

[^2]: To see how you can create similar kinds of plots check [this post](http://rpubs.com/bradleyboehmke/weather_graphic).

repository: luajalla/type-nat

repo_name: luajalla/type-nat


text:
# CRUD Operations on Model with Child Records

This scenario demonstrates performing Create, Read, Update, and Delete operations on an object that includes a collection of child records.

## Scenario Prototype

@snippet cs [..\Recipes.Interfaces\IProductLine`1.cs] IProductLine{TProduct}
@snippet cs [..\Recipes.Interfaces\IProduct.cs] IProduct

@snippet cs [..\Recipes\ModelWithChildren\IModelWithChildrenScenario`2.cs] IModelWithChildrenScenario{TProductLine, TProduct}

## ADO.NET

@snippet cs [..\Recipes.ADO\ModelWithChildren\ModelWithChildrenScenario.cs] ModelWithChildrenScenario

## Chain

Chain requires operating on the parent and child objects separately.

@snippet cs [..\Recipes.Tortuga.Chain\ModelWithChildren\ModelWithChildrenScenario.cs] ModelWithChildrenScenario

## Dapper

@snippet cs [..\Recipes.Dapper\ModelWithChildren\ModelWithChildrenScenario.cs] ModelWithChildrenScenario

## DbConnector

@snippet cs [..\Recipes.DbConnector\ModelWithChildren\ModelWithChildrenScenario.cs] ModelWithChildrenScenario

## Entity Framework 6

TODO

## Entity Framework Core

When performing an update, ensure that the `EntityState` is correctly set for each child record based on whether it is an insert or update.

For partial deletes, you have to explicitly find and delete the child rows that are no longer needed.

For deletes, ensure the child record in DBContext is setup with `.OnDelete(DeleteBehavior.Cascade)`

@snippet cs [..\Recipes.EntityFrameworkCore\ModelWithChildren\ModelWithChildrenScenario.cs] ModelWithChildrenScenario

## LINQ to DB

TODO

## LLBLGen Pro

With LLBLGen Pro we can utilize straightforward ORM functionality with this, utilizing the stand-alone, database agnostic Unit of Work
class for easy transaction management and order-of-operation control. For the partial deletes two variants are implemented, one
with a removal tracker, which tracks which entities are removed and which can be deleted in one go (the `Update` method in the alt repository),
and one with a direct delete on the table using a not-in predicate, available in the regular `Update` method.

Nested deletes are implemented explicitly, as cascading deletes aren't supported at the ORM level; to have cascading deletes you
have to set up the deletes as cascading on the foreign key constraint.

@snippet cs [..\Recipes.LLBLGenPro\Recipes\ModelWithChildren\ModelWithChildrenScenario.cs] ModelWithChildrenScenario

Alternative Update method using removal tracker functionality

@snippet cs [..\Recipes.LLBLGenPro\Recipes\ModelWithChildren\ModelWithChildrenScenarioAlt.cs] ModelWithChildrenScenarioAlt

## NHibernate

By default, NHibernate does not support a clean separation between the data access layer and the rest of the application. This is due to the way the lazy-loading works, which requires an active `ISession` even when lazy-loading isn't desired.

The work-around is to explicitly trigger lazy-loading when the child rows are desired. When the child rows are not desired, block lazy-loading by setting the collection property to an empty list.

For partial deletes, ensure that you are using `cascade="all-delete-orphan"`. Otherwise it will ignore the missing child rows. (Alternately, you can pass in a separate list of rows to delete.)

@snippet cs [..\Recipes.NHibernate\ModelWithChildren\ModelWithChildrenScenario.cs] ModelWithChildrenScenario

## RepoDb

@snippet cs [..\Recipes.RepoDb\ModelWithChildren\ModelWithChildrenScenario.cs] ModelWithChildrenScenario

## ServiceStack

@snippet cs [..\Recipes.ServiceStack\ModelWithChildren\ModelWithChildrenScenario.cs] ModelWithChildrenScenario

license: Unlicense

repository: mikependon/DotNet-ORM-Cookbook

branch_name: refs/heads/master

extension: md

gha_license_id: Unlicense

license_type: permissive

repo_name: mikependon/DotNet-ORM-Cookbook


text:
# Resources

Using your existing Terraform templates, you can take full advantage of the Spot platform. You can use the Spot Terraform plugin to interact with the platform, create Elastigroups, containerized workloads (with Ocean), and managed instances. See the [Spot Provider](https://www.terraform.io/docs/providers/spotinst/index.html) page to take advantage of all the resources available.

## Prerequisite:

You already have a Spot account and have completed the Spot Terraform [Installation](tools-and-provisioning/terraform/getting-started/install-terraform) and [Configuration](tools-and-provisioning/terraform/getting-started/configuration).


text:
# Audit Resolver

![](https://img.shields.io/npm/v/npm-audit-resolver.svg)
![](https://img.shields.io/static/v1?label=finally&message=released&color=green&style=flat-square)

A tool for building a responsible but practical supply chain security practice.

`npm audit` is great. `npm audit fix` is also there if you didn't know. But not everything can be fixed right away and you need to manage your security and make decisions about the dependencies you use.

> I built audit-resolver after a few weeks of trying to run audit as a step in CI and failing each time there's a vulnerability. There were just too many irrelevant or unfixed ones and my team needed a way to manage the situation.

Audit resolver creates a `audit-resolve.json` file in your app and interactively helps you manage security of your dependencies.
You can decide what to ignore and for how long, or track what's been fixed before.
The `audit-resolve.json` file sits in the repository and you can see who decided to ignore what and when.

I'm participating in [Package Vulnerability Management & Reporting Collaboration Space](https://github.com/openjs-foundation/pkg-vuln-collab-space) where I intend to donate parts of the audit-resolver's core.

## Changes in version 3

Due to changes introduced by npm7+ the option to fix an individual package is no longer available from npm and wasn't always working correctly anyway. By virtue of "doing one thing and one thing well" this package will no longer provide that option.
You can run `npm audit fix` before resolving unfixable issues. `resolve-audit` will offer you that option first before asking any questions about specific issues.

Requires npm v7.24.2+ or yarn installed alongside
Works with node 14+ (could run on 12 and npm6 but no guarantees going forward)
*You can use audit resolver v2.x with npm6.*
*Yarn support was not heavily tested across versions, but works well with current yarn 1 and 3 at the time of writing*

## Install

```
npm install -g npm-audit-resolver
```

## Usage

Go into the project folder and run

```
resolve-audit
```

It goes through the results of `npm audit` and lets you decide what to do with the issues.
The decisions you make are stored in `audit-resolve.json` to keep track of it in version control and have a log of who decided to do what and when.

### Arguments

```
--yarn switches to yarn instead of npm.
--yarn-berry switches to yarn2 or yarn3 instead of npm.
--migrate forces migration to the new audit-resolve.json file and format even if no modifications are made to decisions
--mock used in tests
```

All other arguments are passed down to the npm/yarn audit call

### Running in CI

One of the problems npm-audit-resolver solves is running audit as part of your build pipeline.
You don't want to break your CI for a few days waiting to get a fix on a dependency, but at the same time ignoring the whole class of issues or the audit result entirely means you'll rarely notice it at all.

Run
```
check-audit
```

This command will only exit with an error if a human needs to make new decisions about vulnerabilities and commit the `audit-resolve.json` file. If all issues are addressed, your build can pass.

For JSON output (similar to `npm audit --json`), run
```
check-audit --json
```

All other arguments are passed down to the npm/yarn audit call

## Features

If `npm audit fix` can help, you'll be prompted to run it first.

For all vulnerabilities found, you get to choose between the following options:

- remind in 24h - Lets you ignore an issue temporarily to make the build pass until a fix is known
- ignore - Adds the particular dependency paths and advisories to be ignored in the future. If the same issue in the same package comes up, but it's a dependency of another package, it won't get ignored. If a new issue is found in the package, it doesn't get ignored. You can decide if the decision expires or not.
- delete - Removes your dependency that brought the vulnerability in its dependencies.
- skip and quit, obviously

audit-resolve.json is formatted, so git history has a trace of who addressed which vulnerability, when and how.

### Why would I ignore security vulnerabilities?

Because otherwise running `npm audit` as part of your CI is not practical.

- dev dependencies! a DOS vulnerability in your test runner's dependency is not a showstopper
- build tooling vulnerability
- dependencies of a tool you use very narrowly and can prove it's safe
- new vulnerability without a fix and you want to wait for a fix while running your builds (there's a remind me in 24h option available)
- Further in the future - because a maintainer you trust has checked and the vulnerability in their dependency tree that you pulled is not affecting the package you're using

repository: naugtur/npm-audit-resolver

gha_language: JavaScript

repo_name: naugtur/npm-audit-resolver


text:
# Effe

[![Build Status](https://travis-ci.com/GettEngineering/effe.svg?branch=master)](https://travis-ci.com/GettEngineering/effe)
[![codecov](https://codecov.io/gh/GettEngineering/effe/branch/master/graph/badge.svg)](https://codecov.io/gh/GettEngineering/effe)
[![codebeat badge](https://codebeat.co/badges/9c74d700-ebf8-4b76-8405-1950874576c4)](https://codebeat.co/projects/github-com-maratori-testpackage-master)
[![Maintainability](https://api.codeclimate.com/v1/badges/bf753d7560c8e4aa5cf0/maintainability)](https://codeclimate.com/github/GettEngineering/effe/maintainability)
[![Go Report Card](https://goreportcard.com/badge/github.com/GettEngineering/effe)](https://goreportcard.com/report/github.com/GettEngineering/effe)
[![GitHub](https://img.shields.io/github/license/GettEngineering/effe.svg)](LICENSE)
[![GoDoc](https://godoc.org/github.com/GettEngineering/effe?status.svg)](http://pkg.go.dev/github.com/GettEngineering/effe)

**Effe** is an _orchestration_ engine for business-logic

## Installing

```go
go get github.com/GettEngineering/effe/cmd/effe
```

## Run

```bash
$ effe -h
Usage of effe:
  -d    draw diagrams for business flows
  -out string
        draw output directory (default "graphs")
  -v    show current version of effe
```

and ensuring that `$GOPATH/bin` is added to your `$PATH`.

## Documentation & Getting Started

http://gettengineering.github.io/effe

[Getting Started](https://gettengineering.github.io/effe/GettingStarted/) guide.

## Issues/Problems/Ideas

https://github.com/GettEngineering/effe/issues

## Get support

Effe is maintained by Gett. Use github issue tracking for any support request.

## License

Copyright (c) 2020 Gett

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

 IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

license: MIT

repository: GettEngineering/effe

gha_language: Go

gha_license_id: MIT

repo_name: GettEngineering/effe


text:
# Scenes > Actors

***

## Create Actor

###  Create Actor

![create actortype at x number y number at front](https://static.stencyl.com/pedia2/block-images/scene/actors/create-actor3.png)

Creates an actor at the specified position.

```
createRecycledActor([ACTOR TYPE], [NUMBER], [NUMBER], Script.FRONT);
createRecycledActor([ACTOR TYPE], [NUMBER], [NUMBER], Script.MIDDLE);
createRecycledActor([ACTOR TYPE], [NUMBER], [NUMBER], Script.BACK);
```

***

###  Create Actor on Layer

![create actortype at x number y number on layer with id object](https://static.stencyl.com/pedia2/block-images/scene/actors/create-actor-on-layer.png)

Creates an actor at the specified position and layer (given a Layer ID or Name).

```
createRecycledActorOnLayer([ACTOR TYPE], [NUMBER], [NUMBER], engine.getLayerById([INT]));
createRecycledActorOnLayer([ACTOR TYPE], [NUMBER], [NUMBER], engine.getLayerByName([TEXT]));
```

***

## Get Actor

###  Get Actor Instance

![actor](https://static.stencyl.com/pedia2/block-images/scene/actors/actor.png)

Let you choose a specific actor instance to return.

```
[ACTOR]
```

***

###  For Each Actor On Screen ...

![for each](https://static.stencyl.com/pedia2/block-images/scene/actors/actors-on-screen.png)

Loops over all actors on the screen. Use the embedded `actor on screen` block to refer to each actor.

```
engine.allActors.reuseIterator = false;
for(actorOnScreen in engine.allActors) {
  if(actorOnScreen != null && !actorOnScreen.dead && !actorOnScreen.recycled && actorOnScreen.isOnScreenCache) {
    [ACTIONS]
  }
}
engine.allActors.reuseIterator = true;
```

***

###  For Each Actor in Region ...

![for each region](https://static.stencyl.com/pedia2/block-images/scene/actors/actors-in-region.png)

Loops over all actors inside the specified region. Use the embedded `actor within region` block to refer to each actor.

```
for(actorInRegion in getActorsInRegion([REGION])) {
  if(actorInRegion != null && !actorInRegion.dead) {
    [ACTIONS]
  }
}
```

***

###  For Each Actor of Type ...

![for each actortype](https://static.stencyl.com/pedia2/block-images/scene/actors/actors-of-type3.png)

Loops over all actors of a given Actor Type (whether on or off screen). Use the embedded `actor of type` block to refer to each actor.

```
for(actorOfType in getActorsOfType([ACTOR TYPE])) {
  if(actorOfType != null && !actorOfType.dead && !actorOfType.recycled) {
    [ACTIONS]
  }
}
```

***

###  For Each Actor of Actor Group ...

![for each actorgroup](https://static.stencyl.com/pedia2/block-images/scene/actors/actors-in-group.png)

Loops over all actors inside the specified Actor Group (whether on or off screen). Use the embedded `actor of group` block to refer to each actor.

```
for(actorInGroup in [ACTOR GROUP].list) {
  if(actorInGroup != null && !actorInGroup.dead && !actorInGroup.recycled) {
    [ACTIONS]
  }
}
```

***

repository: Stencyl/stencylpedia

gha_language: Batchfile

repo_name: Stencyl/stencylpedia


text:
[![Build Status](http://ci.sagrid.ac.za/buildStatus/icon?job=weka-deploy)](http://ci.sagrid.ac.za/job/weka-deploy)

# weka-deploy

This repository contains build and test scripts the  WEKA data mining tool :

> Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, Ian H. Witten (2009); The WEKA Data Mining Software: An Update; SIGKDD Explorations, Volume 11, Issue 1.

repository: SouthAfricaDigitalScience/weka-deploy

repo_name: SouthAfricaDigitalScience/weka-deploy


text:
# week9.Readme_Generator
  [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)

  ![screenshot](https://github.com/tronciu92/week9.Readme_Generator/blob/master/run/Readme_Generator_GIF.gif)

  ### Project Description
  When creating an open source project on GitHub, it is important to have a quality README with information about the app--what is the app for, how to use the app, how to install it, how to report issues, and how to make contributions so that other developers are more likely to use and contribute to the success of the project. A command-line application will allow for quick and easy generation of a project README to get started quickly. This will allow a project creator to spend more time working on finishing the project and less time creating a good README.

  ### Install
  ```md
  npm i
  ```

  ### Usage
  ```md
  node index
  ```

  ### Language
  ```md
  Javascript, Node.js
  ```

  ### Test
  ```md
  npm run test
  ```

  ## About
  * Project Home Page:
  * Project Github: https://github.com/tronciu92/week9.Readme_Generator
  * Author: [`tronciu92`](https://github.com/tronciu92)

  ## Questions
  *  For more question please contact me<EMAIL_ADDRESS>


text:
# reserve - caching library based around Redis EXPIRE

[![Build Status](https://travis-ci.org/nickcharlton/reserve-ruby.svg?branch=master)](https://travis-ci.org/nickcharlton/reserve-ruby)
[![Coverage Status](https://img.shields.io/coveralls/nickcharlton/reserve-ruby.svg)](https://coveralls.io/r/nickcharlton/reserve-ruby)
[![Inline docs](http://inch-ci.org/github/nickcharlton/reserve-ruby.svg?branch=master)](http://inch-ci.org/github/nickcharlton/reserve-ruby)

This gem provides simple object caching support using Redis.

It works around the concept of "faulting"; you request an object from the
cache and if it already exists it will return the cached object. If not, it'll
execute the associated block to generate it instead.

Using Redis' `EXPIRE` functionality, the cached object can be set to
automatically disappear after a set period of time.

## Installation

Reserve isn't (yet) on [RubyGems][], so add the repo to your [Gemfile][]:

```ruby
gem 'reserve', :git => 'https://github.com/nickcharlton/reserve-ruby.git'
```

## Usage

```ruby
reserve = Reserve.new(Redis.new)

# store an item with a given key
item = reserve.store :item do
  { value: "this is item" }
end

item # => { value: "this is item" }
```

Both `Reserve.new` and `store` accept a hash of options, but they're a little
bit different for each:

For `Reserve.new`:

```ruby
default_timeout: 3600 # the default timeout to use with store.
key_prefix: 'reserve' # the prefix to use in Redis.
```

For `store`:

```ruby
timeout: 3600 # the timeout to use when storing this item.
skip_cache: true # skip the cache in this invokation (great for testing).
```

## Contributing

1. Fork it
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Add some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
5. Create a new Pull Request

## Author

Copyright (c) 2014 Nick Charlton<EMAIL_ADDRESS>
[RubyGems]: http://rubygems.org
[Gemfile]: http://bundler.io

repository: nickcharlton/reserve-ruby

repo_name: nickcharlton/reserve-ruby


text:
[Documentation Home](readme.md)

# Step Templates
A step template is a JSON representation of a step function. There are two ways to get this representation: 1) manually
create the JSON based on the properties of the step function or 2) use the provided [annotations](step-annotations.md)
when creating the step and run the **metalus-utils** [metadata extractor utility](metadata-extractor.md) to scan step
jars and produce the JSON.

In order to generate the proper step template, start with a step function:

```scala
  def readFromPath(path: String, options: DataFrameReaderOptions = DataFrameReaderOptions(),
                   pipelineContext: PipelineContext): DataFrame = {
    DataFrameSteps.getDataFrameReader(options, pipelineContext).load(path)
  }
```

This step function is required to exist within a scala **object**. The example step function has three parameters, one of which
is a _PipelineContext_. The *path* and *options* parameters **must** be represented in the step template, but the
*pipelineContext* parameter is a special library parameter that should **not** be a part of the step template.

## [Metadata Extractor](metadata-extractor.md)
The easiest way to generate step templates is to add an annotation to the step object and step function that may be
scanned using the [metadata extractor utility](metadata-extractor.md). The **StepFunction** annotation provides
information that cannot be gathered from the step function as well as mark the function as being a step. The annotation
**must** be fully filled in with the following parameters: id, displayName, description, type, category. A description
of each is provided in the [JSON Step Template](step-templates.md) section. The **StepParameters** annotation provides
additional information for each parameter. The _description_ being the most important since it cannot be derived.

```scala
  @StepFunction("87db259d-606e-46eb-b723-82923349640f",
    "Load DataFrame from HDFS path",
    "This step will read a dataFrame from the given HDFS path",
    "Pipeline",
    "InputOutput")
  @StepParameters(Map("path" -> StepParameter(None, Some(true), None, None, None, None, Some("The HDFS path to load data into the DataFrame")),
    "options" -> StepParameter(None, Some(false), None, None, None, None, Some("The options to use when loading the DataFrameReader"))))
```

The scala object containing the step function(s) must have the **StepObject** annotation. Here is a complete list of
[annotations](step-annotations.md).

The [metadata extractor utility](metadata-extractor.md) will infer the remaining information from the function itself. Tags
will be automatically populated with the name of the jar file where the step function was located.

## Manual
In the absence of annotated code, the step template can be manually created.

## JSON Step Template
Below is a step template based on the example step function. The first five attributes cannot be pulled from the step
function. Either an annotation must be provided or the step function developer would need to provide the information.

```json
    {
      "id": "87db259d-606e-46eb-b723-82923349640f",
      "displayName": "Load DataFrame from HDFS path",
      "description": "This step will read a dataFrame from the given HDFS path",
      "type": "Pipeline",
      "category": "InputOutput",
      "params": [
        {
          "type": "text",
          "name": "path",
          "required": false,
          "parameterType": "String"
        },
        {
          "type": "object",
          "name": "options",
          "required": false,
          "className": "com.acxiom.pipeline.steps.DataFrameReaderOptions",
          "parameterType": "com.acxiom.pipeline.steps.DataFrameReaderOptions"
        }
      ],
      "engineMeta": {
        "spark": "HDFSSteps.readFromPath",
        "pkg": "com.acxiom.pipeline.steps",
        "results": {
          "primaryType": "org.apache.spark.sql.DataFrame"
        }
      },
      "tags": [
        "metalus-common_2.11-spark_2.4-1.5.0-SNAPSHOT.jar"
      ]
    }
```

### Metadata Attributes
These attributes provide basic information about the step function. The **displayName** attribute may be used during pipeline
execution for logging, but otherwise these attributes are not required for execution.

* **id** - A unique GUID which can be used to identify this step function
* **displayName** - A short name that may be presented to a user in a UI
* **description** - A longer explanation of what this step does
* **type** - The type of step. It can be one of the following values: Pipeline, pipeline, branch, fork, join or step-group
* **category** - Represents a group which this step should belong. This is useful for UIs and queries to provide a common grouping of steps

### Tags
Tags provide an additional layer of grouping within a query or UI. In the example above, the name of the jar file(s) which
contained the step functions have been added. Tags are free form strings that can be anything.

### Engine Meta
The engine meta section contains the metadata required to execute the step function at runtime. The **spark** attribute
is required and must contain the object name containing the step function and the function name. It should be in the form
of **<Scala_Object>.<function_name>**. The **pkg** attribute lists the scala package where the step object may be found.
This is not required, but is useful for systems that may use this information to help construct applications at runtime.

#### Results
The results attribute is not required, but is useful in indicating what a step function may return. This information may
be used in the future as a way to verify that step mappings are compatible. Step function can only return a single value,
but using the **PipelineStepResponse** class, multiple values may be returned. The **primaryType** value will either be
the actual return type or if a PipelineResponse is returned, the developer may choose to provide an additional annotation
which indicates the return type for primary and secondary. The attribute **secondaryTypes** may be added as a JSON object
containing name value pairs.

### Params
The **params** array contains all of the step function parameters except those of type **PipelineContext**. Each parameter
has seven possible parameters:

* **type** - The parameter [type](parameter-mapping.md#types). Must be one of: integer, list, boolean, script, scalaScript, string, text, result, object
* **name** - The name of the parameter. This **must** match the name of the parameter on the step function.
* **description** - Describes what this parameter should do.
* **required** - Boolean flag indicating whether the requirement of this value
* **defaultValue** - An optional default value that will be used when no value is provided
* **language** - An optional script language this parameter expects the value to conform. This should only be present if the **type** is set to *script*
* **className** - If the **type** is **object**, then this should represent the fully qualified class name that is expected
* **parameterType** - An optional attribute that represents the fully qualified class name (or primitive name) of the
step function parameter. This may be set regardless of type

#### Branch Steps
Branch steps differ from all other steps in that additional **params** of type **result** may be added which are used to
determine the next step which will be executed. The name of these parameters must match the possible primary return type
value. Only branch steps should have result types and must contain at least one. There is no limit to the number of result
parameters that may be added.

#### Scala Script Parameters
Step parameters given a type of **scalascript** will be compiled and evaluated, with the result of the script passed to the step.
These scripts consist of a binding section and a scala script. The bindings section must be enclosed by parenthesis.
Each binding contains a name, value, and type, broken out as **\<name>:\<value>:\<type>**. Multiple bindings can be provided using a comma separated list.
Colons can be escaped using a backslash.
A sample scala script is provided below:
```scala
(list:@GetList:Option[List[String]],pattern:!mypattern || [a-z]+:String)
// script
list.getOrElse(List()).filter(s => s.matches(pattern))
```

repository: dafreels/metalus

branch_name: refs/heads/develop

gha_language: Scala

repo_name: dafreels/metalus


text:
For example keeping the errors on file

``` Go
logFile, err := os.Create("gitter.log")
api.SetDebug(true, logFile)
```

##### App Engine

Initialize app engine client and continue as usual

``` Go
c := appengine.NewContext(r)
client := urlfetch.Client(c)

api := gitter.New("YOUR_ACCESS_TOKEN")
api.SetClient(client)
```

[Documentation](https://godoc.org/github.com/sromku/go-gitter)

repository: status-im/matterbridge

repo_name: status-im/matterbridge


text:
---
title: "FUSEing for Fun"
date: "2022-03-26"
showthedate: true
tags: ["programming"]
---

Something that I find absolutely fascinating
are [Filesystems in USErspace (FUSE)](https://en.wikipedia.org/wiki/Filesystem_in_Userspace).

### What is FUSE?

FUSE, to put it simply, allows people to implement filesystems in userspace. Filesystems are traditionally implemented
in the kernel, but when using FUSE, non-privileged programs can be used to create custom filesystems.

### Why is it cool?

Filesystems don't need to be concrete! You can implement virtual filesystems with little or no limitations.

* Don't want files to write to disk? Implement a "ramdisk" in FUSE.
* Want to use Amazon S3 as if it is a hard drive on your computer?
  Use [s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse).
* Want to be evil 😈 and store files on Google sheets? Use
  [spreadsheetfs](https://github.com/GunshipPenguin/spreadsheetfs).

 This is an unfinished project and only
> for research or recreational purposes only.

It features an append-only filesystem that acts similarly to
a [log-structured filesystem](https://en.wikipedia.org/wiki/Log-structured_file_system).

There are two channels:

* \#tx: stores transaction data
* \#data: stores file data in 8MB chunks

### Transaction format

```go
type Tx struct {
	Tx        int       `json:"tx"`
	Type      int       `json:"type"`
	Path      string    `json:"path"`
	FileIDs   []string  `json:"ids"`
	Checksums []string  `json:"sums"`
	Size      int64     `json:"size"`
	Mtim      time.Time `json:"mtim"`
	Ctim      time.Time `json:"ctim"`
}
```

* Tx: 0 for write, 1 for delete
* Type: 0 for file, 1 for folder
* Path: absolute path of file or folder
* FileIDs: file ids for file data
* Checksums: SHA1 checksums for file ids
* Size: size of file in bytes
* Mtim: modification time of file
* Ctim: creation time of file

For example, the following json represents a transaction that creates a file with path `/test.mp4` with a size of ~35MB.

```json
{
  "tx": 0,
  "type": 0,
  "path": "/test.mp4",
  "ids": [
    "956040610224148480",
    "956040626284146698",
    "956040643749224478",
    "..."
  ],
  "sums": [
    "Y9Whjuk_kbopDYx7cdSHXrzApvk=",
    "sxFp01p0Q52hF-q8LWGi1DoXX-M=",
    "7PljyMpvTDE-cfaZtm532OTwG7U=",
    "..."
  ],
  "mtim": "2022-03-22T23:01:08.3596736-04:00",
  "ctim": "2022-03-22T23:01:07.9266501-04:00",
  "size": 348437445
}
```

## Implementation overview

### At Startup

All transactions are downloaded from the #tx channel and applied in sequential order in a radix tree structure.

Why in a radix tree structure?

Radix trees have fast prefix lookups which is beneficial for querying file and folder paths.

Startup times can get long when there is a long transaction history. To speed startup times multiple transactions can be
compacted into one message. Transaction order is still preserved and if no more transactions can fit in a message, the
transactions are compacted into the next message and so on.

### Opening a folder

The path is queried in the radix tree and the immediate children (files and folders) are listed along with basic
metadata such as size, modification/creation times, etc.

### Opening a file

A background process will load the file into a store backed by memory or disk. The first and last pieces (same behavior as BitTorrent) are
downloaded first so that certain applications are able to preview the file.

### Reading a file

A [sparse array implementation](https://github.com/darenliang/dsfs/blob/master/utils.go#L18) is used to determine which
file chunks are loaded. The implementation is optimized for space efficiency and determines if a contiguous range of
bytes is loaded or if bytes can be read in O(n log n) time with O(1) space. The read command will hang (up to a
specified timeout) until bytes can be read.

### Writing to a file

Writing to the file happens in memory. The changes are not immediately reflected on the remote filesystem as writes
often happen in small chunks.

### Closing a file

Each opened file has a dirty bit associated with it which is set to true when the file is modified in memory. Upon
closing the file, the dirty bit is checked and file data is uploaded. SHA1 checksums are used to ensure that there are
no unnecessary chunks uploads.

To avoid running into Discord ratelimits, transactions are buffered and flushed every 5 seconds. Multiple transactions
can be packed within one message given that most transactions take up a small fraction of the ~~8MB~~ [25MB](https://twitter.com/discord/status/1645522780337885184) file size limit.

### Realtime synchronization

Syncing between clients occurs when a client sends a transaction in the #tx channel. Another client picks up the
transaction and applies it on their filesystem. When a file is already opened, SHA1 checksums are used to ensure that
only modified chunks are downloaded and patched.

### Automatic snapshots

Since the filesystem is append-only, each historic state of the filesystem is saved and can be recovered in the future
by replaying transactions up to a specific date and time.

### Cross-platform support

[Cgofuse](https://github.com/winfsp/cgofuse) is used which supports Windows, macOS and Linux.

The FUSE libraries required for each platform:

* Windows: [WinFsp](https://github.com/winfsp/winfsp)
* macOS: [macFUSE](https://osxfuse.github.io/)
* Linux: [libfuse](https://github.com/libfuse/libfuse)

### Performance

Read speeds are usually unbounded since reading file chunks is done through Discord's CDN instead of Discord's API.

Discord API ratelimits are dynamic, but if we assume the rate limit is 1 message per second, here are the theoretical write speeds possible with dsfs:

| Free User | Nitro Basic | Nitro   |
|-----------|-------------|---------|
| 25MB/s    | 50MB/s      | 500MB/s |

## Areas of improvement

Renaming folders is horribly inefficient as it involves renaming all children. This issue can be solved by giving each
file/folder a permanent id and create a mapping between paths and ids.

When performing many operations at once, you can get ratelimited. This issue can be alleviated by running a connection
pool to artificially increase the ratelimits.

A lot of permissions and attributes are not implemented. In most systems, the size attribute is all that matters. Some
attributes such as access times is unfeasible to implement as the attribute would be stale most of the time.

## Basic Demo

![Demo](/img/fuse/demo.gif)

## Source code

Visit dsfs on GitHub


text:
# sockpuppet
i needed sockets but i didn't want to write all the code myself. so i went and "stole" code from [here](https://github.com/saprykin/plibsys).

## usage
see `sockaddr.h` and `socket.h` if you want to use this libray. there is an example in `example/`. to build it run `sh build.sh.bat [name of example]` if you are on unix or `./build.sh.bat [name of example]` if you are in a mingw-like environment. it should without one though. i think.

repository: emekoi/sockpuppet

repo_name: emekoi/sockpuppet


text:
# AutoQuit

This is a POEHUD plugin that will work with x64 version of PoeHUD.

# Installing AutoQuit in PoeHUD

- Copy this repo into poehud->plugins folder
- Download ([currports x64bit version](http://www.nirsoft.net/utils/cports-x64.zip)) and put cports.exe in c:\Windows\System32


text:
# Instructions to run the workshop image on the Cloud
This is a good option if you are using a loaner Chromebook at the workshop, or if you prefer to work in the Cloud. These instructions will show you how to start a Google Compute Engine (GCE) VM with Docker installed, and how to run the container there.

## Initial setup

### 1. Create a Google Cloud Platform account

Sign up for a [free trial](https://cloud.google.com/free-trial/) of Google Cloud Platform (GCP). You will need a credit card to sign up, and you will receive $300 of free credits. *Note:* you will not be billed, unless you decide to renew after the trial ends.
* If you already have an account, or have run through your trial credits, see one of the workshop instructors. We will give you additional credits to apply to your account.

**Important note:** the author of this workshop is not interested in selling you Cloud services. If you prefer, feel free to use an alternative Cloud provider. The following instructions are written assuming you are using Google Cloud.

#### 1.1 Enable the necessary APIs

1. Go to the Google Cloud console: console.cloud.google.com
1. Select or create a project using the project drop-down at upper-left ('My First Project' in image below)
1. Click on the “hamburger” menu at upper-left, and then “API Manager”.
1. On the left nav, choose "Dashboard" if not already selected, then choose "+Enable API" in the top-middle of page.
1. Enter "Google Compute Engine API" in the search box and click it when it appears in the list of results.
1. Click on “Enable” (top-middle of page).

![Hamburger menu](../images/hamburger.png)

![API Manager](../images/api_manager.png)

### 2. Connect to your project's Cloud Shell

Click on the Cloud Shell icon (leftmost icon in the set of icons at top-right of the page).

![Cloud Shell](../images/cloudshell2.jpg)

Click on "Start Cloud Shell" on the bottom right of the pop-up screen. You should now see a terminal at the bottom of your window for the Cloud Shell with the text "Welcome to Cloud Shell! Type "help" to get started."

### 3.

### 4. Set up a firewall rule for your project that will allow access to the IPython notebook server

```shell
gcloud compute firewall-rules create workshop --allow tcp:8888
```

### 5. SSH into the new GCE instance, in a new browser window

- Click on the “hamburger” menu at upper-left, and then “Compute Engine”
- Find your instance in the list (mid-page)
- Write down the "External IP", this is the IP of your Cloud instance
- Logon to instance by clicking on the “SSH” pulldown menu on the right. Select “Open in browser window”.
- A new browser window will open, with a command line into your GCE instance. Confirm that you wish to initiate an SSH connection to the instance.

### 6. Start the Docker container in the GCE image (in the newly opened SSH browser window):

Run this command to download and run the container, mounting ```~/shared``` on your instance as ```~/workshop/shared``` inside the running container.

```shell
mkdir ~/shared
docker run -v ~/shared:/workshop/shared -it -p 8888:8888 randomforests/wtm:v1
```

### 7. Start a notebook server inside the running container (in the newly opened SSH browser window):

In this step, we will start an IPython Notebook server that runs inside the container. We will then connect to it using a web browser on your laptop (or Chromebook).

*Note:* we will use IPython notebooks for a coding exercise, and as an easy way to copy images you create from the Cloud VM to your Chromebook using a UI (this is a bit of a hack, but what the hey).

**Step 7a)** Start the notebook server.

```
# cd /workshop/scripts
# sh notebook.sh
```

You will see output on your terminal to indicate the server is running. If you want to stop the notebook server later, press *Control-C* (but do not do this now, we'll need it running in the next step)

**Step 7b)** Copy the login token.

Notice the second from the last line contains a login token. Copy this; you will need it in a moment to connect to the server.

![Token](../images/token.png?raw=true)

**Step 7c)** Use a web browser on your laptop or Chromebook to connect to the server

Open a web browser on your laptop. Use the "External IP" from step #5 followed by :8888, i.e. ```<External_IP>:8888``` in the address bar.

Your browser window should look like this.

**Note:** in the address bar, you will have entered the external IP address of your Cloud VM instead of 'localhost'.

![Login screen](../images/login.png?raw=true)

Paste the login token you coped in the previous step to connect to the server (if you copy/paste, make sure there is no newline splitting the token value, i.e. that you are pasting the token value printed on a single line). Your window should now look like this.

![A picture of a directory in a web browser](../images/notebook.png?raw=true)

### 8. Practice running commands inside the container, and downloading the images you create
This steps assumes you have already started a notebook server inside the running container as described in step 7 above. We will now run a command inside the running container to create an image, then download it using the IPython Notebook server's UI.

**Step 8a)** Open a second SSH window into the running container.
We use the second window to run shell commands (say, to stylize images), and later, we will use the IPython Notebook server to download the images we create to your laptop (or Chromebook).
This is what you need to do:
- Do SSH logon to your GCE instance (see step 5 above on how to do that)
- Then logon to your docker instance by typing: ```docker run -v ~/shared:/workshop/shared -it randomforests/wtm:v1```
  (note, omit the "-p 8888:8888" flag from step 5, since our notebook server uses that port map already)

**Step 8b)** Use your web browser connected to the IPython Notebook server to open the ```shared``` directory.

![Shared](../images/shared.png?raw=true)

Notice, that it's empty.

![Empty](../images/empty.png?raw=true)

**Step 8c)** Now, we will create an image inside this directory by running a terminal command in the SSH window connected to the running container.

Run these commands inside the new SSH window.

```
# cd /workshop/scripts
# sh fast_style.sh
```

**Step 8d)** When the command completes, refresh your browser window. You should see an image inside the shared directory.

![Not empty](../images/not_empty.png?raw=true)

Click on it to open it, then save it your laptop (or Chromebook) using your web browser.

### 9. Practice uploading images to the container
We will now upload an image from your Chromebook to the running container.

**Step 9a)** Use a web browser to download your favorite [image](https://upload.wikimedia.org/wikipedia/commons/a/af/Cara_de_quem_caiu_do_caminh%C3%A3o..._%28cropped%29.jpg) to your laptop or Chromebook.

**Step 9b)** Upload it to the shared folder using the Notebook's UI.

Click the upload button:

![Upload1](../images/upload1.png?raw=true)

Then navigate to your image, select it, and click Open.

![Upload2](../images/upload2.png?raw=true)

You will see a new row in the shared directory, with a blue upload button on the far right. Click that to upload your image.

![Upload3](../images/upload3.png?raw=true)

And your image is now available on the container. The path to this directory on the container is ```/workshop/shared```.

## If you need to restart the container later

If you later exit your container and then want to restart it again, you can find the container ID by running the following in your VM:

```shell
docker ps -a
docker start <container_id>
```
Once the workshop container is running again, you can exec back into it like this:

```shell
docker exec -it <container_id> bash
```
# Next steps.
That's it! You now know how to run commands on the running container, and how to upload and download images. Preview the [exercise](exercises.md) instructions, if you like.

repository: jhsiaomei/WTM

repo_name: jhsiaomei/WTM


text:
# dom-diff

Creates diffs of DOM or a virtual DOM that looks like the real DOM.

## Installation

```
npm install dom-diff --save
```

## Usage

```js
var diff = require("dom-diff/diff");
var apply = require("dom-diff/patch");

var a = document.createElement("div");
var b = document.createElement("div");

b.appendChild(document.createElement("span"));

var patches = diff(a, b);
apply(a, patches);
```

## Modules

### dom-diff/diff

```
diff(Element, Element) -> patches
```

Given two elements, diffs them and returns an object containing patches.

### dom-diff/patch

```
patch(Element, patches)
```

Given an element and a set of patches, the patches will be applied to the element.

### dom-diff/serialize

```
serialize(patches) -> serializedPatches
```

Given a `patches` object will serialize it into a plain object that can be sent across WebWorker boundaries. You can further serialize this object to JSON if you need to send it across the network.

## License

MIT

repository: canjs/dom-diff

repo_name: canjs/dom-diff


text:
---
id: trends
title: Comparing Statistical Trends
sidebar_label: Trends
description: See how grouping tests reveals outcome patterns across isolated variables, such as browser, operating system, or date to optimize your tests.
---
import useBaseUrl from '@docusaurus/useBaseUrl';

The Trends section of the Insights feature provides a variety of data visualizations to give you a holistic perspective of your test outcomes. The following table describes each section.

    Section
    Statistical Information

    Number of Tests
    The total number of tests run during the specified time period, separated in increments relative to the overall duration. For example, increments may be every 10 minutes for a time period of one hour, while increments might be daily for a 30-day time period.

    Pass/Fail Rate
    For each increment in the time period, the percentage of tests that:

Completed: Ran to completion, but did not have a pass or fail status.

Passed: Ran to completion with a status of Passed.

Failed: Ran to completion with a status of Failed.

Errored: Did not run to completion due to a fatal error.

    Number of Errors
    The total number or errors that occurred during the specified time period, sorted by individual error message.

    Build and Test Statistics
    A snapshot of all tests run during the time period, displayed in separate tabs based on whether the test is or is not assigned a Build ID.For each test listed, basic data about the time the test was executed, the time it took to run, the Sauce Labs user who ran it, and its outcome. Tests in the Builds tab have an additional statistic — Efficiency — that indicates whether the tests in the build run in parallel to optimize the execution time for the entire build.
This visualization can be further filtered to show only tests with a failed and/or errored status.

## Drilling Down on Visualizations

The visualizations shown in the **Trends** section are interactive; you can hover over any of the bars to view a statistics overview for that increment, or you can click-drag across the bars to redraw the graph for a narrower time period. The latter action updates the time period filter at the top of the page accordingly.

## Using Trends Data to Improve Testing

The trend visualizations can provide you with a quick overview of what's going on with your tests as a whole, and applying filters to the visualizations enables you to dig into the data to generate answers to specific questions about test performance. Let's look at an example showing how we can use these tools to answer real questions about our site and our tests.

### Comparing Test Results on Chrome 50 and 55

To find out how well a site under test performs against a browser update, we start by filtering our data to isolate only the relevant tests — those that are owned by the same organization — were run over the past seven days on Windows 7 for Chrome 55 and Chrome 50. This is a typical use case to compare a set of new tests for a recent browser release against the baseline of an established set of tests for a previous version of the same browser.

#### Chrome 50

As the figure below shows, more than 3,500 tests were run on Windows 7 for Chrome 50 in the past seven days, with a 41% pass rate.

There are no errors, indicating that this is a robust set of tests, but a large number of tests ran to completion without reporting a Pass or Fail status. Hovering over one of the bars in the graph shows that these no-status completions account for about 65% of the tests in every time increment.

While the tests themselves perform well, it's difficult to judge how well the site functions when completed tests do not offer a definitive outcome. To provide a better baseline for cross-browser comparison, [annotate](/basics/test-config-annotation/test-annotation) these tests with relevant status using the Jobs API or the Selenium Javascript Executor.

#### Chrome 55

Now let's change the Browser filter to Chrome 55where nearly 4000 tests ran during the same seven-day period.

The pass rate for these tests is lower than Chrome 55, at 32% and we still see a lot of completed tests without status. We see some failures showing up, but there are no errors, indicating that the tests themselves seem to execute successfully for both browser versions.

#### Conclusions

In summary, we see from this browser version comparison:

* Overall coverage is consistent.
* Test performance is good for both, with low error rates.
* Site functionality is ambiguous for both browsers due to the high number of completed tests with no status.

#### Next Steps

Since our comparison suggests that the tests themselves are strong, but yielded some uncertainty about how well the site performs in either Chrome version, we isolate the time interval that contains the first failing test at 4:00PM on February 7. Then, we can drill down to the five-minute scale to find the exact test that failed.

Once we have isolated the failing test, we can check the **Builds** list and find the failing test, **TestCompareBrowserVersion**. Click that test name to see the **Test Details** page, where you can review the videos, screenshots, logs, and metadata that can help you determine why the test failed for Chrome 55.

### Using the Efficiency Metric to Optimize Tests

The **Builds and Test Statistics** section of the **Trends** page provides an **Efficiency** metric for builds that indicates the percentage of tests in the build that are running in parallel.

#### Benchmarking Efficiency

The Efficiency metric is expressed as a percentage because it is based on how long the build took to run as compared to the duration of the longest test within it. For example, let's say that Build A contains four tests with these run times:

|Test|Run Time|
|---|---|
|T1 |30 secs|
|T2	|60 secs|
|T3	|45 secs|
|T4	|30 secs|

In this build, T2 serves as the benchmark test because it takes the longest to run at 60 seconds. If the entire build takes 60 seconds to run, then it has achieved full efficiency, because all the tests are running in parallel, and the Efficiency metric would be 100%.

Consider another example, Build B:

|Test|Run Time|
|---|---|
|T1	|15 secs|
|T2	|20 secs|
|T3	|10 secs|
|T4 |45 secs|
|T5	|30 secs|
|T7	|10 secs|
|T8	|20 secs|
|T9	|15 secs|

In this example, T4 serves as the benchmark for the build efficiency, because it takes the longest to run at 45 seconds. However, even if the test has complete parallelization (the entire build runs in 45 seconds), given that the majority of the tests are considerably shorter than 45 seconds, the overall efficiency of this build is still not optimal. Reducing the run time of T4 (and even T5) would improve the build efficiency.

#### Improving Efficiency

An Efficiency score of less than 100% means that the entire build took longer to run than the longest test within it, which is an indicator that all the tests in the build are running in parallel. If, on the other hand, the build in our first example ran in 115 seconds compared to the longest test of 60 seconds, its efficiency would be around 52% because the tests are clearly not running in parallel.

The following table provides some guidance for how you might improve your build efficiency based on your score.

|Efficiency|Degree of Parallelization|Guidance|
|---|---|---------|
|0%	|Sequential	|The build took as long to run as the sum of each run time of all the tests within it, which means that the tests ran in sequential order. Consider using a test framework to run your tests in parallel. |
|1 - 90%	|Semi-parallel	|The build took less time to run than the sum of all test run times, which means that some tests ran in parallel and some ran in sequential order. Consider reorganizing your tests into small, atomic, and autonomous validations of very focused functionality to make sure they aren't dependent on one another to complete before they can execute.|
|91 - 100%	|Parallel	|The build took approximately the same amount of time to run as the longest test within it, meaning that most, if not all, the tests ran simultaneously. You can still potentially improve the overall efficiency of your build by breaking your longer running tests into smaller, shorter tests, if possible. In the benchmarking example for Build A, if T2 could be broken down into two tests that ran for 30 seconds each, you would improve the efficiency of that build by 25%, since the longest running test within it would be 45 seconds instead of 60 seconds.|

See our [Short Tests in Parallel](https://saucelabs.com/blog/speeding-up-your-tests-short-tests-in-parallel) blog for a more comprehensive discussion about improving the speed and efficiency of your builds.

repository: leftxs/sauce-docs

branch_name: refs/heads/main

repo_name: leftxs/sauce-docs


text:
---
sidebar_label: SnapshotOptions.interestingOnly
---

# SnapshotOptions.interestingOnly property

Prune uninteresting nodes from the tree.

#### Signature:

```typescript
interface SnapshotOptions {
  interestingOnly?: boolean;
}
```

#### Default value:

true

gha_language: TypeScript


text:
# PostgreSQL for Schemamama

A PostgreSQL adapter for the lightweight database migration system [Schemamama](https://github.com/SkylerLipthay/schemamama). Depends on the `postgres` crate.

## Installation

If you're using Cargo, just add Schemamama to your `Cargo.toml`:

```toml
[dependencies]
schemamama = "0.3"
schemamama_postgres = "0.3"
postgres = "0.17"
```

## Usage

First, define some migrations:

```rust
#[macro_use]
extern crate schemamama;
extern crate schemamama_postgres;
extern crate postgres;

use postgres::{Client, Transaction, error::Error as PostgresError, NoTls};
use schemamama::{Migration, Migrator};
use schemamama_postgres::{PostgresAdapter, PostgresMigration};

struct CreateUsers;
// Instead of using sequential numbers (1, 2, 3...), you may choose to use a collaborative
// versioning scheme, such as epoch timestamps.
migration!(CreateUsers, 1, "create users table");

impl PostgresMigration for CreateUsers {
    fn up(&self, transaction: &mut Transaction) -> Result<(), PostgresError> {
        transaction.execute("CREATE TABLE users (id BIGINT PRIMARY KEY);", &[]).map(|_| ())
    }

    fn down(&self, transaction: &mut Transaction) -> Result<(), PostgresError> {
        transaction.execute("DROP TABLE users;", &[]).map(|_| ())
    }
}

struct CreateProducts;
migration!(CreateProducts, 2, "create products table");

impl PostgresMigration for CreateProducts {
    // ...
}
```

Then, run the migrations!

```rust
let url = "postgres://postgres@localhost";
let client = Client::connect(url, NoTls).unwrap();
let adapter = PostgresAdapter::new(&mut client);
// Create the metadata tables necessary for tracking migrations. This is safe to call more than
// once (`CREATE TABLE IF NOT EXISTS schemamama` is used internally):
adapter.setup_schema().unwrap();

let mut migrator = Migrator::new(adapter);

migrator.register(Box::new(CreateUsers));
migrator.register(Box::new(CreateProducts));

// Execute migrations all the way upwards:
migrator.up(None);
assert_eq!(migrator.current_version(), Some(2));

// Reverse all migrations:
migrator.down(None);
assert_eq!(migrator.current_version(), None);
```

## Testing

To run `cargo test`, you must have PostgreSQL running locally with a user role named `postgres` with login access to a database named `postgres`. All tests will work in the `pg_temp` schema, so the database will not be modified.


text:
### Contribute to the core

Chyrp is developed using a community approach and we encourage participation. If you are interested in
contributing please take a second to read our [Coding Style](STYLE). Any contributions that do not meet these guidelines will not be accepted. Otherwise:
fork, commit & make pull-request. If you don't know how: [http://help.github.com/forking/]
(http://help.github.com/forking/)

### Git the Source

Another method of grabbing Chyrp and staying up to date with its development, is to use our Git repository.
            To do this you must have Git installed.

* Initial clone:

git clone git://github.com/chyrp/chyrp.git

* Stay updated:

git pull # and then run upgrade.php

### Contribute a module/feather/theme

Do you have something that might be of interest to other Chyrp users? Create a [module](https://github.com/chyrp/chyrp/wiki/Anatomy-of-a-Module) maybe?

If it's properly coded, and it complies with our [Coding Style](STYLE), you can add it to the Extend section, we can publish it on this page and maybe even bundle it with Chyrp future releases if it adds benefit for Chyrp users.

repository: chyrp/chyrp

gha_language: PHP

gha_license_id: NOASSERTION

repo_name: chyrp/chyrp


text:
# Donation Requests engine for Refinery CMS.

## How to build this engine as a gem

    cd vendor/engines/donation_requests
    gem build refinerycms-donation_requests.gempspec
    gem install refinerycms-donation_requests.gem

    # Sign up for a http://rubygems.org/ account and publish the gem
    gem push refinerycms-donation_requests.gem

repository: aarswells/AndrewsDistributing

repo_name: aarswells/AndrewsDistributing


text:
### Symbol Tables

The primary purpose of a symbol table is to associate a value with a key. The client can insert key-value pairs into the symbol table with the expectation of later being able to search for the value associated with a given key, from among all of the key-value pairs that have been put into the table.

Search is so important to so many computer applications that symbol tables are available as high-level abstractions in many programming environments, including Java—we shall discuss Java’s symbol-table implementations in Section 3.5.

#### Definition

A symbol table is a data structure for key-value pairs that supports two operations: insert (put) a new pair into the table and search for (get) the value associated with a given key.

        application
        purpose of search
        key
        value

        dictionary
        find definition
        word
        definition

        book index
        find relevant pages
        term
        list of page numbers

        file share
        find song to download
        name of song
        computer ID

        account management
        process transactions
        account number
        transaction details

        web search
        find relevant web pages
        keyword
        list of page names

        compiler
        find type and value
        variable name
        type and value

Typical symbol-table applications

##### Generics

As we did with sorting, we will consider the methods without specifying the types of the items being processed, using generics. For symbol tables, we emphasize the separate roles played by keys and values in search by specifying the key and value types explicitly instead of viewing keys as implicit in items as we did for priority queues in Section 2.4.

##### Duplicate keys
We adopt the following conventions in all of our implementations:

* Only one value is associated with each key (no duplicate keys in a table).
* When a client puts a key-value pair into a table already containing that key (and an associated value), the new value replaces the old one.

##### Null keys

Keys must not be null. As with many mechanisms in Java, use of a null key results in an exception at runtime (see the third Q&A on page 387).

##### Null values

We also adopt the convention that no key can be associated with the value null. This convention is directly tied to our specification in the API that get() should return null for keys not in the table, effectively associating the value null with every key not in the table.

##### Deletion

Deletion in symbol tables generally involves one of two strategies: lazy deletion, where we associate keys in the table with null, then perhaps remove all such keys at some later time; and eager deletion, where we remove the key from the table immediately.

#### Shorthand methods

For clarity in client code, we include the methods contains() and isEmpty() in the API, with the default one-line implementations shown here.

##### Iteration

To enable clients to process all the keys and values in the table, we might add the phrase implements Iterable<Key> to the first line of the API to specify that every implementation must implement an iterator() method that returns an iterator having appropriate implementations of hasNext() and next(), as described for stacks and queues in Section 1.3.

##### Key equality

Determining whether or not a given key is in a symbol table is based on the concept of object equality, which we discussed at length in Section 1.2 (see page 102).

#### Ordered symbol tables

In typical applications, keys are Comparable objects, so the option exists of using the code a.compareTo(b) to compare two keys a and b. Several symbol-table implementations take advantage of order among the keys that is implied by Comparable to provide efficient implementations of the put() and get() operations.

##### Minimum and maximum

Perhaps the most natural queries for a set of ordered keys are to ask for the smallest and largest keys. We have already encountered these operations, in our discussion of priority queues in Section 2.4. In ordered symbol tables, we also have methods to delete the maximum and minimum keys (and their associated values).

##### Floor and ceiling

Given a key, it is often useful to be able to perform the floor operation (find the largest key that is less than or equal to the given key) and the ceiling operation (find the smallest key that is greater than or equal to the given key).

##### Rank and selection

The basic operations for determining where a new key fits in the order are the rank operation (find the number of keys less than a given key) and the select operation (find the key with a given rank).

##### Range queries

How many keys fall within a given range (between two given keys)? Which keys fall in a given range? The two-argument size() and keys() methods that answer these questions are useful in many applications, particularly in large databases.

##### Exceptional cases

When a method is to return a key and there is no key fitting the description in the table, our convention is to throw an exception (an alternate approach, which is also reasonable, would be to return null in such cases).

##### Shorthand methods

As we have already seen with isEmpty() and contains() in our basic API, we keep some redundant methods in the API for clarity in client code.

##### Key equality (revisited)

The best practice in Java is to make compareTo() consistent with equals() in all Comparable types.

##### Cost model

Whether we use equals() (for symbol tables where keys are not Comparable) or compareTo() (for ordered symbol tables with Comparable keys), we use the term compare to refer to the operation of comparing a symboltable entry against a search key.

#### Sample clients

While we defer detailed consideration of applications to Section 3.5, it is worthwhile to consider some client code before considering implementations.

##### Test client

For tracing our algorithms on small inputs we assume that all of our implementations use the test client below, which takes a sequence of strings from standard input, builds a symbol table that associates the value i with the ith string in the input, and then prints the table.

#### Performance client

FrequencyCounter (on the next page) is a symbol-table client that finds the number of occurrences of each string (having at least as many characters as a given threshold length) in a sequence of strings from standard input, then iterates through the keys to find the one that occurs the most frequently. This client is an example of a dictionary client, an application that we discuss in more detail in Section 3.5.

FrequencyCounter is surrogate for a very common situation. Specifically, it has the following characteristics, which are shared by many other symbol-table clients:

* Search and insert operations are intermixed.
* The number of distinct keys is not small.
* Substantially more searches than inserts are likely.
* Search and insert patterns, though unpredictable, are not random.

##### Sequential search in an unordered linked list

One straightforward option for the underlying data structure for a symbol table is a linked list of nodes that contain keys and values, as in the code on the facing page.

##### Proposition A

Search misses and insertions in an (unordered) linked-list symbol table having N key-value pairs both require N compares, and search hits N compares in the worst case. In particular, inserting N distinct keys into an initially empty linked-list symbol table uses ~![](http://latex.codecogs.com/gif.latex?\frac{N^2}2) compares.

##### Corollary

Inserting N distinct keys into an initially empty linked-list symbol table uses ~![](http://latex.codecogs.com/gif.latex?\frac{N^2}2) compares.

#### Binary search in an ordered array

Next, we consider a full implementation of our ordered symbol-table API. The underlying data structure is a pair of parallel arrays, one for the keys and one for the values.

##### Binary search

The reason that we keep keys in an ordered array is so that we can use array indexing to dramatically reduce the number of compares required for each search, using the classic and venerable binary search algorithm that we used as an example in Chapter 1.

This recursive rank() preserves the following properties:

* If key is in the table, rank() returns its index in the table, which is the same as
the number of keys in the table that are smaller than key.
* If key is not in the table, rank() also returns the number of keys in the table
that are smaller than key.

##### Other operations

Since the keys are kept in an ordered array, most of the order-based operations are compact and straightforward, as you can see from the code on page 382. For example, a call to select(k) just returns keys[k].

##### Analysis of binary search

The recursive implementation of rank() also leads to an immediate argument that binary search guarantees fast search, because it corresponds to a recurrence relation that describes an upper bound on the number of compares.

##### Proposition B

Binary search in an ordered array with N keys uses no more than lg N  1 compares for a search (successful or unsuccessful).

##### Proposition B (continued)

Inserting a new key into an ordered array of size N uses ~ 2N array accesses in the worst
case, so inserting N keys into an initially empty table uses ~![](http://latex.codecogs.com/gif.latex?N^2) array accesses in the worst case.

#### Preview

Binary search is typically far better than sequential search and is the method of choice in numerous practical applications.

#### Full implementation of Binary Search in Symbol Table

```
public class BinarySearch<Key implements Comparable<Key>, Value> {

    private Key[] keys;
    private Value[] values;

    public BinarySearchST(int capacity){
        keys=(Key[])new Comparable[capacity];
        values=(Value[])new Object[capacity];
    }

    public int size(){
        return N;
    }

    public Value get(Key key){
        if(isEmpty())
            return null;
        int i=rank(key);
        if(i<N&&keys[i].compareTo(key)==0)
            return values[i];
        else
            return null;
    }

    public int rand(Key key){
        int low=0,high=N-1;
        while(low<=high<N-1){
            int mid=low+(high-low)/2;
            int cmp=key.compareTo(keys[mid]);
            if(cmp<0)
                high=mid-1;
            else if(cmp>0)
                low=mid+1;
            else
                return mid;
        }
    }

    public void put(Key key, Value value){
        int i=rank(key);
        if(i<N&&keys[i].copareTo(key)==0){
            values[i]=value;
            return;
        }
        for(int j=N;j>i;j--){
            keys[j]=keys[j-1];
            values[j]=values[j-1];
        }
        keys[i]=key;
        values[i]=value;
        N++;
    }

    public void delete(Key key);

    public Key min(){
        return keys[0];
    }

    public Key max(){
        return keys[N-1];
    }

    public Key select(int k){
        return keys[k];
    }

    public Key ceiling(Key key){
        int i=rank(key);
        return keys[i];
    }

    public Key floor(Key key);

    public Key delete(Key key);

    public Iterable<Key> keys(Key low, Key high){
        Queue<Key> q=new Queue<Key>();
        for(int i=rank(low);i<rank(high);i++){
            q.enqueue(keys[i]);
        }
        if(contains(high)){
            q.enqueue(keys[rank(high)]);
        }
        return q;
    }
}
```

        algorithm(data structure)
        worst-case cost(after N inserts)
        average-case cost(after N random inserts)
        efficiently support ordered operations?

        search
        insert
        search hit
        insert

        sequential search(unordered linked list)
        N
        N

        N
        No

        binary search(ordered array)

        2N

        N
        yes

BinarySearchST costs

        underlying data structure
        implementation
        pros
        cons

        linked list(sequential search)
        SequentialSearchST
        best for tiny STs
        slow for large STs

        ordered array(binary search)
        BinarySearchST
        optimal search and space, order-based ops
        slow insert

        binary search tree
        BST
        easy to implement, order-based ops
        no guarantees space for links

        balanced BST
        RedBlackBST
        optimal search and insert, order-based ops
        space for links

        hash table
        SeparateChainingHashST
LinearProbingHashST
        fast search/insert for common types of data
        need hash for each type no order-based ops space for links/empty

Cost summary for basic symbol-table implementations

repository: annjeff/algorithm-notes

filename: Symbol Tables.md

repo_name: annjeff/algorithm-notes


text:
English | [简体中文](README.md)

# Mobile Model Zoo

## Models

This directory contains models optimized for mobile applications, at present the following models included:

| Backbone                 | Architecture              | Input | Image/gpu [1](#gpu) | Lr schd       | Box AP | Download | PaddleLite Model Download |
| :----------------------- | :------------------------ | :---: | :--------------------: | :------------ | :----: | :------- | :------------------------ |
| MobileNetV3 Small        | SSDLite                   | 320   | 64                     | 400K (cosine) | 16.2   | [Link](https://paddlemodels.bj.bcebos.com/object_detection/mobile_models/ssdlite_mobilenet_v3_small.pdparam) | [Link](https://paddlemodels.bj.bcebos.com/object_detection/mobile_models/lite/ssdlite_mobilenet_v3_small.tar) |
| MobileNetV3 Small        | SSDLite Quant [2](#quant) | 320   | 64                     | 400K (cosine) | 15.4   | [Link](https://paddlemodels.bj.bcebos.com/object_detection/mobile_models/ssdlite_mobilenet_v3_small_quant.tar) | [Link](https://paddlemodels.bj.bcebos.com/object_detection/mobile_models/lite/ssdlite_mobilenet_v3_small_quant.tar) |
| MobileNetV3 Large        | SSDLite                   | 320   | 64                     | 400K (cosine) | 23.3   | [Link](https://paddlemodels.bj.bcebos.com/object_detection/mobile_models/ssdlite_mobilenet_v3_large.pdparam) | [Link](https://paddlemodels.bj.bcebos.com/object_detection/mobile_models/lite/ssdlite_mobilenet_v3_large.tar) |
| MobileNetV3 Large        | SSDLite Quant [2](#quant) | 320   | 64                     | 400K (cosine) | 22.6   | [Link](https://paddlemodels.bj.bcebos.com/object_detection/mobile_models/ssdlite_mobilenet_v3_large_quant.tar) | [Link](https://paddlemodels.bj.bcebos.com/object_detection/mobile_models/lite/ssdlite_mobilenet_v3_large_quant.tar) |
| MobileNetV3 Large w/ FPN | Cascade RCNN              | 320   | 2                      | 500k (cosine) | 25.0   | [Link](https://paddlemodels.bj.bcebos.com/object_detection/mobile_models/cascade_rcnn_mobilenetv3_fpn_320.tar) | [Link](https://paddlemodels.bj.bcebos.com/object_detection/mobile_models/lite/cascade_rcnn_mobilenetv3_fpn_320.tar) |
| MobileNetV3 Large w/ FPN | Cascade RCNN              | 640   | 2                      | 500k (cosine) | 30.2   | [Link](https://paddlemodels.bj.bcebos.com/object_detection/mobile_models/cascade_rcnn_mobilenetv3_fpn_640.tar) | [Link](https://paddlemodels.bj.bcebos.com/object_detection/mobile_models/lite/cascade_rcnn_mobilenetv3_fpn_640.tar) |
| MobileNetV3 Large        | YOLOv3                    | 320   | 8                      | 500K          | 27.1   | [Link](https://paddlemodels.bj.bcebos.com/object_detection/yolov3_mobilenet_v3.pdparams) | [Link](https://paddlemodels.bj.bcebos.com/object_detection/mobile_models/lite/yolov3_mobilenet_v3.tar) |
| MobileNetV3 Large        | YOLOv3 Prune 2 | 320   | 8                      | -             | 24.6   | [Link](https://paddlemodels.bj.bcebos.com/object_detection/mobile_models/yolov3_mobilenet_v3_prune75875_FPGM_distillby_r34.pdparams) | [Link](https://paddlemodels.bj.bcebos.com/object_detection/mobile_models/lite/yolov3_mobilenet_v3_prune86_FPGM_320.tar) |

**Notes**:

-   [1] All models are trained on 8 GPUs.
-   [2] See the note section on [SSDLite quantization](#Notes-on-SSDLite-quant)。
-   [3] See the note section on [how YOLO head is pruned](#Notes-on-YOLOv3-pruning).

## Benchmarks Results

-   Models are benched on following chipsets with [Paddle-Lite](https://github.com/PaddlePaddle/Paddle-Lite) 2.6 (to be released)
    -   Qualcomm Snapdragon 625
    -   Qualcomm Snapdragon 835
    -   Qualcomm Snapdragon 845
    -   Qualcomm Snapdragon 855
    -   HiSilicon Kirin 970
    -   HiSilicon Kirin 980

-   With 1 CPU thread (latency numbers are in ms)

    |                  | SD625   | SD835   | SD845   | SD855   | Kirin 970 | Kirin 980 |
    |------------------|---------|---------|---------|---------|-----------|-----------|
    | SSDLite Large    | 289.071 | 134.408 | 91.933  | 48.2206 | 144.914   | 55.1186   |
    | SSDLite Large Quant |  |  |   |  |    |    |
    | SSDLite Small    | 122.932 | 57.1914 | 41.003  | 22.0694 | 61.5468   | 25.2106   |
    | SSDLite Small Quant |  |  |  | |   |    |
    | YOLOv3 baseline  | 1082.5  | 435.77  | 317.189 | 155.948 | 536.987   | 178.999   |
    | YOLOv3 prune     | 253.98  | 131.279 | 89.4124 | 48.2856 | 122.732   | 55.8626   |
    | Cascade RCNN 320 | 286.526 | 125.635 | 87.404  | 46.184  | 149.179   | 52.9994   |
    | Cascade RCNN 640 | 1115.66 | 495.926 | 351.361 | 189.722 | 573.558   | 207.917   |

-   With 4 CPU threads (latency numbers are in ms)

    |                  | SD625   | SD835   | SD845   | SD855   | Kirin 970 | Kirin 980 |
    |------------------|---------|---------|---------|---------|-----------|-----------|
    | SSDLite Large    | 107.535 | 51.1382 | 34.6392 | 20.4978 | 50.5598   | 24.5318   |
    | SSDLite Large Quant |  |  |   |  |    |    |
    | SSDLite Small    | 51.5704 | 24.5156 | 18.5486 | 11.4218 | 24.9946   | 16.7158   |
    | SSDLite Small Quant |  |  |  | |   |    |
    | YOLOv3 baseline  | 413.486 | 184.248 | 133.624 | 75.7354 | 202.263   | 126.435   |
    | YOLOv3 prune     | 98.5472 | 53.6228 | 34.4306 | 21.3112 | 44.0722   | 31.201    |
    | Cascade RCNN 320 | 131.515 | 59.6026 | 39.4338 | 23.5802 | 58.5046   | 36.9486   |
    | Cascade RCNN 640 | 473.083 | 224.543 | 156.205 | 100.686 | 231.108   | 138.391   |

## Notes on SSDLite quantization

We use a complete quantitative training method to train the SSDLite model. It is trained for a total of 400,000 rounds with the 8-card GPU. We freeze `res_conv1` and `se_block`. The command used is listed bellow:

```shell
python slim/quantization/train.py --not_quant_pattern res_conv1 se_block \
                -c configs/ssd/ssdlite_mobilenet_v3_large.yml \
                --eval
```

For more quantization tutorials, please refer to [Model Quantization Compression Tutorial](../../docs/advanced_tutorials/slim/quantization/QUANTIZATION.md)

## Notes on YOLOv3 pruning

We pruned the YOLO-head and distill the pruned model with YOLOv3-ResNet34 as the teacher, which has a higher mAP on COCO (31.4 with 320\*320 input).

The following configurations can be used for pruning:

-   Prune with fixed ratio, overall prune ratios is 86%

    ```shell
    --pruned_params="yolo_block.0.0.0.conv.weights,yolo_block.0.0.1.conv.weights,yolo_block.0.1.0.conv.weights,yolo_block.0.1.1.conv.weights,yolo_block.0.2.conv.weights,yolo_block.0.tip.conv.weights,yolo_block.1.0.0.conv.weights,yolo_block.1.0.1.conv.weights,yolo_block.1.1.0.conv.weights,yolo_block.1.1.1.conv.weights,yolo_block.1.2.conv.weights,yolo_block.1.tip.conv.weights,yolo_block.2.0.0.conv.weights,yolo_block.2.0.1.conv.weights,yolo_block.2.1.0.conv.weights,yolo_block.2.1.1.conv.weights,yolo_block.2.2.conv.weights,yolo_block.2.tip.conv.weights" \
    --pruned_ratios="0.75,0.75,0.75,0.75,0.75,0.75,0.75,0.75,0.75,0.75,0.75,0.75,0.875,0.875,0.875,0.875,0.875,0.875"
    ```
-   Prune filters using [FPGM](https://arxiv.org/abs/1811.00250) algorithm:

    ```shell
    --prune_criterion=geometry_median
    ```

## Upcoming

-   [ ] More models configurations
-   [ ] Quantized models

repository: ellinyang/PaddleDetection

repo_name: ellinyang/PaddleDetection


text:
# legged_control
## Introduction
legged_control is an NMPC-WBC legged robot control stack and framework based on [OCS2](https://github.com/leggedrobotics/ocs2) and [ros-control](http://wiki.ros.org/ros_control).

The advantage shows below:
1. To the author's best knowledge, this framework is probably the best-performing open-source legged robot MPC control framework;
2. You can deploy this framework in your A1 robot within a few hours;
3. Thanks to the ros-control interface, you can easily use this framework for your custom robot.

I believe this framework can provide a high-performance and easy-to-use model-based baseline for the legged robot community.

https://user-images.githubusercontent.com/21256355/192135828-8fa7d9bb-9b4d-41f9-907a-68d34e6809d8.mp4

## Installation
### Source code
The source code is hosted on GitHub: [qiayuanliao/legged_control](https://github.com/qiayuanliao/legged_control).
```
# Clone legged_control
git clone git@github.com:qiayuanliao/legged_control.git
```

### OCS2
OCS2 is a huge monorepo; **DO NOT** try to compile the whole repo. You only need to compile `ocs2_legged_robot_ros` and its dependencies following the step below.

1. You are supposed to clone the OCS2, pinocchio, and hpp-fcl as described in the documentation of OCS2.
   ```
   # Clone OCS2
   git clone git@github.com:leggedrobotics/ocs2.git
   # Clone pinocchio
   git clone --recurse-submodules https://github.com/leggedrobotics/pinocchio.git
   # Clone hpp-fcl
   git clone --recurse-submodules https://github.com/leggedrobotics/hpp-fcl.git
   # Clone ocs2_robotic_assets
   git clone https://github.com/leggedrobotics/ocs2_robotic_assets.git
   # Install dependencies
   sudo apt install liburdfdom-dev liboctomap-dev libassimp-dev
   ```
2. Compile the `ocs2_legged_robot_ros` package with [catkin tools](https://catkin-tools.readthedocs.io/en/latest/) instead of `catkin_make`. It will take you about ten minutes.
   ```
   catkin config -DCMAKE_BUILD_TYPE=RelWithDebInfo
   catkin build ocs2_legged_robot_ros
   ```
   Ensure you can command the ANYmal as shown in the [document](https://leggedrobotics.github.io/ocs2/robotic_examples.html#legged-robot) and below.
   ![](https://leggedrobotics.github.io/ocs2/_images/legged_robot.gif)

### Build
Build the source code of `legged_control` by:
```
catkin build legged_controllers unitree_description
```

Build the simulation (**DO NOT** run on the onboard computer)
```
catkin build legged_gazebo
```
Build the hardware interface real robot. If you use your computer only for simulation, you **DO NOT** need to compile `unitree_hw` (TODO: add a legged prefix to the package name)
```
catkin build unitree_hw
```

## Quick Start

1. Set your robot type as an environment variable: ROBOT_TYPE

```
export ROBOT_TYPE=a1
```

2. Run the simulation:

```
roslaunch unitree_description empty_world.launch
```
Or on the robot hardware:
```
roslaunch unitree_hw unitree_hw.launch
```

3. Load the controller:

```
roslaunch legged_controllers load_controller.launch cheater:=false
```

4. Start the `legged_controller` or `legged_cheater_controller`, **NOTE that you are not allowed to start the `legged_cheater_controller` in real hardware!**

```
rosservice call /controller_manager/switch_controller "start_controllers: ['controllers/legged_controller']
stop_controllers: ['']
strictness: 0
start_asap: false
timeout: 0.0"
```

Or, you can start the controller using `rqt_controller_manager` GUI:

```
sudo apt install ros-noetic-rqt-controller-manager
rosrun rqt_controller_manager rqt_controller_manager
```

5. Set the gait in the terminal of `load_controller.launch`, then use RViz (you need to add what you want to display by yourself) and control the robot by `cmd_vel` and `move_base_simple/goal`:

![ezgif-5-684a1e1e23.gif](https://s2.loli.net/2022/07/27/lBzdeRa1gmvwx9C.gif)

### Note
-  **THE GAIT AND THE GOAL ARE COMPLETELY DIFFERENT AND SEPARATED!**  You don't need to type stance while the robot is lying on the ground **with four foot touching the ground**; it's completely wrong since the robot is already in the stance gait.
-  The target_trajectories_publisher is for demonstration. You can combine the trajectory publisher and gait command into a very simple node to add gamepad and keyboard input for different gaits and torso heights and to start/stop controller (by ros service).

## Framework
The system framework diagram is shown below.

![](docs/system_diagram.png)

- The robot torso's desired velocity or position goal is converted to state trajectory and then sent to the NMPC;
- The NMPC will evaluate an optimized system state and input.
- The Whole-body Controller (WBC) figures out the joint torques according to the optimized states and inputs from the NMPC.
- The torque is set as a feed-forward term and is sent to the robot's motor controller.
Low-gain joint-space position and velocity PD commands are sent to the robot's motors to reduce the shock during foot contact and for better tracking performance.
- The NMPC and WBC need to know the current robot state, the base orientation, and the joint state, all obtained directly from the IMU and the motors. Running in the same loop with WBC, a linear Kalman filter[1] estimates the base position and velocity from base orientation, base acceleration, and joint foot position measurements.

## Module
The main module of the entire control framework is NMPC and WBC, and the following is only a very brief introduction.

While the cost function is simply the quadratic cost of tracking the error of all states and the input, the system dynamics uses centroidal dynamics with the following constraints:

- Friction cone;
- No motion at the standing foot;
- The z-axis position of the swinging foot satisfies the gait-generated curve.

To solve this optimal control problem, a multiple shooting is formulated to transcribe the optimal control problem to a nonlinear program (NLP) problem, and the NLP problem is solved using Sequential Quadratic Programming (SQP). The QP subproblem is solved using HPIPM. For more details [2, 3]

### WBC
![](docs/tasks.png)

WBC only considers the current moment. Several tasks are defined in the table above. Each task is the equality constraints or inequality constraints on decision variables. The decision variables of WBC are:

$$
\mathbf{x}_{wbc} = [\ddot{\mathbf{q}}^T, \mathbf{f}_c^T, \mathbf{\tau}^T]^T
$$

where $\ddot{\mathbf{q}}$ is acceleration of generalized coordinate, $\mathbf{\tau}$ is the joint torque. The WBC solves the QP problem in the null space of the higher priority tasks' linear constraints and tries to minimize the slacking variables of inequality constraints. This approach can consider the full nonlinear rigid body dynamics and ensure strict hierarchy results. For more details [4].

## Deploy and Develop
### A1 robot

People with ROS foundation should be able to run through simulation and real machine deployment within a few hours. The following shows some known laboratories that have run through this framework on their own A1 objects. and the time spent
The table below shows the labs successfully deploying this repo in their **real A1**; feel free to open a PR to update it. (because the code they got at the time was not stable, so the spend time cannot represent the their level).

| Lab | XPeng Robotics | Unitree | Hybrid Robotics |
| ---- | ---- | ---- | ---- |
| Spend Time | 1 day | - | 2 hours |

I recommended to use an external computing device such as NUC to run this control framework. The author uses the 11th generation of NUC, and the computing frequency of NMPC can be close to 200Hz.

### Your costum rorbots

Deploying this framework to your robot is very simple, the steps are as follows:

- Imitate the `UnitreeHW` class in legged_examples/legged_unitree/unitree_hw
, inherit `LeggedHW` and implement the `read()` and `write()` functions of the hardware interface;
- Imitate the legged_examples/legged_unitree/unitree_description, write the xarco of the robot and generate the URDF file, note that the names of the joint and link need to be the same as unitree_description.

## Reference

[1] T. Flayols, A. Del Prete, P. Wensing, A. Mifsud, M. Benallegue, and O. Stasse, “Experimental evaluation of simple estimators for humanoid robots,” IEEE-RAS Int. Conf. Humanoid Robot., pp. 889–895, 2017, doi: 10.1109/HUMANOIDS.2017.8246977.

[2] J. P. Sleiman, F. Farshidian, M. V. Minniti, and M. Hutter, “A Unified MPC Framework for Whole-Body Dynamic Locomotion and Manipulation,” IEEE Robot. Autom. Lett., vol. 6, no. 3, pp. 4688–4695, 2021, doi: 10.1109/LRA.2021.3068908.

[3] R. Grandia, F. Jenelten, S. Yang, F. Farshidian, and M. Hutter, “Perceptive Locomotion through Nonlinear Model Predictive Control,” (submitted to) IEEE Trans. Robot., no. August, 2022, doi: 10.48550/arXiv.2208.08373.

[4] C. Dario Bellicoso, C. Gehring, J. Hwangbo, P. Fankhauser, and M. Hutter, “Perception-less terrain adaptation through whole body control and hierarchical optimization,” in IEEE-RAS International Conference on Humanoid Robots, 2016, pp. 558–564, doi: 10.1109/HUMANOIDS.2016.7803330.

repository: CiaranZ/CIaran

repo_name: CiaranZ/CIaran