text:
---
layout: post
title: Academic Papers Need 5 Abstracts
date: 2018-01-22 13:28:19
categories: musings
---

Recently, I've been learning in two ways, under two different sets of
circumstances. At work, I've been learning about modern climate change
models, and in particular, the statistical methods that go into
generating curring-edge climate predictions, policy, and data analysis.
This has been really interesting! It's also somewhat stressful, given
the nature of how my livelihood is tied to understanding those papers.

The other way that I've been learning is by taking up (**big
sigh**) guitar in my spare time. I've been playing the drums for a while
and learned just a touch of music theory when I was younger, so I'm not
quite starting out at square one. Much like complicated statistical
models, there's a ton of theory and skill that goes into understanding
how talented guitarists play. This has been really interesting! But
generally, my comprehension of music theory and those damned instruments
that have pitch (read: pretty much all of them, other than _some_ percussion) is
at a much lower level than my comprehension of climate models.

So there's been a lot of learning. Over the holidays, I fell into a trap
most everyone falls into these days... The YouTube hole. And from that
YouTube hole I discovered this:

[![Fall into Youtube hole at your own risk](https://img.youtube.com/vi/eRkgK4jfi6M/0.jpg)](https://www.youtube.com/watch?v=eRkgK4jfi6M)

I didn't know who Jacob Collier was before watching this video. I did know who Herbie Hancock
was. But this video! This video series! I love it, and here's why:

In the video, harmony is explained 5 times. The first time, it takes
about 10 seconds. The second, about 30. The third? A couple minutes. And
we go on in that manner until jazz legend Herbie Hancock and
best-of-the-musical-petri-dish-prodigal-son Jacob Collier spend a while
doing a little talking, some playing, and a lot of
talking-through-playing as they explain what harmony is by showing, not
telling. It's brilliant.

As I wade through academic papers, I wish I had these 5 explanations.
When you read a paper, you generally get two, maybe two and a half of
these explanations. Working backwards through those 5 explanations, an
academic paper definitely has an explanation for other experts (the 5th
explanation). There's an introduction, which may or may not count as a
complete distillation of the ideas in the paper. We'll call this a half.
Then there's an abstract or executive summary, where the main ideas of
the paper are conveyed in a couple hundred words. Depending on how you
feel about introductions, this is either the 3rd or 4th explanation.

Often, researchers or students have to read tens or hundreds of papers in
a short period of time. Why not give the 1st and 2nd explanation too?
This would have benefits for both authors and readers. Richard Feynmann,
famed physicist and science explainer, is known for claiming that if
somebody can't distill something to at least a college freshman level of
understanding, they don't really understand what's happening. He took
pride in making very complicated topics very easy to understand, most
notably in his Feynmann Diagrams, a series of lines that explain quantum
mechanics (truly).

It wouldn't take much time to write, and would save readers countless
hours. Give us the 1st and 2nd explanations! Why not?


text:
---
layout: post
title:  "Chapter 12"
date:   2015-01-14 23:30:00 +0900
subtitle: ""
author: Madao
background: '/img/posts/vol-1.jpg'
tag: Volume 1
---
「How is the cultivation going?」

「I have confirmed that the cultivation is progressing smoothly at all the 6 locations. The harvest amount is also roughly as assumed」

「Then, I want to expand it as soon as possible, but……..」

「Regretfully, if we enlarge the scope of it any more than this, we won’t have enough manpower」

「And if it isn’t inspected, farmhouses which selfishly start their own cultivation due to being dazzled by the profits will also appear」

It had been 2 weeks since the start of trial run for the implementation of  LP farming, and Kazuki, while confirming the results with Norman and Jake, was discussing about the prospects from here on out with them.
The place was obviously Harold’s room. Recently, he was seldom alone in the room. Anyone out of Norman, Jake and Zen being in the room became numerous.

「I can’t do anything about the deficiency in manpower. Isn’t there anybody else in this mansion we can use?」

「It is difficult in the present condition. If we ask Elias or Saxon to cooperate, it will just be a temporary measure……….」

Norman raised up the names of the 2 soldiers who helped in the plan to rescue Clara.
But asking them to go to the agricultural district for inspection on top of their original work, it wasn’t very realistic. As Norman himself had stated, it would just become a temporary measure.

「Tsk, then how about I hire people from the outside…………?」

「From the outside, is it?」

「For example, if I ask my father that I want personal aides…………No, that’ll be no good. If it’s some guy recommended by father, moving around will become even more harder」

Jake stopped Kazuki (Harold), who seemed to be immersed while murmuring to himself.

「Harold-sama, first I want to continue the report」

「Nn? Ah, the matter about the soil」

「Yes. Regarding the soil of the testing field, no huge changes were observed」

In short, the testing field was the field present in Jake’s house. He had been cultivating nothing but the 3 types of vegetable there.
The thing he had wanted to confirm was whether problems related to the soil would arise when they did this over a long period of time or when they used a large amount of LP. If due to this, the soil withered away and if it couldn’t be used a field for farming anymore, they would need to suspend the use of LP immediately.

「Including this time’s, how many rotations has it been?」

「It’ll be red groot 7 times, bell tubers 6 times and bluna 11 times. I even had the cooks sample them regularly, but there were no differences in the taste」

「The notice isn’t bad, but the number of trials is too less. Continue the cultivation testing and observe the progression. And then………..」

Like this, it became regular for Kazuki to keep them company once in few days and working hard to grasp the current circumstances. Although he did have the sense of fulfillment he hadn’t experienced in his student days, even now his future was full of troubles.
And since he was using his head in a completely different way from how he used it during classes in high school or university, maybe as a recoil, his body started demanding to go rowdy.

And so, after finishing the discussion, holding the sword, Kazuki carried his legs towards the completely familiar opening in the forest expanding behind the mansion, which they had used for the plan to rescue Clara.
It was the only place where he could swing the sword with all of his strength.

As soon as Kazuki arrived, he lightly loosened his body and immediately commenced his normal routine.
The 3 hit combo of slashing downwards diagonally from above, cutting upwards, slicing while rotating.

But the sharpness of the swing and the speed of the sword had reached to a level which couldn’t be compared to when he had just started training. As though it would slice apart the air itself, a swordplay which would overwhelm those who saw it.
But Kazuki’s (Harold’s) sword technique didn’t stop only at that.

Closing his eyes, he calmed his breath and sharpened his nerves.
It became silent and as though to break it, Kazuki started moving.

Within a second, he had unleashed the 3 hit combo, similar to the one before.The difference was in the next part.
The blade which was swung out due to the rotating slash was clad in –Lightning.

「『Raijin(Thunder God)』」

As though in concurrence with that word, the instant the blade stabbed the ground, a bright electric current was fired.
With Kazuki at the center, 8 streaks of electricity attacked the surroundings. One gouged the ground out, one scorched a rock, one broke a tree trunk.

The reach of the streak was about 3 meters.  Kazuki fired an attack which could strike in all the directions, as though it could ignore a numerical disadvantage. But he muttered as though he was dissatisfied.

「It won’t even be useful at this level」

The attack Kazuki had fired off a moment ago was called 『Raijin』,  one of the elementary level skills in 『Brave Hearts』.
It looked to be quite flashy, and when it had succeeded for the first time, Kazuki himself had been frightened, but in the game it was just a weak skill which consumed 5 MP (Magic Points). It could be guessed that since the protagonist would learn it at level 1, it would be no good.

Then the reason why he was dissatisfied was because, mysteriously a feeling that he could “do much better” sprouted out inside him. That feeling wasn’t necessarily wrong since the number of lightning streaks were 4 at the beginning.
Fundamentally, it was possible for Harold to use magic of all attributes, but among them, lightning symbolized him. And since that Harold’s body was shouting “This still isn’t my best”, as though to answer that appeal, Kazuki was single-mindedly swinging his sword to perfect the ideal Raijin that had crossed his mind.

That was why he hadn’t noticed. That the him right now was ascending to heights which couldn’t possibly be reached by a 10 year old child.
That how extremely abnormal it was.
He hadn’t noticed how he would look from another person’s view.

>(This is.., I have seen something serious, haven’t I ～?)

As though nothing had changed inside, Juno had a calm expression, but unusually, there was a drop of sweat flowing down her cheek.
After reaching a certain point in investigating the Stokes house, when she had set out to seriously investigate Harold, on the first day itself she encountered a scene which made her doubt her own eyes.
After meeting him, I’ve done nothing but doubt my five senses- she leaked out a faint but wry smile.

But she couldn’t just keep on smiling. No matter how she thought about it, Harold wasn’t normal.
He was easily swinging an iron sword whose length was as tall as himself, and that sword speed also wouldn’t lose out to that of an experienced swordsman’s. Adding onto that, he could also handle swordsmanship which used magic.

The problem was whether that secret had the risk of causing harm to the Sumeragi family and Erica.

According to the plan prepared beforehand, she had thought of contacting him head on, but she felt that it would be better to rework her plan to include some means of feeling around just in case. Thinking that, the instant she turned on her heels, planning to leave without making any sound, that happened.

Behind Juno who had her back towards Harold, ‘Gan’- with a heavy shocking sound, the air shook.

>(–!?)

To the abrupt situation, Juno reflexively ducked her body.
The identity of the sound was Harold’s sword. Right now, it had left its owner’s hand and was stabbed deeply into the tree, behind which Juno had hidden her body.
It was a surprise attack which had aimed for the gap in the instant she had switched her consciousness. If the tree hadn’t been there, without a doubt, it would have pierced through her.

To that fact, Juno went pale. Even so, the reason she hadn’t given out any voice was because of the results of the training and experience she had undergone.
But due the surprise being too much, she had magnificently stamped down the thicket, which was a fatal error.

「Who is it? Don’t keep sneakily hiding and show yourself」

Harold’s sharp voice rang out.
Juno thought of planning to make a getaway as it was, but thinking that there was no way she could fool him, who had accurately seen through her covert action, she gave up and appeared in front of Harold.

Thereupon, Harold’s eyes widened by a small amount, and then, his expression changed minutely.

>(That just now was relief………was it ～?)

To read that change was the power of Juno’s unique observing eye.
But she didn’t know what it meant.

Compared to that, Kazuki was completely flustered.
Since he had been extremely concentrated on practising Raijin, the sword had slipped out of his right hand and flown towards an unexpected direction, and had stopped after stabbing a tree which was more than 10 meters away.
He was surprised because there was some kind of presence there.

Thinking that maybe it had slammed into a person and injured them, panicking greatly, he called out. On the verge of running off in the direction of the thicket, Juno appeared from the shadow of the tree.
From the looks of it, since she didn’t appear to be injured, Kazuki was relieved. He had almost become a murderer.

「I’m sorry Harold-sama ～, it was unintentional ～」

While Kazuki was worried about wrenching out words of apology even with Harold’s mouth, for some reason Juno lowered her head.
It seemed as though she was apologizing for peeking at Kazuki’s (Harold’s) training. For Kazuki, that didn’t matter at all.

「Hanging around in the shadows, insolent. If you don’t learn of your position, you’ll be injured severely」

If you’re going to watch, stay in a position where I can see you. Since if I don’t know where you are, I might injure you- not even a bit of Kazuki’s concerns were transmitted.
Rather, this was what Juno heard.

>(He’s giving a warning huh ～………)

Easily grasping her presence, without even offering any room to resist, without taking any action and suppressing her, it was brilliantly done. He wasn’t someone who she could take on as an opponent, when there was an existence like Erica near her, who needed to be protected.
He must have also warned her like this since he understood that.

— That there wouldn’t be a next time if she went further than this

It wasn’t that she had the intention to relax because Harold was a child. She also wasn’t optimistic.
But still, he had easily surpassed her. As though he had seen through everything from the beginning.

It was her complete defeat. Was she really aware that she had fought Harold? He had shown her the difference in ability to even make her feel that question.
He not only had physical power, but was also resourceful.

「Fuh, Well fine. By the way, what is the sheltered girl doing?」
>(TL – He calls her hakoirimusume – literally daughter in a box, which basically means that she has been sheltered her whole life and is very naive)

Harold suddenly asked that to Juno, who wasn’t replying.
At the abrupt change in the topic, even while being confused, Juno answered the question.

「Erica-sama’s body still isn’t in a very good condition ～. She might be a bit bewildered by the unfamiliar environment ～」

>(Just for her physical condition being poor, 2 weeks is too long. There shouldn’t have been a setting of her having a weak condition though……….)

After coming back from going around the town, Erica had told that she wasn’t feeling well, and from then on she had confined herself in the room for nearly the whole time and hadn’t appeared. Thanks to that, he was fully devoted to the trial implementation of LP farming, but since it was so long, his concern gradually increased.
Maybe it’s some kind of flag- his crisis perception rang alarm bells. Well, it was already too late though.

「Perhaps, she is homesick.  Even though it’s temporary, since she is your fiancee, wouldn’t it be fine to be a little gentler ～?」

If he could do that, he wouldn’t be so troubled.

「Worthless. As you’ve said, the relationship between us is temporary.  I might have talked a bit too much)

He still couldn’t make it known that he had plans to cancel the engagement in future.
The only person who knew about this was Tasuku, who had read the letter. And although it was Kazuki who had written it, even he didn’t think that Tasuku would completely believe in it.

That was why, at this stage, if somebody came to know that Harold Stokes himself had the intention of cancelling the engagement, the risks would be too high.
At the present stage, he didn’t even have any prospects to prepare the plan.

「There is no need for me to explain it to you」

Without even being able to gloss over it decently, spitting out the words of a sore loser, Kazuki went back to the mansion as though running away from Juno.

Author’s Note-

I think that this kind of development is the classical way of misunderstandings.
Too easy? I don’t hear anything.

Various thoughts and opinions were there, but to be frank, since Juno is a character who appeared only because I wanted to do this in the opening, there’s not much of a deep setting for having her hold the unknown device.
And later, she might not come out normally other than for the role of combat instructor.
And that too, the plan is for her to simply play a connecting role.

Next, I’ll properly write with Erica playing the main part.
Have to make her evaluation of Harold fall even more!

license: MIT

branch_name: refs/heads/master

extension: markdown

license_type: permissive


text:
---
layout: post
title: "Linux JDK Environment Variables Config"
categories: Java JDK
tags: JDK Linux
description: Java config memo
---

* content
{:toc}

Java Develop Kit need to be find in system.

Edit /etc/profile, maybe need to be root.

Append these sentence.

```
export JAVA_HOME=/home/username/JavaDevEnv/jdk[version]
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
```

then logout and login.

Try some command to see if worked.

```
>java -version
```

extension: md


text:
---
title: Go-To
date: 2018-08-21 22:19:50 +0000
category: Long
technique: In glass
ingredients: |-
  1½ oz dry gin
  ¾ oz St-Germain
  ¾ oz lime juice
  2 slice cucumber
  8 mint leaf
  1 top ginger beer
source-url: https://punchdrink.com/recipes/go-to/
image: "/assets/img/go-to.jpg"
layout: recipe-single

---
Combine gin, St. Germain and lime juice in a collins glass.
Add ice then top with mint and cucumber. Stir in glass.
Top with ginger beer.

repository: pfdt/bisou-jekyll-theme

gha_language: JavaScript

gha_license_id: MIT

repo_name: pfdt/bisou-jekyll-theme


text:
# ὡσεί

<!-- Status: S2=NeedsFinalCheck -->
<!-- Lexica used for edits: BDAG, FFM, LN, A-S  -->

## Word data

* Strongs: G56160

* Alternate spellings:

* Principle Parts:

* Part of speech: adverb

* Instances in NT: 21 (in 20 verses)

* All Scriptures cited: Yes

## Etymology:

* LXX/Hebrew glosses:

* Time Period/Ancient Authors:

* Related words:

* Antonyms for all senses

* Synonyms for all senses:

## Senses

### Sense 1.0:

#### Definition:

Used as a conjunction to denote a comparison;

#### Glosses:

as;  like (something); as if;

#### Explanation:

#### Citations:

[εἶδεν](../G37080/01.md) [τὸ](../G35880/01.md) [πνεῦμα](../G41510/01.md) [τοῦ](../G35880/01.md) [θεοῦ](../G23160/01.md) [καταβαῖνον](../G25970/01.md) ὡσεὶ [περιστερὰν](../G40580/01.md) [ἐρχόμενον](../G20640/01.md) [ἐπ’](../G19090/01.md) [αὐτόν](../G08460/01.md), "He saw the Spirit of God coming down like a dove and resting upon him", [Matt 3:16](mat 3:16); [Matt 9:36](mat 9:36); [26](mrk 9:26); [Luke 24:11](luk 24:11); [Acts 2:3](act 2:3); [Acts 6:15](act 6:15); [Rom 6:13](rom 6:13); [Heb 1:12](heb 1:12).

### Sense 2.0:

#### Definition:

Used to denote, adverbally, approximation with numbers and measures;

#### Glosses:

about;

#### Explanation:

#### Citations:

[οἱ](../G35880/01.md) [δὲ](../G11610/01.md) [ἐσθίοντες](../G20680/01.md) [ἦσαν](../G99999/01.md) [ἄνδρες](../G04350/01.md) ὡσεὶ [πεντακισχίλιοι](../G40000/01.md), "Those who ate were about five thousand men", [Matt 14:21](mat 14:21); [Luke 3:23](luk 3:23); [Luke 9:14](luk 9:14) {2 instances); [28](luk 9:28); [Luke 22:41](luk 22:41); [59](luk 22:59); [Luke 23:44](luk 23:44); [Acts 2:41](act 2:41); [Acts 10:3](act 10:3); [Acts 19:7](act 19:7).

repository: mandolyte/uw-obsidian

branch_name: refs/heads/main

repo_name: mandolyte/uw-obsidian


text:
# Systemic risk

Go back to the [[Economic_Glossary]]

The RISK of damage being done to the health of the FINANCIAL SYSTEM as a whole. A constant concern of BANK regulators is that the collapse of a single bank could bring down the entire financial system. This is why regulators often organise a rescue when a bank gets into financial difficulties. However, the expectation of such a rescue may create a MORAL HAZARD, encouraging banks to behave in ways that increase systemic risk. Another concern of regulators is that the RISK MANAGEMENT methods used by banks are so similar that they may increase systemic risk by creating a tendency for crowd behaviour. In particular, problems in one market may cause banks in general to liquidate positions in other markets, causing a vicious cycle of LIQUIDITY being withdrawn from the financial system as everybody rushes for the emergency exit at once. (See CAPITAL ASSET PRICING MODEL.)

path: /Economics/Glossary/Systemic Risk.Md

filename: Systemic Risk.Md


text:
# A Novel Automated Curriculum Strategy to Solve Hard Sokoban Planning Instances

```
@incollection{mctsgn_neurips20,
title = {A Novel Automated Curriculum Strategy to Solve Hard Sokoban Planning Instances},
author = {Dieqiao Feng and Carla P. Gomes and Bart Selman},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS) 33},
year = {2020},
publisher = {Curran Associates, Inc.}
}
```

links
- [neurips](https://nips.cc/Conferences/2020/ScheduleMultitrack?event=18341)


text:
[![Build Status](https://travis-ci.org/KarlWalsh/ComboClient.svg?branch=master)](https://travis-ci.org/KarlWalsh/ComboClient)

ComboClient
===========

##Example usage##
```java
public static void main(final String[] args) {
    final Combo combo = httpCombo(create("http://combo-squirrel.herokuapp.com"));

    combo.facts("chat", Map.class)
            .map(fact -> fact.get("who"))
            .filter(who -> who != null && !who.equals("chatterbot"))
            .forEach(who -> combo.publishFact("chat", chatMessage("Greetings %s, what a wonderful day!", who)));
}
```
The first line creates a combo instance that uses `org.springframework.web.client.RestTemplate` to communicate with the combo server located at the specified URI. *This instance of Combo is thread-safe.*

##Subscribing to topics##

```java
combo.subscribeTo("some.topic", Map.class)
```

This will create a `java.util.stream.Stream<T>` which produces facts as they arrive at the topic `some.topic` (in this example `<T>` is `java.lang.Map`). Gson is used to serilialise the json response from combo, so provide your desired type here. If you don't have one, then `java.lang.Map` may be a good choice.

The above example is mapping all chat facts to a String (the sender of the chat message) and filtering itself out of the incoming facts.

*The `java.util.stream.Stream.forEach` method will not return unless there is an exception. If you wish to subscribe to multiple topics, you will need to handle that yourself.*

##Publishing to topics##

```java
combo.publishFact("another.topic", fact)
```

This will serialise the fact to json (using Gson) and publish the json to the combo server

##Caveats##
Requires Java 8

gha_language: Java


text:
### Firehose to Redshift pipeline with Terraform

Code accompanying explanatory [blog post](https://medium.com/@aoc/kinesis-firehose-to-redshift-pipeline-with-terraform-2261b5afd29d).

### To run this:

First install terraform if you do not have it already.
Then run:
```
terraform init
terraform apply
yarn install
yarn build
yarn server
```

POST some data to `localhost:3000/data` in format `{ "name": "test_value", "value": 1.0 }`

### Destroying created resources
```
terraform destroy
```

This will fail to remove the S3 bucket if there were data created outside of the Terraform configs, so those will have to be deleted manually.

repository: aocenas/redshift-data-pipeline-demo

repo_name: aocenas/redshift-data-pipeline-demo


text:
# svg-mixed-loader

Webpack loader resolving svg into url string and multiple component formats.

## Background

In community svg files are commonly used to introduce icons to web apps, in this case developers are tend to import and svg as a component:

```jsx
import IconPlus from './icon-plus.svg';

const CreateButton = () => (
    <Button>
        <IconPlus />
        Create New
    </Button>
);
```

Also, svg, when itself is a common image format, may used as an image:

```jsx
import logoURL from './logo.svg';

const Header = () => (
    <div style={{display: 'flex'}}>
        <img width={180} height={30} src={logoURL} />
        <div style={{flex: 1}}>
            {/* navigation}*/}
        </div>
    </div>
);
```

As these use cases are conflicting to each other, this loader sovles this by export multiple possible formats from an svg file.

## Install

`@ecomfe/svg-mixed-loader` requires url-loader and file-loader as peer dependencies.

```shell
npm install -D @ecomfe/svg-mixed-loader url-loader file-loader
```

## Usage

```js
// webpack.config.js
module.exports = {
    module: {
        rules: [
            {
                test: /\.svg$/,
                use: [
                    {
                        loader: '@ecomfe/svg-mixed-loader',
                        options: {
                            // options
                        },
                    },
                    // You may add other preprocess loaders like svgo-loader or img-loader here
                ],
            },
        ],
    },
};
```

## Output

This loader currently create a default export of image URL, and a named export `ReactComponent` as a react component.

```js
import React from 'react';

export default 'data:image/svg;...';

export const ReactComponent = props => React.createElement(/* ... */);
ReactComponent.displayName = 'AutoResolvedComponentName';
```

## Options

```js
{
    // This option will be forwarded to url-loader, can set to false to disable generation of default export
    url: {
        limit: 1024,
    },
    // Can be:
    //
    // 1. true: generate react component as ReactComponent named export with default option
    // 2. false: don't generate react component
    // 3. object: customize react code generation, see below
    //
    // Default to false.
    react: {
        // Can be multiple formats:
        //
        // 1. true: auto resolve a component name from resource path
        // 2. false: remove display name from output
        // 3. string: a static display name
        // 4. function: customize display by (resourcePath) => string
        //
        // Default to true.
        displayName: true,
    }
}
```

By default `@ecomfe/svg-mixed-loader` generates only default export, behaves exactly the same as `url-loader`,
if `ReactComponent` named export is wanted, pass `{react: true}` as options is a quick solution.

repository: ecomfe/svg-mixed-loader

repo_name: ecomfe/svg-mixed-loader


text:
---
title: "Erlang"
date: 2018-11-17T08:00:00-00:00
draft: false
class: "resized-logo"
logo: /images/erlang-opencensus.png
---

In this quickstart, using OpenCensus Erlang, you will gain hands-on experience with:
{{% children %}}

For full API references, please take a look at:

Resource|Link
---|---
HexDoc|https://hexdocs.pm/opencensus/
Github repository|https://github.com/census-instrumentation/opencensus-erlang


text:
---
title: "Production-ready object detection model training workflow with HPE
  Machine Learning Development Environment "
date: 2023-06-16T16:00:00.000Z
featuredBlog: true
priority: 2
author: Andrew Mendez
authorimage: /img/andrew.jpeg
disable: false
tags:
  - hpe-machine-learning-development-environment
  - machine-learning
  - data-ml-engineer
---
This in-depth blog tutorial is divided into five separate sections, where I will recount the seamless user experience one has when working with [HPE Machine Learning Development Environment](https://www.hpe.com/us/en/solutions/artificial-intelligence/machine-learning-development-environment.html), pointing out how easy it is to achieve machine learning at scale with HPE.

Throughout the different parts of this tutorial, I will review the end-to-end training of an object detection model using NVIDIA’s PyTorch Container from [NVIDIA's NGC Catalog](https://www.nvidia.com/en-us/gpu-cloud/), a Jupyter Notebook, the open-source training platform from [Determined AI](http://www.determined.ai/), and [Kserve](https://www.kubeflow.org/docs/external-add-ons/kserve/kserve/) to deploy the model into production.

# Part 1: End-to-end example training object detection model using NVIDIA PyTorch Container from NGC

## [](https://github.com/interactivetech/e2e_blogposts/blob/main/ngc_blog/E2E-Part-1-Installation.md#installation)Installation

- - -

***Note: this object detection demo is based on the [TorchVision package GitHub repository](https://github.com/pytorch/vision/tree/v0.11.3).***

This notebook walks you each step to train a model using containers from the NGC Catalog. We chose the GPU optimized PyTorch container as an example. The basics of working with docker containers apply to all NGC containers.

## NGC

The [NGC catalog from NVIDIA](https://www.nvidia.com/en-us/gpu-cloud/) offers ready-to-use containers, pre-trained models, SDKs, and Helm charts for diverse use cases and industries to speed up model training, development, and deployment. For this example, I'm pulling the popular PyTorch container from NGC and show you how to:

* Install the Docker engine on your system
* Pull a PyTorch container from the NGC catalog using Docker
* Run the PyTorch container using Docker

Let's get started!

- - -

### [](https://github.com/interactivetech/e2e_blogposts/blob/main/ngc_blog/E2E-Part-1-Installation.md#1-install-the-docker-engine)1. Install the Docker Engine

Go to the [Docker Installation Engine documentation](https://docs.docker.com/engine/install/) to install the Docker Engine on your system.

### [](https://github.com/interactivetech/e2e_blogposts/blob/main/ngc_blog/E2E-Part-1-Installation.md#2-download-the-tensorflow-container-from-the-ngc-catalog)2. Download the TensorFlow container from the NGC Catalog

Once the Docker Engine is installed on your machine, visit the [NVIDIA NGC Container Catalog](https://catalog.ngc.nvidia.com/containers) and search for[](https://ngc.nvidia.com/catalog/containers)and search for the TensorFlow container. Click on the TensorFlow card and copy the pull command. [![](https://raw.githubusercontent.com/kbojo/images/master/NGC.png)](https://raw.githubusercontent.com/kbojo/images/master/NGC.png)

Open the command line of your machine and paste the pull command into your command line. Execute the command to download the container.

`$ docker pull nvcr.io/nvidia/pytorch:21.11-py3`

The container starts downloading to your computer. A container image consists of many layers; all of them need to be pulled.

### [](https://github.com/interactivetech/e2e_blogposts/blob/main/ngc_blog/E2E-Part-1-Installation.md#3-run-the-tensorflow-container-image)3. Run the TensorFlow container image

Once the container download is completed, run the following code in your command line to run and start the container:

`$ docker run -it --gpus all -p 8888:8888 -v $PWD:/projects --network=host nvcr.io/nvidia/pytorch:21.11-py3`

[![](https://raw.githubusercontent.com/kbojo/images/master/commandline1.png)](https://raw.githubusercontent.com/kbojo/images/master/commandline1.png)

### [](https://github.com/interactivetech/e2e_blogposts/blob/main/ngc_blog/E2E-Part-1-Installation.md#4-install-jupyter-lab-and-open-a-notebook)4. Install Jupyter lab and open a notebook

Within the container, run the following commands:

`pip install torchvision==0.11.3 jupyterlab`

`jupyter lab --ip=<IP_ADDRESS> --port=8888 --allow-root`

Open up your favorite browser and enter: [http://localhost:8888/?token=*yourtoken](http://localhost:8888/?token=*yourtoken)*. [![](https://raw.githubusercontent.com/kbojo/images/master/commandline2.png)](https://raw.githubusercontent.com/kbojo/images/master/commandline2.png)

You should see the Jupyter Lab application. Click on the plus icon to launch a new Python 3 notebook. Follow the instructions regarding the image classification with the TensorFlow example provided in Part 2.

Now that you have your Docker engine installed and the PyTorch Container running, we need to fetch and prepare our training dataset. You'll see that coming up in Part 2 below.

- - -

# Part 2: Data preparation

*Note this Demo is based on NGC Docker image* `nvcr.io/nvidia/pytorch:21.11-py3`

This notebook walks you through each step required to train a model using containers from the NGC catalog. We chose the GPU optimized PyTorch container as an example. The basics of working with docker containers apply to all NGC containers.

Here, I will show you how to:

* Download the xView Dataset
* How to convert labels to coco format
* How to conduct the preprocessing step, **Tiling**: slicing large satellite imagery into chunks
* How to upload to S3 bucket to support distributed training

Let's get started!

- - -

## Pre-reqs, set up Jupyter Notebook environment using NGC container

### Execute Docker run to create NGC environment for data preparation

Make sure to map host directory to Docker directory. You will use the host directory again to do the following:

* `docker run   --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 -v /home/ubuntu:/home/ubuntu  -p 8008:8888 -it nvcr.io/nvidia/pytorch:21.11-py3  /bin/bash`

### Run Jupyter Notebook command within Docker container to access it on your local browser

* `cd /home/ubuntu`
* `jupyter lab --ip=<IP_ADDRESS> --port=8888 --NotebookApp.token='' --NotebookApp.password=''`
* `git clone https://github.com/interactivetech/e2e_blogposts.git`

### Download the xView dataset

The dataset you will be using is from the [DIUx xView 2018 Challenge](https://challenge.xviewdataset.org) by U.S. National Geospatial-Intelligence Agency (NGA). You will need to [create an account](https://challenge.xviewdataset.org/welcome), agree to the terms and conditions, and download the dataset manually.

You can also [download the dataset](https://challenge.xviewdataset.org/data-download).

```python
# run pip install to get the SAHI library
!pip install sahi scikit-image opencv-python-headless==<IP_ADDRESS>
```

```python
# Example command to download train images with wget command, you will need to update the url as the token is expired"
!wget -O train_images.tgz "https://d307kc0mrhucc3.cloudfront.net/train_images.tgz?Expires=1680923794&Signature=pn0R9k3BpSukGEdjcNx7Kvs363HWkngK8sQLHxkDOqqkDAHSOCDBmAMAsBhYZ820uMpyu4Ynp1UAV60OmUURyvGorfIRaVF~jJO8-oqRVLeO1f24OGCQg7HratHNUsaf6owCb8XXy~3zaW15FcuORuPV-2Hr6Jxekwcdw9D~g4M2dLufA~qBfTLh3uNjWK5UCAMvyPz2SRLtvc3JLzGYq1eXiKh1dI9W0DyWXov3mVDpBdwS84Q21S2lVi24KJsiZOSJqozuvahydW2AuR~tbXTRbYtmAyPF9ZqT8ZCd9MLeKw2qQJjb7tvzaSZ0F9zPjm2RS8961bo6QoBVeo6kzA__&Key-Pair-Id=APKAIKGDJB5C3XUL2DXQ"
```

```python
# Example command to download train images with wget command, you will need to update the url as the token is expired"
!wget -O train_labels.tgz "https://d307kc0mrhucc3.cloudfront.net/train_labels.tgz?Expires=1680923794&Signature=YEX~4gioZ7J0pAjEPx7BjJfnOa2j412mx2HlStlqa0cHj-T0T21vo17S8Fs71DXgPlZ5qnIre2-icc7wQ~EuQV-HL1ViS8qH1Aubgj9i0pnHZL07ktiyulX7QStOLywxJ7bOOmQ37iFF~-OcJW3MZfQCTWrP~LdlZMmXz0yGs5WEIYeMyvfUfIhGvrpHcJ14Z3czasSMeOKfwdQsUJoRcFTbmlbZk98IVeEWjmnGTfxGbPBdMmQ96XdT4NohggtzGdqeZhGNfwm7dKGSUbXvGCoFe~fIjBz0~5BvB6rNIaMaFuBA6aGTbCLeG8FlvijcECouhZdMTHmQUlgtSlZjGw__&Key-Pair-Id=APKAIKGDJB5C3XUL2DXQ"
```

```python
# unzip images and labels from /home/ubuntu/e2e_blogposts/ngc_blog
!tar -xf train_images.tgz -C xview_dataset/
```

```python
# unzip labels from /home/ubuntu/e2e_blogposts/ngc_blog directory
!tar -xf train_labels.tgz -C xview_dataset/
```

## 1. Convert TIF to RGB

```python
# Here loop through all the images and convert them to RGB, this is important for tiling the images and training with pytorch
# will take about an hour to complete
!python data_utils/tif_2_rgb.py --input_dir xview_dataset/train_images \
  --out_dir xview_dataset/train_images_rgb/
```

## 2. How to convert labels to COCO format

Run a script to convert the dataset labels from .geojson format to COCO format. [Read more details about the COCO format at this link.](https://www.immersivelimit.com/tutorials/create-coco-annotations-from-scratch)

The result will be two files (in COCO formal) generated `train.json` and `val.json`

```python
# make sure train_images_dir is pointing to the .tif images
!python data_utils/convert_geojson_to_coco.py --train_images_dir xview_dataset/train_images/ \
  --train_images_dir_rgb xview_dataset/train_images_rgb/ \
  --train_geojson_path xview_dataset/xView_train.geojson \
  --output_dir xview_dataset/ \
  --train_split_rate 0.75 \
  --category_id_remapping data_utils/category_id_mapping.json \
  --xview_class_labels data_utils/xview_class_labels.txt
```

## 3. Slicing/Tiling the dataset

Here, you will be using the SAHI library to slice our large satellite images. Satellite images can be up to 50k^2 pixels in size, which wouldn't fit in GPU memory. You can alleviate this problem by slicing the image.

```python
!python data_utils/slice_coco.py --image_dir xview_dataset/train_images_rgb/ \
  --train_dataset_json_path xview_dataset/train.json \
  --val_dataset_json_path xview_dataset/val.json \
  --slice_size 300 \
  --overlap_ratio 0.2 \
  --ignore_negative_samples True \
  --min_area_ratio 0.1 \
  --output_train_dir xview_dataset/train_images_rgb_no_neg/ \
  --output_val_dir xview_dataset/val_images_rgb_no_neg/
```

## 4. Upload to S3 bucket to support distributed training

Now, you can upload your exported data to a publicly accessible AWS S3 bucket. For a large-scale distributed experiment, this will enable you to access the dataset without installing the dataset on the device.
View [Determined Documentation](https://docs.determined.ai/latest/model-dev-guide/load-model-data.html) and [AWS instructions](https://codingsight.com/upload-files-to-aws-s3-with-the-aws-cli/) to learn how to upload your dataset to an S3 bucket. Review the `S3Backend` class in `data.py`

Once you create an S3 bucket that is publicly accessible, here are example commands to upload the preprocessed dataset to S3:

* `aws s3 cp --recursive xview_dataset/train_sliced_no_neg/   s3://determined-ai-xview-coco-dataset/train_sliced_no_neg`
* `aws s3 cp --recursive xview_dataset/val_sliced_no_neg/   s3://determined-ai-xview-coco-dataset/val_sliced_no_neg`

Now that the satellite imagery data is in an S3 bucket and is prepped for distributed training, you can progress to model training and inferencing via the NGC container.

# Part 3: End-to-End example training object detection model using NVIDIA PyTorch container from NGC

## Training and inference via NGC Container

This notebook walks you through each step to train a model using containers from the NGC Catalog. I chose the GPU-optimized PyTorch container for this example. The basics of working with Docker containers apply to all NGC containers.

We will show you how to:

* Execute training an object detection model on satellite imagery using TensorFlow and Jupyter Notebook
* Run inference on a trained object detection model using the SAHI library

Note this object detection demo is based on [this PyTorch repo](https://github.com/pytorch/vision/tree/v0.11.3) and ngc docker image `nvcr.io/nvidia/pytorch:21.11-py3`

It is assumed that, by now, you have completed step 2 of dataset preprocessing and have your tiled satellite imagery dataset completed and in the local directory `train_images_rgb_no_neg/train_images_300_02`

Let's get started!

## Execute Docker run to create NGC environment for data prep

Make sure to map host directory to Docker directory. You will use the host directory again to:

* `docker run   --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 -v /home/ubuntu:/home/ubuntu  -p 8008:8888 -it nvcr.io/nvidia/pytorch:21.11-py3  /bin/bash`

## Run Jupyter Notebook command within Docker container to access it on your local browser

* `cd /home/ubuntu`
* `jupyter lab --ip=<IP_ADDRESS> --port=8888 --NotebookApp.token='' --NotebookApp.password=''`

```python
%%capture
!pip install cython pycocotools matplotlib terminaltables
```

## TLDR; Run training job on 4 GPUs

The below cell will run a multi-gpu training job. This job will train an object detection model (faster-rcnn) on a dataset of satellite imagery images that contain 61 classes of objects.

* Change `nproc_per_node` argument to specify the number of GPUs available on your server

```python
!torchrun --nproc_per_node=4 detection/train.py\
    --dataset coco --data-path=xview_dataset/ --model fasterrcnn_resnet50_fpn --epochs 26\
    --lr-steps 16 22 --aspect-ratio-group-factor 3
```

### 1. Object detection on satellite imagery with PyTorch (single GPU)

Follow and run the code to train a Faster RCNN FPN (Resnet50 backbone) that classifies images of clothing.

```python
import sys
sys.path.insert(0,'detection')
```

```python
# Import python dependencies
import datetime
import os
import time

import torch
import torch.utils.data
import torchvision
import torchvision.models.detection
import torchvision.models.detection.mask_rcnn

from coco_utils import get_coco, get_coco_kp

from group_by_aspect_ratio import GroupedBatchSampler, create_aspect_ratio_groups
from engine import train_one_epoch, evaluate

import presets
import utils
from coco_utils import get_coco, get_coco_kp
from train import get_dataset, get_transform
from group_by_aspect_ratio import GroupedBatchSampler, create_aspect_ratio_groups
from engine import train_one_epoch, evaluate
from models import build_frcnn_model
from PIL import Image
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor
from collections import OrderedDict
from tqdm import tqdm
from vis_utils import load_determined_state_dict, visualize_pred, visualize_gt, predict
import numpy as np

import matplotlib.pyplot as plt
```

```python
output_dir='output'
data_path='xview_dataset/'
dataset_name='coco'
model_name='fasterrcnn_resnet50_fpn'
device='cpu'
batch_size=8
epochs=26
workers=4
lr=0.02
momentum=0.9
weight_decay=1e-4
lr_scheduler='multisteplr'
lr_step_size=8
lr_steps=[16, 22]
lr_gamma=0.1
print_freq=20
resume=False
start_epoch=0
aspect_ratio_group_factor=3
rpn_score_thresh=None
trainable_backbone_layers=None
data_augmentation='hflip'
pretrained=True
test_only=False
sync_bn=False
```

```python
# Import the dataset.
# Data loading code
print("Loading data")

dataset, num_classes = get_dataset(dataset_name, "train", get_transform(True, data_augmentation),
                                   data_path)
dataset_test, _ = get_dataset(dataset_name, "val", get_transform(False, data_augmentation), data_path)
print(dataset.num_classes)
print("Creating data loaders")
train_sampler = torch.utils.data.RandomSampler(dataset)
test_sampler = torch.utils.data.SequentialSampler(dataset_test)
group_ids = create_aspect_ratio_groups(dataset, k=aspect_ratio_group_factor)
train_batch_sampler = GroupedBatchSampler(train_sampler, group_ids, batch_size)
train_batch_sampler = torch.utils.data.BatchSampler(
            train_sampler, batch_size, drop_last=True)

data_loader = torch.utils.data.DataLoader(
    dataset, batch_sampler=train_batch_sampler, num_workers=workers,
    collate_fn=utils.collate_fn)

data_loader_test = torch.utils.data.DataLoader(
    dataset_test, batch_size=1,
    sampler=test_sampler, num_workers=0,
    collate_fn=utils.collate_fn)
```

```python
# Getting three examples from the test dataset
inds_that_have_boxes = []
test_images = list(data_loader_test)
for ind,(im,targets) in tqdm(enumerate(test_images),total=len(list(data_loader_test))):
    # print(ind,targets)
    if targets[0]['boxes'].shape[0]>0:
        # print(targets[0]['boxes'].shape[0])
        # print(ind,targets)
        inds_that_have_boxes.append(ind)

images_t_list=[]
targets_t_list=[]
for ind in tqdm(range(3)):
    im,targets = test_images[ind]
    images_t_list.append(im[0])
    targets_t_list.append(targets[0])
```

```python
# Let's have a look at one of the images. The following code visualizes the images using the matplotlib library.
im = Image.fromarray((255.*images_t_list[0].cpu().permute((1,2,0)).numpy()).astype(np.uint8))
plt.imshow(im)
plt.show()
```

```python
# Let's look again at the first three images, but this time with the class names.

for i,t in zip(images_t_list,targets_t_list):
    im = Image.fromarray((255.*i.cpu().permute((1,2,0)).numpy()).astype(np.uint8))
    plt.imshow(im)
    plt.show()
    im = visualize_gt(i,t)
    plt.imshow(im)
    plt.show()
```

```python
# Let's build the model:
print("Creating model")
print("Number of classes: ",dataset.num_classes)
model = build_frcnn_model(num_classes=dataset.num_classes)
_=model.to('cpu')
```

```python
# Compile the model:
# Define loss function, optimizer, and metrics.
params = [p for p in model.parameters() if p.requires_grad]
optimizer = torch.optim.SGD(
    params, lr=lr, momentum=momentum, weight_decay=weight_decay)

lr_scheduler = lr_scheduler.lower()
if lr_scheduler == 'multisteplr':
    lr_scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer,
                                                        milestones=lr_steps,
                                                        gamma=lr_gamma)
elif lr_scheduler == 'cosineannealinglr':
    lr_scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=epochs)
else:
    raise RuntimeError("Invalid lr scheduler '{}'. Only MultiStepLR and CosineAnnealingLR "
                       "are supported.".format(args.lr_scheduler))
```

```python
# Train the model:
# Let's train 1 epoch. After every epoch, training time, loss, and accuracy will be displayed.
print("Start training")
start_time = time.time()
for epoch in range(start_epoch, epochs):
    train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq)
    lr_scheduler.step()
    if output_dir:
        checkpoint = {
            'model': model.state_dict(),
            'optimizer': optimizer.state_dict(),
            'lr_scheduler': lr_scheduler.state_dict(),
            'args': args,
            'epoch': epoch
        }
        utils.save_on_master(
            checkpoint,
            os.path.join(output_dir, 'model_{}.pth'.format(epoch)))
        utils.save_on_master(
            checkpoint,
            os.path.join(output_dir, 'checkpoint.pth'))

    # evaluate after every epoch
    evaluate(model, data_loader_test, device=device)

total_time = time.time() - start_time
total_time_str = str(datetime.timedelta(seconds=int(total_time)))
print('Training time {}'.format(total_time_str))
```

```python
def build_frcnn_model2(num_classes):
    print("Loading pretrained model...")
    # load an detection model pre-trained on COCO
    model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)

    # get the number of input features for the classifier
    in_features = model.roi_heads.box_predictor.cls_score.in_features
    # replace the pre-trained head with a new one
    model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
    model.min_size=800
    model.max_size=1333
    # RPN parameters
    model.rpn_pre_nms_top_n_train=2000
    model.rpn_pre_nms_top_n_test=1000
    model.rpn_post_nms_top_n_train=2000
    model.rpn_post_nms_top_n_test=1000
    model.rpn_nms_thresh=1.0
    model.rpn_fg_iou_thresh=0.7
    model.rpn_bg_iou_thresh=0.3
    model.rpn_batch_size_per_image=256
    model.rpn_positive_fraction=0.5
    model.rpn_score_thresh=0.05
    # Box parameters
    model.box_score_thresh=0.0
    model.box_nms_thresh=1.0
    model.box_detections_per_img=500
    model.box_fg_iou_thresh=1.0
    model.box_bg_iou_thresh=1.0
    model.box_batch_size_per_image=512
    model.box_positive_fraction=0.25
    return model
```

```python
# Let's see how the model performs on the test data:
model = build_frcnn_model2(num_classes=61)
ckpt = torch.load('model_8.pth',map_location='cpu')
model.load_state_dict(ckpt['model'])
_=model.eval()
```

```python
_=predict(model,images_t_list,targets_t_list)
```

In the next part of this blog post, I will show you how to scale your model training using using distributed training within HPE Machine Learning Development Environment & System.

# Part 4: Training on HPE Machine Learning Development & System

[HPE Machine Learning Development Environment](https://www.hpe.com/us/en/solutions/artificial-intelligence/machine-learning-development-environment.html) is a training platform software that reduces complexity for ML researchers and helps research teams collaborate. HPE combines this incredibly powerful training platform with best-of-breed hardware and interconnect in [HPE Machine Learning Development System](https://www.hpe.com/us/en/hpe-machine-learning-development-system.html), an AI turnkey solution that will be used for the duration of the tutorial.

This notebook walks you through the commands to run the same training you did in stepin Step 3, but using the HPE Machine Learning Development Environment together with the PyTorchTrial API.
All the code is configured to run out of the box. The main change is defining a `class ObjectDetectionTrial(PyTorchTrial)` to incorporate the model, optimizer, dataset, and other training loop essentials.
You can view implementation details by looking at `determined_files/model_def.py`

Here, I will show you how to:

* Run a distributed training experiment
* Run a distributed hyperparameter search

*Note: This notebook was tested on a deployed HPE Machine Learning Development System cluster, running HPE Machine Learning Development Environment (0.21.2-dev0).*

Let's get started!

- - -

## Pre-req: Run startup-hook.sh

This script will install some python dependencies, and install dataset labels needed when loading the xView dataset:

```bash
## Temporary disable for Grenoble Demo
wget "https://determined-ai-xview-coco-dataset.s3.us-west-2.amazonaws.com/train_sliced_no_neg/train_300_02.json"
mkdir /tmp/train_sliced_no_neg/
mv train_300_02.json /tmp/train_sliced_no_neg/train_300_02.json
wget "https://determined-ai-xview-coco-dataset.s3.us-west-2.amazonaws.com/val_sliced_no_neg/val_300_02.json"
mkdir /tmp/val_sliced_no_neg
mv val_300_02.json /tmp/val_sliced_no_neg/val_300_02.json
```

*Note that completing this tutorial requires you to upload your dataset from Step 2 into a publicly accessible S3 bucket. This will enable for a large scale distributed experiment to have access to the dataset without installing the dataset on device. View [Determined Documentation](https://docs.determined.ai/latest/model-dev-guide/load-model-data.html) and [AWS instructions](https://codingsight.com/upload-files-to-aws-s3-with-the-aws-cli/) to learn how to upload your dataset to an S3 bucket. Review the* `S3Backend` class in `data.py`

When you define your S3 bucket and uploaded your dataset, make sure to change the `TARIN_DATA_DIR` in `build_training_data_loader` with the defined path in the S3 bucket.

```python
def build_training_data_loader(self) -> DataLoader:
    # CHANGE TRAIN_DATA_DIR with different path on S3 bucket
    TRAIN_DATA_DIR='determined-ai-xview-coco-dataset/train_sliced_no_neg/train_images_300_02/'

    dataset, num_classes = build_xview_dataset(image_set='train',args=AttrDict({
                                            'data_dir':TRAIN_DATA_DIR,
                                            'backend':'aws',
                                            'masks': None,
                                            }))
    print("--num_classes: ",num_classes)

    train_sampler = torch.utils.data.RandomSampler(dataset)

    data_loader = DataLoader(
                             dataset,
                             batch_sampler=None,
                             shuffle=True,
                             num_workers=self.hparams.num_workers,
                             collate_fn=unwrap_collate_fn)
    print("NUMBER OF BATCHES IN COCO: ",len(data_loader))# 59143, 7392 for mini coco
```

```python
!bash demos/xview-torchvision-coco/startup-hook.sh
```

## Define environment variable DET_MASTER and login in terminal

Run the below commands in a terminal, and complete logging into the Determined cluster by changing <username> to your username.

* `export DET_MASTER=<IP_ADDRESS>`
* `det user login <username>`

## Define Determined experiment

In [Determined](https://www.determined.ai/), a *trial* is a training task that consists of a dataset, a deep learning model, and values for all of the model’s hyperparameters. An *experiment* is a collection of one or more trials: an experiment can either train a single model (with a single trial), or can train multiple models via a hyperparameter sweep a user-defined hyperparameter space.

Here is what a configuration file looks like for a distributed training experiment.

Below is what the `determined_files/const-distributed.yaml` contents look like. `slots_per_trial: 8` defines that we will use 8 GPUs for this experiment.

Edit the change the workspace and project settings in the `determined_files/const-distributed.yaml` file

```yaml
name: resnet_fpn_frcnn_xview_dist_warmup
workspace: <WORKSPACE_NAME>
project: <PROJECT_NAME>
profiling:
 enabled: true
 begin_on_batch: 0
 end_after_batch: null
hyperparameters:
    lr: 0.01
    momentum: 0.9
    global_batch_size: 128
    # global_batch_size: 16
    weight_decay: 1.0e-4
    gamma: 0.1
    warmup: linear
    warmup_iters: 200
    warmup_ratio: 0.001

    step1: 18032 # 14 epochs: 14*1288 == 18,032
    step2: 19320 # 15 epochs: 15*1288 == 19,320
    model: fasterrcnn_resnet50_fpn
    # Dataset
    dataset_file: coco
    backend: aws # specifiy the backend you want to use.  one of: gcs, aws, fake, local
    data_dir: determined-ai-coco-dataset # bucket name if using gcs or aws, otherwise directory to dataset
    masks: false
    num_workers: 4
    device: cuda
environment:
    image: determinedai/environments:cuda-11.3-pytorch-1.10-tf-2.8-gpu-mpi-0.19.10
    environment_variables:
        - NCCL_DEBUG=INFO
        # You may need to modify this to match your network configuration.
        - NCCL_SOCKET_IFNAME=ens,eth,ib
bind_mounts:
    - host_path: /tmp
      container_path: /data
      read_only: false
scheduling_unit: 400
min_validation_period:
    batches: 1288 # For training

searcher:
  name: single
  metric: mAP
  smaller_is_better: true
  max_length:
    batches: 38640 # 30*1288 == 6440# Real Training
records_per_epoch: 1288
resources:
    slots_per_trial: 8
    shm_size:<PHONE_NUMBER>
max_restarts: 0

entrypoint: python3 -m determined.launch.torch_distributed --trial model_def:ObjectDetectionTrial
```

```python
# Run the below cell to kick off an experiment
!det e create determined_files/const-distributed.yaml determined_files/
```

```
Preparing files to send to master... 237.5KB and 36 files
Created experiment 77
```

## Launching a distributed hyperparameter search experiment

To implement an automatic hyperparameter tuning experiment, define the hyperparameter space, e.g. by listing the decisions that may impact model performance. You can specify a range of possible values in the experiment configuration for each hyperparameter in the search space.

View the `x.yaml` file that defines a hyperparameter search where the model architecture that achieves the best performance on the dataset is found..

```yaml
name: xview_frxnn_search
workspace: Andrew
project: Xview FasterRCNN
profiling:
 enabled: true
 begin_on_batch: 0
 end_after_batch: null
hyperparameters:
    lr: 0.01
    momentum: 0.9
    global_batch_size: 128
    weight_decay: 1.0e-4
    gamma: 0.1
    warmup: linear
    warmup_iters: 200
    warmup_ratio: 0.001
    step1: 18032 # 14 epochs: 14*1288 == 18,032
    step2: 19320 # 15 epochs: 15*1288 == 19,320
    model:
      type: categorical
      vals: ['fasterrcnn_resnet50_fpn','fcos_resnet50_fpn', 'ssd300_vgg16','ssdlite320_mobilenet_v3_large','resnet152_fasterrcnn_model','efficientnet_b4_fasterrcnn_model','convnext_large_fasterrcnn_model','convnext_small_fasterrcnn_model']

    # Dataset
    dataset_file: coco
    backend: aws # specifiy the backend you want to use.  one of: gcs, aws, fake, local
    data_dir: determined-ai-coco-dataset # bucket name if using gcs or aws, otherwise directory to dataset
    masks: false
    num_workers: 4

    device: cuda
environment:
    environment_variables:
        - NCCL_DEBUG=INFO
        # You may need to modify this to match your network configuration.
        - NCCL_SOCKET_IFNAME=ens,eth,ib
bind_mounts:
    - host_path: /tmp
      container_path: /data
      read_only: false
scheduling_unit: 400
# scheduling_unit: 40
min_validation_period:
    batches: 1288 # For Real training
searcher:
  name: grid
  metric: mAP
  smaller_is_better: false
  max_length:
    batches: 51520 # 50*1288 == 51520# Real Training

records_per_epoch: 1288
resources:
    # slots_per_trial: 16
    slots_per_trial: 8
    shm_size:<PHONE_NUMBER>
max_restarts: 0

entrypoint: python3 -m determined.launch.torch_distributed --trial model_def:ObjectDetectionTrial
```

```python
# Run the below cell to run a hyperparameter search experiment
!det e create determined_files/const-distributed-search.yaml determined_files/
```

```
Preparing files to send to master... 312.2KB and 40 files
Created experiment 79
```

## Load checkpoint of trained experiment

Replace the `<EXP_ID>` and run the below cells with the experiment ID once the experiment is completed.

```python
from determined_files.utils.model import build_frcnn_model
from utils import load_model_from_checkpoint
from determined.experimental import Determined,client
```

```python
experiment_id = 76
MODEL_NAME = "xview-fasterrcnn"
checkpoint = client.get_experiment(experiment_id).top_checkpoint(sort_by="mAP", smaller_is_better=False)
print(checkpoint.uuid)
loaded_model = load_model_from_checkpoint(checkpoint)
```

Now that you have a checkpoint from the trained object detection model, you can deploy it to Kserve to run inference and predictions.

# Part 5: Deploying trained model on Kserve

This notebook walks you each step to deploy a custom object detection model on KServe.

Here, I will show you how to:

* Install Kserve natively using Kind and Knative
* Create a Persistent Volume Claim for local model deployment
* Preparing custom model for Kserve inference
* Deploying model using a KServe InferenceService
* Complete a sample request and plot predictions

*Note: This notebook was tested on a Linux-based machine with Nvidia T4 GPUs. We also assume Docker is installed in your Linux system/environment*

Let's get started!

## Pre-reqs: Setting up Python and Jupyter Lab environment

Run the below commands to set up a Python virtual environment, and install all the Python packages needed for this tutorial

```cwl
sudo apt-get update && sudo apt-get  install python3.8-venv
python3 -m venv kserve_env
source kserve_env/bin/activate
pip install kserve jupyterlab torch-model-archiver
pip install torch==1.11.0 torchvision==0.12.0 matplotlib
jupyter lab --ip=<IP_ADDRESS> \
  --port=8008 \
  --NotebookApp.token='' \
  --NotebookApp.password=''
```

## Install Kserve natively using Kind and Knative

### Install Kind

Open a terminal and run the following bash commands to install a Kubernetes cluster using Kind:

* `curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.18.0/kind-linux-amd64`
* `chmod +x ./kind`
* `sudo mv ./kind /usr/local/bin/kind`

After running these commands, create a cluster by running the command: `kind create cluster`

### Install Kubectl

Run the following bash commmands in a terminal to to install the kubectl runtime:

* `curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"`
* `curl -LO "https://dl.k8s.io/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl.sha256"`
* `sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl`

### Install Kserve

Run this bash script to install KServe onto our default Kubernetes cluster, note this will install the following artifacts:

* ISTIO_VERSION=1.15.2, KNATIVE_VERSION=knative-v1.9.0, KSERVE_VERSION=v0.9.0-rc0, CERT_MANAGER_VERSION=v1.3.0
* `bash e2e_blogposts/ngc_blog/kserve_utils/bash_scripts/kserve_install.sh`

### Patch domain for local connection to KServe cluster/environment

Run this command to patch your cluster when you want to connect to your cluster on the same machine:

`kubectl patch cm config-domain --patch '{"data":{"example.com":""}}' -n knative-serving`

### Run port forwarding to access KServe cluster

* `INGRESS_GATEWAY_SERVICE=$(kubectl get svc --namespace istio-system --selector="app=istio-ingressgateway" --output jsonpath='{.items[0].metadata.name}')`
* `kubectl port-forward --namespace istio-system svc/${INGRESS_GATEWAY_SERVICE} 8080:80`

Make sure to open a new terminal to continue the configuration.

### Create a persistent volume claim for local model deployment

You will be creating a persistent volume claim to host and access the PyTorch-based object detection model locally. A persistent volume claim requires three k8s artifacts:

* A persistent volume
* A persistent volume claim
* A k8s pod that connects the PVC to be accessed by other k8s resources

### Creating a persistent volume and persistent volume claim

Below is the yaml definition that defines the Persistent Volume (PV) and a PersistentVolumeClaim (PVC). We already created a file that defines this PV in `k8s_files/pv-and-pvc.yaml`

```yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: task-pv-volume
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/home/ubuntu/mnt/data"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: task-pv-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
```

To create the PV and PVC, run the command: `kubectl apply -f k8s_files/pv-and-pvc.yaml`

### Create k8s pod to access PVC

Below is the yaml definition that defines the k8s Pod that mounts the PersistentVolumeClaim (PVC). We already created a file that defines this PV in `k8s_files/model-store-pod.yaml`

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: model-store-pod
spec:
  volumes:
    - name: model-store
      persistentVolumeClaim:
        claimName: task-pv-claim
  containers:
    - name: model-store
      image: ubuntu
      command: [ "sleep" ]
      args: [ "infinity" ]
      volumeMounts:
        - mountPath: "/pv"
          name: model-store
      resources:
        limits:
          memory: "1Gi"
          cpu: "1"
```

To create the Pod, run the command: `kubectl apply -f k8s_files/model-store-pod.yaml`

## Preparing custom model for Kserve inference

Here we will complete some preparation steps to deploy a trained custom FasterRCNN Object Detection model using KServe. A pre-requisite is to download a checkpoint from a determined experiement. You can read this [tutorial](https://docs.determined.ai/latest/training/model-management/checkpoints.html#downloading-checkpoints-using-the-cli) on how to download a checkpoint using the Determined CLI. For this tutorial, you can download an already prepared checkpoint using the following bash command:

* `wget -O kserve_utils/torchserve_utils/trained_model.pth https://determined-ai-xview-coco-dataset.s3.us-west-2.amazonaws.com/trained_model.pth`

### Stripping the checkpoint of the optimizer state dictionary

Checkpoints created from a Determined experiment will save both the model parameters and the optimizer parameters. You will need to strip the checkpoint of all parameters except the model parameters for inference. Run the bash command to generate `train_model_stripped.pth`:

Run the below command in a terminal:

```cwl
python kserve_utils/torchserve_utils/strip_checkpoint.py --ckpt-path kserve_utils/torchserve_utils/trained_model.pth \
  --new-ckpt-name kserve_utils/torchserve_utils/trained_model_stripped.pth
```

### Run TorchServe Export to create .mar file

Run the below command to export the PyTorch checkpoint into a .mar file that is required for torchserve inference. The Kserve InferenceService will automatically deploy a Pod with a docker image that support TorchServe inferencing.

```cwl
torch-model-archiver --model-name xview-fasterrcnn \
  --version 1.0 \
  --model-file kserve_utils/torchserve_utils/model-xview.py \
  --serialized-file kserve_utils/torchserve_utils/trained_model_stripped.pth \
  --handler kserve_utils/torchserve_utils/fasterrcnn_handler.py \
  --extra-files kserve_utils/torchserve_utils/index_to_name.json
```

After command finishes, run the command to move the file to our prepared `model-store/` directory:

* `cp xview-fasterrcnn.mar kserve_utils/torchserve_utils/model-store -v`

### Copy `config/` and `model-store/` folders to the K8S PVC Pod

This is the directory structure needed to prepare your custom PyTorch model for KServe inferencing:

```markdown
├── config
│   └── config.properties
├── model-store
│   ├── properties.json
│   └── xview-fasterrcnn.mar
```

#### What the config.properties file looks like

```markdown
inference_address=http://<IP_ADDRESS>:8085
management_address=http://<IP_ADDRESS>:8085
metrics_address=http://<IP_ADDRESS>:8082
grpc_inference_port=7070
grpc_management_port=7071
enable_metrics_api=true
metrics_format=prometheus
number_of_netty_threads=4
job_queue_size=10
enable_envvars_config=true
install_py_dep_per_model=true
model_store=/mnt/models/model-store
model_snapshot={"name": "startup.cfg","modelCount": 1,"models": {"xview-fasterrcnn": {"1.0": {"defaultVersion": true,"marName": "xview-fasterrcnn.mar","serialized-file":"trained_model_stripped.pth","extra-files":"index_to_name.json","handler":"fasterrcnn_handler.py","minWorkers": 1,"maxWorkers": 5,"batchSize": 1,"maxBatchDelay": 100,"responseTimeout": 120}}}}
```

#### What the properties.json looks like

```json
[
    {
    "model-name": "xview-fasterrcnn",
    "version": "1.0",
    "model-file": "",
    "serialized-file": "trained_model_stripped.pth",
    "extra-files": "index_to_name.json",
    "handler": "fasterrcnn_handler.py",
    "min-workers" : 1,
    "max-workers": 3,
    "batch-size": 1,
    "max-batch-delay": 100,
    "response-timeout": 120,
    "requirements": ""
  }
]
```

Note that there is a `config/` folder that includes a config.properties. This defines A. There is also a `model-store/` directory that contains are exported models and a `properties.json` file. You will need this file for B.

Now, run several kubectl commands to copy over these folders into your Pod and into the PVC defined directory.

* `kubectl cp kserve_utils/torchserve_utils/config/ model-store-pod:/pv/config/`
* `kubectl cp kserve_utils/torchserve_utils/model-store/ model-store-pod:/pv/model-store/`

Run these commands to verify the contents have been copied over to the pod.

* `kubectl exec --tty model-store-pod -- ls /pv/config`
* `kubectl exec --tty model-store-pod -- ls /pv/model-store`

## Deploying a model using a KServe InferenceService

### Create Inference Service

Below is the yaml definition that defines the KServe InferenceService that deploys models stored in the PVC. A file that defines this PV has already been created in `k8s_files/torch-kserve-pvc.yaml`

```yaml
apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
  name: "torchserve"
spec:
  predictor:
    pytorch:
      storageUri: pvc://task-pv-claim/
```

To create the Pod, run the command: `kubectl apply -f kserve_utils/k8s_files/torch-kserve-pvc.yaml`

* After running the previous command `kubectl get inferenceservice`, you should see that the inferenceservice is not loaded yet
* Keep running the command every minute until you see the InferenceService loaded (view screenshot below of example)
* Next, run command `kubectl get pods` to get underlying pod that is running inference service. Copy the pod name (example seen in screenshot)
* Finally run command: `kubectl logs -f <POD_NAME>` to see the logs and if the model was successfully loaded.

### Complete a sample request and plot predictions

```python
import json
import requests
import base64
from PIL import Image
from PIL import ImageDraw
import matplotlib.pyplot as plt
```

```python
filename='kserve_utils/torchserve_utils/example_img.jpg'
im = Image.open(filename)
```

Here is the test image that will be sent to the deployed model.

```python
im
```

![png](/img/output_26_0.png)

Now, encode the image into the base64 binary format.

```python
image = open(filename, 'rb')  # open binary file in read mode
image_read = image.read()
image_64_encode = base64.b64encode(image_read)
bytes_array = image_64_encode.decode('utf-8')
request = {
  "instances": [
    {
      "data": bytes_array
    }
  ]
}
result_file = "{filename}.{ext}".format(filename=str(filename).split(".")[0], ext="json")
print("Result File: ",result_file)
with open(result_file, 'w') as outfile:
    json.dump(request, outfile, indent=4, sort_keys=True)
```

```
Result File:  kserve_utils/torchserve_utils/example_img.json
```

Now you can submit the image to the deployed endpoint and visualize the predictions!

```python
headers = {
    "Host": "torchserve.default.example.com"
}

data = open(result_file)
response = requests.post('http://localhost:8080/v1/models/xview-fasterrcnn:predict', headers=headers, data=data)

resp = json.loads(response.content)

print("Number of Predictions: ", len(resp['predictions'][0]))
```

```
Number of Predictions:  95
```

```python
draw = ImageDraw.Draw(im)
```

```python
for pred in resp['predictions'][0]:
    assert len(list(pred.keys())) == 2
    cl_name = list(pred.keys())[0]
    bboxes = pred[cl_name]
    # print(cl_names)
    # print(bboxes)
    # print("score: ",pred['score'])
    if pred['score'] > 0.4:
        # bboxes = [int(i) for i in bboxes]
        # print(bboxes[0],type(bboxes[0]))
        draw.rectangle([bboxes[0],bboxes[1],bboxes[2],bboxes[3]],outline=(255,0,0),fill=None,width=1)
        draw.text([bboxes[0],bboxes[1]-10],"{} :{:.2f}".format(cl_name,pred['score']),fill=(250,0,0))
```

Here, you can see the model predictions overlaid onto the input image.

```python
plt.figure(figsize=(12,12))
plt.imshow(im)
```

```
<matplotlib.image.AxesImage at 0x7ff6341ccf10>
```

![png](/img/output_34_1.png)

## Conclusion

There are numerous ways to get started with end-to-end ML model training with HPE. You can find the [full example on GitHub here](https://github.com/interactivetech/e2e_blogposts/tree/main/ngc_blog) To get started with Determined AI’s open source model training platform, visit the [Documentation](https://docs.determined.ai/latest/) page.

If you’re ready to begin your ML journey with HPE, we’re excited to help you get started! [HPE Machine Learning Development Environment](https://www.hpe.com/us/en/solutions/artificial-intelligence/machine-learning-development-environment.html) comes with premium HPE support, which you can read more about [here](https://www.hpe.com/us/en/solutions/artificial-intelligence/machine-learning-development-environment.html).

HPE also offers a purpose-built, turnkey AI infrastructure for model development and training
at scale. [Get in touch with us about HPE Machine Learning Development System here](https://www.hpe.com/us/en/hpe-machine-learning-development-system.html).

repository: hpe-dev-incubator/hpe-dev-portal

gha_language: HTML

repo_name: hpe-dev-incubator/hpe-dev-portal


text:
# ColorGame
Color Guessing Game using  HTML , CSS and JavaScript and Tests your RGB guessing skills
Simple Game
Click on Color Squares as you guess the Randomized RGB codes at the top

[![HitCount](http://hits.dwyl.io/The-King-in-the-North/ColorGame.svg)](http://hits.dwyl.io/The-King-in-the-North/ColorGame)

repository: The-King-in-the-North/ColorGame

repo_name: The-King-in-the-North/ColorGame


text:
# cdpsnarf

Notes
-------

Help Text
-------
```
CDPSnarf v0.1.6 [$Rev: 797 $] initiated.
   Author: Tasos "Zapotek" Laskos
<EMAIL_ADDRESS><EMAIL_ADDRESS>   Website: http://github.com/Zapotek/cdpsnarf

cdpsnarf -i <dev> [-h] [-w savefile] [-r dumpfile] [-d]

   -i      define the interface to sniff on
   -w      write packets to PCAP dump file
   -r      read packets from PCAP dump file
   -d      show debugging information
   -h      show help message and exit
```

Example Usage
-------

Links
-------

repository: ParthaDhar/kaliwiki

repo_name: ParthaDhar/kaliwiki


text:
# Paul

Paul is a library of functions for working with tree data structures such as abstract syntax trees, binary trees, and other nested data structures.

```bash
npm install paul # hey, that rhymes
```

## Usage

```js
var Paul = require('paul');
var Walker = new Paul(['children']);

var tree = {
	value: 10,
	children: [{
		value: 6
	}, {
		value: 4
	}]
};

var sum = Walker.depthReduce(tree, function(num, node) {
	return num + node.value;
}, 0);

require('assert').equal(20, sum);
```

## Documentation

- [`Paul(walkFn)`](https://github.com/andrejewski/paul#paulwalkfn)
	- [`walker(walkFn) func`](https://github.com/andrejewski/paul#walkerwalkfn-func)
	- [`walk(tree, walkFn)`](https://github.com/andrejewski/paul#walktree-walkfn)
	- `prototype`
		- [`map(tree, transform) tree`](https://github.com/andrejewski/paul#maptree-transform-tree)
		- [`filter(tree, predicate) tree`](https://github.com/andrejewski/paul#filtertree-predicate-tree)
		- [`where(tree, properties) tree`](https://github.com/andrejewski/paul#wheretree-properties-tree)
		- [`iterator(tree) Iterator`](https://github.com/andrejewski/paul#iteratortree-iterator)
		- [`forEach(tree, iteratee)`](https://github.com/andrejewski/paul#foreachtree-iteratee)
		- [`find(tree, predicate) node`](https://github.com/andrejewski/paul#findtree-predicate-node)
		- [`findWhere(tree, properties) node`](https://github.com/andrejewski/paul#findwheretree-properties-node)
		- [`reduce(tree, iteratee, [memo]) memo`](https://github.com/andrejewski/paul#reducetree-iteratee-memo-memo)
		- [`parent(tree, node) parentNode`](https://github.com/andrejewski/paul#parenttree-node-parentnode)
		- [`siblings(tree, node) {left, right}`](https://github.com/andrejewski/paul#siblingstree-node-left-right)

**Note:** Methods `iterator()` through `siblings()` are not actual methods and instead there are two methods that differ solely on traversal method. For instance, `find()` must be called as either `depthFind()` or `breadthFind()`. An [explanation](http://stackoverflow.com/a/687752/1444710) of the difference between these traversals may be helpful.

Assume all code examples are preceded by the following:

```js
var Paul = require('paul');
var assert = require('assert');
```

===

### Paul(walkFn)

An instance of Paul will have all of the prototype functions listed above. See `walker(walkFn)` for a description of a valid walking function.

*Note:* the `new` keyword is optional but recommended.

===

### walker(walkFn) func

The function `walker` takes a walking function and returns a function to be used on a node to return its children.

What makes Paul useful is this function. Supply `walker` a walking function with the signature `function(node, walk)`, which decides how a node's children are walked or ignored.

Say you have an AST where you could have one or more child nodes on the properties `left`, `right`, and/or `children` for any given node. The appropriate walk function would be:

```js
function walkFn(node, walk) {
	if(node.left) walk('left');
	if(node.right) walk('right');
	if(node.children) walk('children');
}
var walker1 = Paul.walker(walkFn);
```

Paul walks through the given tree node-by-node with your walk function until no more child nodes are found and the walking ends.

If an AST has a lot of properties that could have child nodes values, `walker` also accepts an array of property keys.

```js
var walker2 = Paul.walker(['left', 'right', 'children']);
// walker 2 does the same as walker 1.
```

===

### walk(tree, walkFn)

Takes a `tree` and a function that accepts a `node` and `walk(node)` callback.

This function is not a method of an instance of Paul for the simple reason that you walk the nodes yourself.

All this function does is pass the given function `walkFn` the `node` to be processed and the function `walk` and return the value computed by the `walkFn`.

Confused? An example will help us both honestly.

```js
var tree = {
	op: '+',
	left: {value: 8},
	right: {
		op: '/',
		left: {value: 20},
		right: {value: 4}
	}
};

var math = Paul.walk(tree, function(node, walk) {
	if(node.op) {
		return '(' + walk(node.left) + node.op + walk(node.right) + ')';
	}
	return node.value;
});

assert.equal(math, '(8+(20/4))');
```

If only the `walkFn` is provided, the function will return a generic function to call on any `tree`.

===

### map(tree, transform) tree

Produces a new tree of nodes by mapping each node in **tree** through a transformation function **transform**. The function is passed one argument `node`.

```js
var pine = {id: "A", kids: [{id: "B"}, {id: "C"}]};
var paul = new Paul(['kids']);

var palm = paul.map(pine, function(node) {
	node.id += node.id;
	return node;
});

var tree = {id: "AA", kids: [{id: "BB"}, {id: "CC"}]};
assert.deepEqual(palm, tree);
```

===

### filter(tree, predicate) tree

Looks through each node in the **tree**, returning a tree of all the nodes that pass a truth test **predicate**. Any nodes that fail the truth test and their sub-nodes will be removed. Sub-nodes will not be tested if the parent node fails.

```js
var pine = {id: "A", kids: [{id: "B"}, {id: "C"}]};
var paul = new Paul(['kids']);

var palm = paul.filter(pine, function(node) {
	return node.id === "A";
});

var tree = {id: "A", kids: []};
assert.deepEqual(palm, tree);
```

===

### where(tree, properties) tree

Looks through each node in the **tree**, returning a tree of all the nodes that contain all of the key-value pairs listed in **properties**. Any nodes that fail the truth test and their sub-nodes will be removed. Sub-nodes will not be tested if the parent node fails.

```js
var pine = {id: "A", kids: [{id: "B"}, {id: "C"}]};
var paul = new Paul(['kids']);

var palm = paul.where(pine, {id: "A"});

var tree = {id: "A", kids: []};
assert.deepEqual(palm, tree);
```

===

### iterator(tree) Iterator

Returns an iterator that returns each node in the tree in the specified traversal order.

- `depthIterator(tree) Iterator`: depth-first
- `breadthIterator(tree) Iterator`: breadth-first

An [explanation](http://stackoverflow.com/a/687752/1444710) of the difference between these traversals may be helpful.

Each `next()` call on the returned iterator returns an object containing a boolean `done` which is true when there are no more nodes to iterator over and `value` which is the node. This conforms to the ES6 [Iterator protocol](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Iteration_protocols).

```js
var tree = {id: "A", kids: [{id: "B"}, {id: "C"}]};
var paul = new Paul(['kids']);

var text = "";
var iter = paul.depthIterator(tree);

while(true) {
	var res = iter.next();
	if(res.done) break;
	text += node.id;
}

assert.equal(text, "ABC");
```

===

### forEach(tree, iteratee)

Iterates over a **tree**, yielding each node in turn to an **iteratee** function. Each invocation of iteratee is called with one argument `node`. Iteration order is decided by which method is used.

- `depthForEach(tree, iteratee)`: depth-first
- `breadthForEach(tree, iteratee)`: breadth-first

```js
var tree = {id: "A", kids: [{id: "B"}, {id: "C"}]};
var paul = new Paul(['kids']);

var text = "";
paul.depthForEach(tree, function(node) {
	text += node.id;
});

assert.equal(text, "ABC");
```

===

### find(tree, predicate) node

Looks through each node in the **tree**, returning the first one that passes a truth test **predicate**, or `undefined` if no node passes the test. The function returns as soon as it finds an acceptable node, and doesn't traverse the entire tree.

- `depthFind(tree, predicate)`: depth-first
- `breadthFind(tree, predicate)`: breadth-first

```js
var tree = {id: "A", kids: [{id: "B"}, {id: "C"}]};
var paul = new Paul(['kids']);

var node = paul.depthFind(tree, function(node) {
	return node.id === "B";
});

assert.deepEqual(node, {id: "B"});
```

===

### findWhere(tree, properties) node

Looks through each node in the **tree**, returning the first node that contains all of the key-value pairs listed in **properties**, or `undefined` if no node passes the test. The function returns as soon as it finds an acceptable node, and doesn't traverse the entire tree.

- `depthFindWhere(tree, properties)`: depth-first
- `breadthFindWhere(tree, properties)`: breadth-first

```js
var tree = {id: "A", kids: [{id: "B"}, {id: "C"}]};
var paul = new Paul(['kids']);

var node = paul.depthFindWhere(tree, {id: "B"}});

assert.deepEqual(node, {id: "B"});
```

===

### reduce(tree, iteratee, [memo]) memo

Also known as inject and foldl, reduce boils down a **tree** of nodes into a single value. **Memo** is the initial state of the reduction, and each successive step of it should be returned by **iteratee**. The iteratee is passed two arguments: the memo, then the node.

If no memo is passed to the initial invocation of reduce, the iteratee is not invoked on the root node of the tree. The root node is instead passed as the memo in the invocation of the iteratee on the next node in the tree.

- `depthReduce(tree, iteratee, [memo])`: depth-first
- `breadthReduce(tree, iteratee, [memo])`: breadth-first

```js
var tree = {id: "A", kids: [{id: "B"}, {id: "C"}]};
var paul = new Paul(['kids']);

var text = paul.depthReduce(tree, function(str, node) {
	return str + node.id;
}, "");

assert.equal(text, "ABC");
```

===

### parent(tree, node) parentNode

Returns the parent node of the given **node**. If the node is not found in the **tree** or the node has no parent (meaning the node is the tree), `undefined` is returned.

- `depthParent(tree, node) parentNode`: depth-first
- `breadthParent(tree, node) parentNode`: breadth-first

```js
var tree = {id: "A", kids: [{id: "B"}, {id: "C"}]};
var paul = new Paul(['kids']);

var nodeB = tree.kids[0];
var parentOfB = paul.depthParent(tree, nodeB);

assert.equal(tree, parentOfB);
```

===

### siblings(tree, node) {left, right}

Returns the sibling nodes of the given **node**. If the node is not found in the **tree** or the node has no parent (meaning the node is the tree), `undefined` is returned. Also if the node is not in an Array member (i.e `{child: node}`), the node cannot have any siblings and thus `undefined` is returned.

Siblings of a node are returned in an object containing two properties `left` and `right`. Siblings that come before the given node are placed in the `left` property. Siblings that come after the given node are placed in the `right` property.

- `depthSiblings(tree, node) {left, right}`: depth-first
- `breadthSiblings(tree, node) {left, right}`: breadth-first

```js
var tree = {id: "A", kids: [{id: "B"}, {id: "C"}, {id: "D"}]};
var paul = new Paul(['kids']);

var nodeC = tree.kids[1];
var siblings = paul.depthSiblings(tree, nodeC);

assert.deepEqual(siblings, {
	left: [{id: "B"}],
	right: [{id: "D"}]
});
```

## Why "Paul"?

I have a history of naming awesome libraries after awesome people I know with [four](https://github.com/andrejewski/reem) [letter](https://github.com/andrejewski/matt) [names](https://github.com/andrejewski/seth). I need this library for further work with my HTML parser project [Himalaya](https://github.com/andrejewski/himalaya). Also, have you ever seen [The Fast and the Furious](http://en.wikipedia.org/wiki/The_Fast_and_the_Furious)?

Yes, this project is dedicated to [Paul Walker](http://en.wikipedia.org/wiki/Paul_Walker).

## Contributing

We can always have more tests: if you find a bug, create an issue or be **fabulous** and fix the problem and write the tests up yourself in a coherent pull request. Same goes for documentation: typo fixes and clearer function descriptions are appreciated.

Run tests with the `npm test` command.

Follow me on [Twitter](https://twitter.com/compooter) for updates or just for the lolz and please check out my other [repositories](https://github.com/andrejewski) if I have earned it. I thank you for reading.

license: ISC

repository: andrejewski/paul

repo_name: andrejewski/paul


text:
# Intro

Mango-ml is a machine learning framework designed for fast experimentation.

Mango provides a set of interfaces and guidelines to build and maintain
machine learning models, including support for advanced logging and keeping track
of parameters between runs.

# Quick start

Mango has a bunch of useful abstractions that encourage separation of concerns
and code reuse:

- Model
- Dataset
- Loader
- Reporter
- Experiment

The following file represent a very simple classification system written with mango.

```
import mango

class Dataset(mango.Dataset):
  seed = mango.Param(int, default=0)

  def build(self):
      random.seed(self.seed)
      X, y = make_classification()
      self.X_train, self.X_test, self.y_train, self.y_test = train_test_split(X, y)

  def train(self):
      self.data = self.X_train, self.y_train

  def eval(self):
      self.data = self.X_test, self.y_test

class Model(mango.Model):
  alpha = mango.Param(float)

  def build(self):
      self.clf = LogisticRegression(alpha=self.alpha)

  def train(self, loader):
      X, y = loader.train()

      self.clf.fit(X, y)

  def eval(self, loader):
      X,  y = loader.eval()
      self.reporter.add_scalar("score", self.clf.score(X, y))

      self.reporter.log('Saving model')
      with self.context['experiment'].file('file.pickle') as fd:
        pickle.dump(fd, self.clf)

class Main(mango.Experiment):

  reporter = mango.CombinedReporter([LogReporter(), TextReporter()])
  model = Model(alpha=0.5)
  dataset = Dataset()
  loader = SimpleLoader(dataset)
  trainer = SimpleTrainer(model, loader)

```

Once we defined our Experiment, we can run it using the following command:

```
mango train example.Main
```

## Minibatch training

Many machine learning system support long running times and minibatch training, we'll see how to create a simple minibatch trainer using mango

```

class Dataset(mango.BatchedDataset):

  def build(self):
      random.seed(self.seed)
      X, y = make_classification()
      self.X_train, self.X_test, self.y_train, self.y_test = train_test_split(X, y)

class BatchLogistic(mango.BatchModel):

  def build(self):
    pass

  def batch(self, batch, step_info):
    pass

  def epoch(self, loader, step_info):
    pass

class MiniBatch(mango.experiment)

  dataset = Dataset()
  model = mango.BatchModel()

```

## Improving code structure

In this section we will see how to train a machine learning system with mango. While mango doesn't enforce a particular file structure, a good standard is to use a typical python package layout. In this example, we will create a package structure containing the models, experiments and datasets modules.

```
hello/
 README
 hello/
    __init__.py
    models.py
    experiments.py
    datasets.py
```

The entry point for a mango application is the Experiment.

repository: gabrielelanaro/mango-ml

repo_name: gabrielelanaro/mango-ml


text:
---
layout: default
---

# 1997-03-21 - Re: Jackboots in Canada

## Header Data

From: Eric Murray<EMAIL_ADDRESS>To<EMAIL_ADDRESS>Message Hash: 777965ff572b2c46d67ab59759625299af577b2bc463360d4b0c17e079589bfc

Message ID<EMAIL_ADDRESS>Reply To<EMAIL_ADDRESS>UTC Datetime: 1997-03-21 22:29:03 UTC

Raw Date: Fri, 21 Mar 1997 14:29:03 -0800 (PST)

## Raw message

```
{% raw  %}From: Eric Murray<EMAIL_ADDRESS>Date: Fri, 21 Mar 1997 14:29:03 -0800 (PST)
To<EMAIL_ADDRESS>Subject: Re: Jackboots in Canada
In-Reply-To<EMAIL_ADDRESS>Message-ID<EMAIL_ADDRESS>MIME-Version: 1.0
Content-Type: text/plain

<EMAIL_ADDRESS>writes:
> AP 3/21/97
>
> Quebec
>
> Spurred by a new outbreak of bombings, the Canadian government agreed
> Thursday to toughen laws to help Quebec stamp out a war between motorcycle
> gangs that has killed more than 30 people.
>
> [snip]
>
> ...Hell's Angels and Rock Machine...have been battling for control of the
> illegal drug trade in Canada.
>
> Canadian justice minister, Allan Rock, said the measures would include
> broadening laws on search warrants, electronic eavesdropping and bail
> conditions for arrested gang members.

The alternative, proposed by some goverment person (governor?) in Quebec, was
to suspend "individual liberties" for bikers, making it possible to
pull one over at any time, search them, and if explosives are found
assume they're guilty until proven innocent.

That last part just floored me, how could anyone be willing to throw
away rights like that?  At least here in the US they're not so
obvious about it- the cash (or cars or computers) are seized and
presumed guilty until proven innocent, and of course you can't mount a
defense if everything you own is seized and your bank accounts frozen, but
you're still presumed innocent.  It was that evil CASH that did it.

--
   Eric Murray<EMAIL_ADDRESS>Network security and encryption consulting.
PGP keyid:E03F65E5 fingerprint:50 B0 A2 4C 7D 86 FC 03  92 E8 AC E6 7E 27 29 AF

{% endraw %}
```

## Thread

+ Return to [March 1997](/archive/1997/03)

+ Return to "[(Alec)camcc@abraxis.com](/authors/_alec_camcc_at_abraxis_com)"
+ Return to "[Duncan Frissell <frissell@panix.com>](/authors/duncan_frissell_frissell_at_panix_com_)"
+ Return to "[Eric Murray <ericm@lne.com>](/authors/eric_murray_ericm_at_lne_com_)"
+ Return to "[snow <snow@smoke.suba.com>](/authors/snow_snow_at_smoke_suba_com_)"
+ Return to "["Timothy C. May" <tcmay@got.net>](/authors/timothy_c_may_tcmay_at_got_net_)"
+ Return to "[Toto <toto@sk.sympatico.ca>](/authors/toto_toto_at_sk_sympatico_ca_)"

+ 1997-03-21 (Fri, 21 Mar 1997 05:33:58 -0800 (PST)) - [Jackboots in Canada](/archive/1997/03/d9a62a54d1a9d2e5fa02c6a9ba7349d123e707ce5f8deb91b0432b1337525627) -<EMAIL_ADDRESS>  + 1997-03-21 (Fri, 21 Mar 1997 14:29:03 -0800 (PST)) - Re: Jackboots in Canada - _Eric Murray<EMAIL_ADDRESS>    + 1997-03-25 (Mon, 24 Mar 1997 20:27:44 -0800 (PST)) - [Re: Jackboots in Canada](/archive/1997/03/ab447dabe2756df9b0fcc670204e0e9d3a75f8831fd7dc09b4a27059ce72b07a) - _snow<EMAIL_ADDRESS>      + 1997-03-25 (Tue, 25 Mar 1997 03:01:57 -0800 (PST)) - [Re: Jackboots in Canada](/archive/1997/03/139bad739fe0e44a08c12de293918fbda87940cc4920a1f772e6d6f23823dc1a) - _Toto<EMAIL_ADDRESS>  + 1997-03-22 (Fri, 21 Mar 1997 16:23:22 -0800 (PST)) - [Re: Jackboots in Canada](/archive/1997/03/23ce16129879004f01db2cebe3fec69163de820486d0b03e9b0fbd47b4ff5766) - _"Timothy C. May"<EMAIL_ADDRESS>  + 1997-03-22 (Fri, 21 Mar 1997 18:00:44 -0800 (PST)) - [Re: Jackboots in Canada](/archive/1997/03/f3b2524c4a14b28b7c8fc4a2f3b885e3f15ffda86ce971736a15f07d6e1afdb8) -<EMAIL_ADDRESS>  + 1997-03-22 (Sat, 22 Mar 1997 05:52:23 -0800 (PST)) - [Re: Jackboots in Canada](/archive/1997/03/ca135c91a2f566a1b81f2f53539d9004de8474c86731de888c802f3c908ee2aa) - _Duncan Frissell<EMAIL_ADDRESS>


text:
# Install Seafile Client on Linux

* [Ubuntu](#wiki-ubuntu)
* [Debian](#wiki-debian)
* [Fedora](#wiki-fedora)
* [Centos/RHEL](#wiki-centos)
* [Arch Linux](#wiki-archlinux)
* [Linux CLI](#wiki-cli)

##
Ubuntu

Ubuntu users can install Seafile client from the [Official PPA](https://code.launchpad.net/~seafile/+archive/ubuntu/seafile-client):

```sh
sudo add-apt-repository ppa:seafile/seafile-client
sudo apt-get update
sudo apt-get install seafile-gui
```

If you only want to install the command-line client, run `sudo apt-get install seafile-cli` instead.

If you want to install the debug symbols (for example, when you want to report a bug), you can install the following packages:

```sh
sudo apt-get install libsearpc-dbg ccnet-dbg libccnet-dbg seafile-daemon-dbg libseafile-dbg seafile-gui-dbg
```

Now you can start seafile client from Unity's dash.

##
Debian

Debian users can install Seafile client from our debian repo:

To install the client, first add the signing key:

```sh
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 8756C4F765C9AC3CB6B85D62379CE192D401AB61
```

Then add the repo to your apt source list, using the line corresponding to your debian version :

```
# For Debian 7
echo deb http://deb.seadrive.org wheezy main | sudo tee /etc/apt/sources.list.d/seafile.list

# For Debian 8
echo deb http://deb.seadrive.org jessie main | sudo tee /etc/apt/sources.list.d/seafile.list

# For Debian 9
echo deb http://deb.seadrive.org stretch main | sudo tee /etc/apt/sources.list.d/seafile.list
```

Update your local apt cache :

```
sudo apt-get update
```

Now install the client:

```sh
sudo apt-get install seafile-gui
```

If you only want to install the command-line client, run `sudo apt-get install seafile-cli` instead.

##
Fedora (Community Maintained)

There is a *community maintained* Seafile Client RPM package in Fedora's [official repository](https://src.fedoraproject.org/rpms/seafile).

##
Centos/RHEL (Community Maintained)

There is a *community maintained* Seafile Client RPM package for CentOS/RHEL https://copr.fedorainfracloud.org/coprs/pkerling/seafile/.

##
Arch Linux (Community Maintained)

There is a *community maintained* Seafile Client package for Arch Linux:

https://aur.archlinux.org/packages/seafile-client/

##
Linux CLI Usage

Please refer to [this documentation](linux-cli.md) for how to use Linux client on a command line server.

repository: haiwen/seafile-user-manual

gha_language: CSS

repo_name: haiwen/seafile-user-manual