text:
---
**
Tutorial navigation
**

| [Introduction](/docs/tutorial.md)                                 | [1. Installation and first-time setup](/docs/tutorial/01_installation.md)                      | [2. Deploy and check a first node](/docs/tutorial/02_first_node.md)                                              | <nobr>👇You are here👇</nobr>
[3. Scale your processes](/docs/tutorial/03_scale.md)                                                                | [4. Testing your processes and platforms](/docs/tutorial/04_test.md)                              | [5. Extend Hybrid Platforms Conductor with your own requirements](/docs/tutorial/05_extend_with_plugins.md)                |
| ---------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------- |
| **[Use-case](/docs/tutorial.md#use-case)**             | **[Dependencies installation](/docs/tutorial/01_installation.md#hpc-dependencies)** | **[Add your first node and its platform repository](/docs/tutorial/02_first_node.md#add-first-node)** | **[Provision our web services platform](/docs/tutorial/03_scale.md#provision)**                           | **[Hello test framework](/docs/tutorial/04_test.md#framework)**                        | **[Create your plugins' repository](/docs/tutorial/05_extend_with_plugins.md#plugins-repo)**                    |
| **[Prerequisites](/docs/tutorial.md#prerequisites)**   | **[Our platforms' main repository](/docs/tutorial/01_installation.md#main-repo)**   | **[Check and deploy services on this node](/docs/tutorial/02_first_node.md#check-deploy)**            | **[Run commands on our new web services](/docs/tutorial/03_scale.md#run)**                                | **[Testing your nodes](/docs/tutorial/04_test.md#nodes-tests)**                        | **[Your own platform handler](/docs/tutorial/05_extend_with_plugins.md#platform-handler)**                      |
| **[Tutorial setup](/docs/tutorial.md#tutorial-setup)** |                                                                                | **[Updating the configuration](/docs/tutorial/02_first_node.md#update)**                              | **[Check and deploy our web services on several nodes at once](/docs/tutorial/03_scale.md#check-deploy)** | **[Testing your platforms' configuration](/docs/tutorial/04_test.md#platforms-tests)** | **[Write your own tests](/docs/tutorial/05_extend_with_plugins.md#test)**                                       |
|                                                                              |                                                                                |                                                                                |                                                                                | **[Other kinds of tests](/docs/tutorial/04_test.md#other-tests)**                      | **[Enough of stdout, we want to report to other tools](/docs/tutorial/05_extend_with_plugins.md#report)**       |
|                                                                              |                                                                                |                                                                                |                                                                                |                                                                                | **[What next?](/docs/tutorial/05_extend_with_plugins.md#what-next)**                                            |

# 3. Scale your processes

In this section we will cover how Hybrid Platforms Conductor scales naturally your DevOps processes.

We'll take a real world example: Web services running on hosts accessible through SSH.
We'll use Docker to have those hosts running, so that even if you don't own an infrastructure you can see go on with this tutorial.

Then we'll see how Hybrid Platforms Conductor helps in checking, deploying, running all those services on those nodes in a very simple way.

## Provision our web services platform

The goal here is top have a full platform provisioned by Docker of web services.
Then we'll play with it.

First we'll create a Docker image with the following features:
* A web server (written in Go and running on port 80) that outputs a simple Hello world, whose message is taken from the file `/root/hello_world.txt`.
* An OpenSSH server that allows SSH connections to the `root` account, authenticated with a RSA key.

We'll do this by creating all files needed, the Dockerfile, and building the image:
```bash
mkdir -p ~/hpc_tutorial/web_docker_image

# The Go web server code
cat <<EOF >~/hpc_tutorial/web_docker_image/main.go
package main

import (
    "fmt"
    "io/ioutil"
    "log"
    "net/http"
    "os"
)

const homepageEndPoint = "/"

// StartWebServer the webserver
func StartWebServer() {
    http.HandleFunc(homepageEndPoint, handleHomepage)
    port := os.Getenv("PORT")
    if len(port) == 0 {
        panic("Environment variable PORT is not set")
    }

    log.Printf("Starting web server to listen on endpoints [%s] and port %s",
        homepageEndPoint, port)
    if err := http.ListenAndServe(":"+port, nil); err != nil {
        panic(err)
    }
}

func handleHomepage(w http.ResponseWriter, r *http.Request) {
    urlPath := r.URL.Path
    log.Printf("Web request received on url path %s", urlPath)
    content, content_err := ioutil.ReadFile("/root/hello_world.txt")
    if content_err != nil {
        fmt.Printf("Failed to read message to display, err: %s", content_err)
    }
    _, write_err := w.Write(content)
    if write_err != nil {
        fmt.Printf("Failed to write response, err: %s", write_err)
    }
}

func main() {
    StartWebServer()
}
EOF

# The hello_world message file
cat <<EOF >~/hpc_tutorial/web_docker_image/hello_world.txt
Hello World!
EOF

# Generate root admin RSA keys
yes y | ssh-keygen -t rsa -b 2048 -C<EMAIL_ADDRESS>-f ~/hpc_tutorial/web_docker_image/hpc_root.key -N ""

# The Docker start script
cat <<EOF >~/hpc_tutorial/web_docker_image/start.sh
#!/bin/bash

# Start sshd as a daemon
/usr/sbin/sshd

# Start web server
sh -c /codebase/bin/server
EOF

# The Dockerfile
cat <<EOF >~/hpc_tutorial/web_docker_image/Dockerfile
# syntax=docker/dockerfile:1
# Pull the image containing Go
FROM golang:1.16.3-buster

# Install the web server
# Create the message file to be displayed by the web server
COPY hello_world.txt /root/hello_world.txt
# Copy the code
COPY main.go /codebase/src/main.go
# Build the binary
RUN cd /codebase && go build -v -o /codebase/bin/server ./src/main.go
# Set the env which will be available at runtime
ENV PORT=80
EXPOSE 80

# Install sshd
RUN apt-get update && apt-get install -y openssh-server
RUN mkdir /var/run/sshd
# Activate root login
RUN sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config
# Speed-up considerably ssh performance and avoid huge lags and timeouts without DNS
RUN sed -i 's/#UseDNS yes/UseDNS no/' /etc/ssh/sshd_config
EXPOSE 22

# Upload our root key for key authentication of root
COPY hpc_root.key.pub /root/.ssh/authorized_keys
RUN chmod 700 /root/.ssh
RUN chmod 400 /root/.ssh/authorized_keys

# Startup script
COPY start.sh /start.sh
RUN chmod +x /start.sh
CMD ["/start.sh"]
EOF

# Build the Docker image named hpc_tutorial_web
DOCKER_BUILDKIT=1 docker build -t hpc_tutorial_web ~/hpc_tutorial/web_docker_image
# =>
# [+] Building 27.7s (20/20) FINISHED
#  => [internal] load build definition from Dockerfile                                                                                0.0s
#  => => transferring dockerfile: 32B                                                                                0.0s
#  => [internal] load .dockerignore                                                                                0.0s
#  => => transferring context: 2B                                                                                0.0s
#  => resolve image config for docker.io/docker/dockerfile:1                                                                                0.6s
#  => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:e2a8561e419ab1ba6b2fe6cbdf49fd92b95912df1cf7d313c3e2230a333fdbcc                                                                               0.0s
#  => [internal] load metadata for docker.io/library/golang:1.16.3-buster                                                                                0.6s
#  => [ 1/13] FROM docker.io/library/golang:1.16.3-buster@sha256:9d64369fd3c633df71d7465d67d43f63bb31192193e671742fa1c26ebc3a6210                                                                               0.0s
#  => [internal] load build context                                                                                0.0s
#  => => transferring context: 1.19kB                                                                                0.0s
#  => CACHED [ 2/13] COPY hello_world.txt /root/hello_world.txt                                                                                0.0s
#  => [ 3/13] COPY main.go /codebase/src/main.go                                                                                0.1s
#  => [ 4/13] RUN cd /codebase && go build -v -o /codebase/bin/server ./src/main.go                                                                                1.8s
#  => [ 5/13] RUN apt-get update && apt-get install -y openssh-server                                                                                18.3s
#  => [ 6/13] RUN mkdir /var/run/sshd                                                                                0.6s
#  => [ 7/13] RUN sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config                                                                                0.7s
#  => [ 8/13] RUN sed -i 's/#UseDNS yes/UseDNS no/' /etc/ssh/sshd_config                                                                                0.6s
#  => [ 9/13] COPY hpc_root.key.pub /root/.ssh/authorized_keys                                                                                0.1s
#  => [10/13] RUN chmod 700 /root/.ssh                                                                                0.6s
#  => [11/13] RUN chmod 400 /root/.ssh/authorized_keys                                                                                0.6s
#  => [12/13] COPY start.sh /start.sh                                                                                0.1s
#  => [13/13] RUN chmod +x /start.sh                                                                                0.7s
#  => exporting to image                                                                                1.7s
#  => => exporting layers                                                                                1.6s
#  => => writing image sha256:38183990af6d364d19e9ba7b45aec02a103c82cf6aaf26a0dfbbdb803e067c3c                                                                                0.0s
#  => => naming to docker.io/library/hpc_tutorial_web                                                                                0.0s
```

So now it's time to create the Docker containers hosting our web services!

We'll create 10 of them, named `webN`, and associate the hostnames `webN.hpc_tutorial.org` to them.
```bash
# Provision 10 containers
for ((i=1;i<=10;i++));
do
   docker run --hostname "web$i.hpc_tutorial.org" --name "web$i" -P -d hpc_tutorial_web
done
```

Then to be closer to a real-world situation, we will use hostnames and IPs to access our web services.
To do that, we will generate the hostnames/ip mapping in the /etc/hosts file of our containers.
```bash
for ((i=1;i<=10;i++));
do
   echo "$(docker container inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' web$i)        web$i.hpc_tutorial.org" >>/etc/hosts
done
```

We can check that our web services are running correctly by using a simple test script:
```bash
cat <<EOF >~/hpc_tutorial/web_docker_image/test.bash
#!/bin/bash

for ((i=1;i<=10;i++));
do
   echo "Container web\$i: \$(curl http://web\$i.hpc_tutorial.org 2>/dev/null)"
done
EOF
chmod a+x ~/hpc_tutorial/web_docker_image/test.bash

~/hpc_tutorial/web_docker_image/test.bash
# =>
# Container web1: Hello World!
# Container web2: Hello World!
# Container web3: Hello World!
# Container web4: Hello World!
# Container web5: Hello World!
# Container web6: Hello World!
# Container web7: Hello World!
# Container web8: Hello World!
# Container web9: Hello World!
# Container web10: Hello World!
```

Please note that if we exit your Docker tutorial container and restart it, you will need to restart your web containers and regenerate their hostname/ip in `/etc/hosts`.
This will be done this way (to be done each time you will restart your tutorial or web containers):
```bash
for ((i=1;i<=10;i++));
do
  docker container start web$i
  echo "$(docker container inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' web$i)        web$i.hpc_tutorial.org" >>/etc/hosts
done
```

Now we are in front of a real-world situation: 10 web services running behind hostnames. Let's see how to handle them with Hybrid Platforms Conductor.

## Run commands on our new web services

In order for Hybrid Platforms Conductor's processes to handle those new web services, we start by adding those new nodes to our inventory, and register the `web-hello` service to them:
```bash
for ((i=1;i<=10;i++));
do
cat <<EOF >>~/hpc_tutorial/my-service-conf-repo/inventory.yaml
web$i:
  metadata:
    description: Web service nbr $i
    hostname: web$i.hpc_tutorial.org
  # The list of service names this node should have
  services:
    - web-hello
EOF
done
```

Now they should appear in our inventory with [`report`](/docs/executables/run.md):
```bash
./bin/report
# =>
# +-------+----------------------+------------------------+-------------+-----------+----+-----------------------+------------+
# | Node  | Platform             | Host name              | IP          | Physical? | OS | Description           | Services   |
# +-------+----------------------+------------------------+-------------+-----------+----+-----------------------+------------+
# | local | my-service-conf-repo |                        |             | No        |    | The local environment | my-service |
# | web1  | my-service-conf-repo | web1.hpc_tutorial.org  | <IP_ADDRESS>  | No        |    | Web service nbr 1     | web-hello  |
# | web10 | my-service-conf-repo | web10.hpc_tutorial.org | <IP_ADDRESS> | No        |    | Web service nbr 10    | web-hello  |
# | web2  | my-service-conf-repo | web2.hpc_tutorial.org  | <IP_ADDRESS>  | No        |    | Web service nbr 2     | web-hello  |
# | web3  | my-service-conf-repo | web3.hpc_tutorial.org  | <IP_ADDRESS>  | No        |    | Web service nbr 3     | web-hello  |
# | web4  | my-service-conf-repo | web4.hpc_tutorial.org  | <IP_ADDRESS>  | No        |    | Web service nbr 4     | web-hello  |
# | web5  | my-service-conf-repo | web5.hpc_tutorial.org  | <IP_ADDRESS>  | No        |    | Web service nbr 5     | web-hello  |
# | web6  | my-service-conf-repo | web6.hpc_tutorial.org  | <IP_ADDRESS>  | No        |    | Web service nbr 6     | web-hello  |
# | web7  | my-service-conf-repo | web7.hpc_tutorial.org  | <IP_ADDRESS> | No        |    | Web service nbr 7     | web-hello  |
# | web8  | my-service-conf-repo | web8.hpc_tutorial.org  | <IP_ADDRESS> | No        |    | Web service nbr 8     | web-hello  |
# | web9  | my-service-conf-repo | web9.hpc_tutorial.org  | <IP_ADDRESS> | No        |    | Web service nbr 9     | web-hello  |
# +-------+----------------------+------------------------+-------------+-----------+----+-----------------------+------------+
```
You can already see that the IP address has already been discovered and added to the nodes' metadata.
This is done thanks to the [`host_ip` CMDB plugin](/docs/plugins/cmdb/host_ip.md).

As our web services require the `root` RSA key to connect to them, let's add it to our ssh agent (you'll have to redo this each time you exit and restart the `hpc_tutorial` container):
```bash
eval "$(ssh-agent -s)"
ssh-add ~/hpc_tutorial/web_docker_image/hpc_root.key
# => Identity added: /root/hpc_tutorial/web_docker_image/hpc_root.key<EMAIL_ADDRESS>```

Now that our nodes are accessible we can perform some commands on them.
The [`run` executable](/docs/executables/run.md) has an extensive CLI to perform many operations on nodes, handling parallel executions, timeouts...
Here we'll see some of those operations that can save hours of manual operations when a large number of nodes is involved.

Run simple commands on all nodes at once, and display them in the output:
```bash
# Execute 1 command on all nodes, using the root user for nodes needing SSH access
./bin/run --ssh_user root --all --command "echo Hostname here is \$(hostname)"
# =>
# Hostname here is e8dddeb2ba25
# Hostname here is web1.hpc_tutorial.org
# Hostname here is web10.hpc_tutorial.org
# Hostname here is web2.hpc_tutorial.org
# Hostname here is web3.hpc_tutorial.org
# Hostname here is web4.hpc_tutorial.org
# Hostname here is web5.hpc_tutorial.org
# Hostname here is web6.hpc_tutorial.org
# Hostname here is web7.hpc_tutorial.org
# Hostname here is web8.hpc_tutorial.org
# Hostname here is web9.hpc_tutorial.org
```

Here a lot has already happened!
We see that the command has been executed on all nodes: the local node (`e8dddeb2ba25` in this output) and the other web* nodes.
However, the local node has no SSH access: it uses the [`local` connector plugin](/docs/plugins/connector/local.md), whereas all other nodes use an SSH access with their IP and SSH user root, thanks to the [`ssh` connector plugin](/docs/plugins/connector/ssh.md).
Connector plugins know which nodes they are able to connect to thanks each node's metadata (in our case `local_node` and `host_ip` metadata were used).
Thanks to this plugins-oriented architecture, Hybrid Platforms Conductor is able to run the same command on all those nodes in the same interface.

What if our commands are long and verbose, and we want to execute them in parallel?
Use the `--parallel` switch, and commands will be run in parallel, dumping the output in files inside the `./run_logs` directory:
```bash
./bin/run --ssh_user root --all --command "echo Hostname here is \$(hostname)" --parallel
# =>
# Executing actions [100%] - |                                                                                C| - [ Queue: 0 - Processing: 0 - Done: 11 - Total: 11 ]

ls -la run_logs
# total 52
# drwxr-xr-x 2 root root 4096 Apr 28 14:44 .
# drwxr-xr-x 6 root root 4096 Apr 28 13:10 ..
# -rw-r--r-- 1 root root   30 Apr 28 14:51 local.stdout
# -rw-r--r-- 1 root root   39 Apr 28 14:51 web1.stdout
# -rw-r--r-- 1 root root   40 Apr 28 14:51 web10.stdout
# -rw-r--r-- 1 root root   39 Apr 28 14:51 web2.stdout
# -rw-r--r-- 1 root root   39 Apr 28 14:51 web3.stdout
# -rw-r--r-- 1 root root   39 Apr 28 14:51 web4.stdout
# -rw-r--r-- 1 root root   39 Apr 28 14:51 web5.stdout
# -rw-r--r-- 1 root root   39 Apr 28 14:51 web6.stdout
# -rw-r--r-- 1 root root   39 Apr 28 14:51 web7.stdout
# -rw-r--r-- 1 root root   39 Apr 28 14:51 web8.stdout
# -rw-r--r-- 1 root root   39 Apr 28 14:51 web9.stdout

cat run_logs/web4.stdout
# => Hostname here is web4.hpc_tutorial.org
```

The [`ssh` connector plugin](/docs/plugins/connector/ssh.md) allows us to not use the `--ssh_user` parameter if we set the `hpc_ssh_user` environment variable.
Let's do it to avoid having to repeat our SSH user on any command line needing it:
```bash
export hpc_ssh_user=root
```

What if we want to run commands on a subset of nodes?
You can select nodes based on their name, regular expressions, nodes lists they belong to, services they contain...
Check the [`run`](/docs/executables/run.md) documentation on `./bin/run --help` for more details.

Here are some examples:
```bash
# Run only on 1 node
./bin/run --command "echo Hostname here is \$(hostname)" --node web4
# =>
# Hostname here is web4.hpc_tutorial.org

# Run on several nodes
./bin/run --command "echo Hostname here is \$(hostname)" --node web4 --node web8
# =>
# Hostname here is web4.hpc_tutorial.org
# Hostname here is web8.hpc_tutorial.org

# Run on nodes selected with regular expressions
./bin/run --command "echo Hostname here is \$(hostname)" --node /web\[135\].*/
# =>
# Hostname here is web1.hpc_tutorial.org
# Hostname here is web10.hpc_tutorial.org
# Hostname here is web3.hpc_tutorial.org
# Hostname here is web5.hpc_tutorial.org

# Run on nodes selected by their service
./bin/run --command "echo Hostname here is \$(hostname)" --node-service my-service
# =>
# Hostname here is e8dddeb2ba25
```

What if we have several commands to execute?
```bash
# Several commands specifid in command line arguments
./bin/run --node /web\[135\].*/ --command "echo Hostname here is" --command hostname
# =>
# Hostname here is
# web1.hpc_tutorial.org
# Hostname here is
# web10.hpc_tutorial.org
# Hostname here is
# web3.hpc_tutorial.org
# Hostname here is
# web5.hpc_tutorial.org

# Run commands from a file
cat <<EOF >my_commands.bash
echo Hostname here is
hostname
EOF
./bin/run --node /web\[135\].*/ --commands-file my_commands.bash
# =>
# Hostname here is
# web1.hpc_tutorial.org
# Hostname here is
# web10.hpc_tutorial.org
# Hostname here is
# web3.hpc_tutorial.org
# Hostname here is
# web5.hpc_tutorial.org
```

So now you already have powerful tools to operate a large number of nodes and platforms, and automate such operations.

Let's apply this to manually change the configuration of our first 5 web servers, and return `Hello Mars!` instead of `Hello World!`.
```bash
./bin/run --node /web\[1-5\]\$/ --command 'echo Hello Mars! >/root/hello_world.txt'
```

And check that the web servers have indeed changed their response, using our test script:
```bash
~/hpc_tutorial/web_docker_image/test.bash
# =>
# Container web1: Hello Mars!
# Container web2: Hello Mars!
# Container web3: Hello Mars!
# Container web4: Hello Mars!
# Container web5: Hello Mars!
# Container web6: Hello World!
# Container web7: Hello World!
# Container web8: Hello World!
# Container web9: Hello World!
# Container web10: Hello World!
```

Now let's use more DevOps processes than manual changes.

## Check and deploy our web services on several nodes at once

Now that we have plenty of web services, let's add a way to configure the services there.
We'll add a new service file in our configuration that can change the hello world message, and include the hostname and IP address of the host.

We add configuration methods that check and deploy the `web-hello` service (we follow the same logic as for our `my-service` service - using the `check` and `deploy` methods):
```bash
cat <<EOF >~/hpc_tutorial/my-service-conf-repo/service_web-hello.rb
# Get actions to check the node's service against the wanted content
#
# Parameters::
# * *node* (String): Node on which we check the service
# Result::
# * Array< Hash<Symbol,Object> >: The list of actions
def check(node)
  # We first dump the wanted content in a temporary file and then we diff it.
  # We will access the node's planet, hostname and IP from its metadata using the NodesHandler API, through the @nodes_handler object
  [
    {
      remote_bash: <<~EOS
        echo 'Hello #{@nodes_handler.get_planet_of(node) || 'World'} from #{@nodes_handler.get_hostname_of(node)} (#{@nodes_handler.get_host_ip_of(node)})' >/tmp/hello_world.txt.wanted
        echo Diffs on hello_world.txt:
        if test -f /root/hello_world.txt; then
          diff /root/hello_world.txt /tmp/hello_world.txt.wanted || true
        else
          echo "Create hello_world.txt from scratch"
          cat /tmp/hello_world.txt.wanted
        fi
      EOS
    }
  ]
end

# Get actions to deploy the node's service against the wanted content
#
# Parameters::
# * *node* (String): Node on which we deploy the service
# Result::
# * Array< Hash<Symbol,Object> >: The list of actions
def deploy(node)
  # We first check, as this will display diffs and prepare the file to be copied.
  # And then we really deploy the file on our node.
  check(node) + [
    {
      remote_bash: <<~EOS
        mkdir -p ~/hpc_tutorial/node
        cp /tmp/hello_world.txt.wanted /root/hello_world.txt
      EOS
    }
  ]
end
EOF
```

We can check that our service is correctly defined by issuing a simple [`check-node`](/docs/executables/check-node.md) on one of the web nodes:
```bash
./bin/check-node --node web1
# =>
# ===== Packaging deployment ==== Begin...
# ===== Packaging deployment ==== ...End
#
# ===== Checking on 1 nodes ==== Begin...
# ===== [ web1 / web-hello ] - HPC Service Check ===== Begin
# ===== [ web1 / web-hello ] - HPC Service Check ===== Begin
# Diffs on hello_world.txt:
# 1c1
# < Hello Mars!
# ---
# > Hello World from web1.hpc_tutorial.org (<IP_ADDRESS>)
# ===== [ web1 / web-hello ] - HPC Service Check ===== End
# ===== [ web1 / web-hello ] - HPC Service Check ===== End
# Executing actions [100%] - |                                                                                C| - [ Queue: 0 - Processing: 0 - Done: 1 - Total: 1 ]
# ===== Checking on 1 nodes ==== ...End
```

If we want to check several nodes at once, we can use [`deploy`](/docs/executables/deploy.md) with the `--why-run` flag, and any nodes selector that we've seen in the previous tutorial section can also be used here.

Example, we want to check the nodes `web4`, `web5` and `web6`:
```bash
./bin/deploy --why-run --node /web[4-6]/
# =>
# ===== Packaging deployment ==== Begin...
# ===== Packaging deployment ==== ...End
#
# ===== Checking on 3 nodes ==== Begin...
# ===== [ web4 / web-hello ] - HPC Service Check ===== Begin
# ===== [ web4 / web-hello ] - HPC Service Check ===== Begin
# Diffs on hello_world.txt:
# 1c1
# < Hello Mars!
# ---
# > Hello World from web4.hpc_tutorial.org (<IP_ADDRESS>)
# ===== [ web4 / web-hello ] - HPC Service Check ===== End
# ===== [ web4 / web-hello ] - HPC Service Check ===== End
# ===== [ web5 / web-hello ] - HPC Service Check ===== Begin
# ===== [ web5 / web-hello ] - HPC Service Check ===== Begin
# Diffs on hello_world.txt:
# 1c1
# < Hello Mars!
# ---
# > Hello World from web5.hpc_tutorial.org (<IP_ADDRESS>)
# ===== [ web5 / web-hello ] - HPC Service Check ===== End
# ===== [ web5 / web-hello ] - HPC Service Check ===== End
# ===== [ web6 / web-hello ] - HPC Service Check ===== Begin
# ===== [ web6 / web-hello ] - HPC Service Check ===== Begin
# Diffs on hello_world.txt:
# 1c1
# < Hello World!
# ---
# > Hello World from web6.hpc_tutorial.org (<IP_ADDRESS>)
# ===== [ web6 / web-hello ] - HPC Service Check ===== End
# ===== [ web6 / web-hello ] - HPC Service Check ===== End
# Executing actions [100%] - |                                                                                C| - [ Queue: 0 - Processing: 0 - Done: 3 - Total: 3 ]
# ===== Checking on 3 nodes ==== ...End
```
We see clearly the differences that would be applied in case we deploy for real.

If you want to execute those in parallel and see outputs in `./run_logs/*.stdout` files, you can use the `--parallel` flag here too!
```bash
./bin/deploy --why-run --node /web[4-6]/ --parallel
# =>
# ===== Packaging deployment ==== Begin...
# ===== Packaging deployment ==== ...End
#
# ===== Checking on 3 nodes ==== Begin...
# Executing actions [100%] - |                                                                                C| - [ Queue: 0 - Processing: 0 - Done: 3 - Total: 3 ]
# Executing actions [100%] - |                                                                                C| - [ Queue: 0 - Processing: 0 - Done: 3 - Total: 3 ]
# ===== Checking on 3 nodes ==== ...End

cat run_logs/web4.stdout
# =>
# ===== [ web4 / web-hello ] - HPC Service Check ===== Begin
# ===== [ web4 / web-hello ] - HPC Service Check ===== Begin
# Diffs on hello_world.txt:
# 1c1
# < Hello Mars!
# ---
# > Hello World from web4.hpc_tutorial.org (<IP_ADDRESS>)
# ===== [ web4 / web-hello ] - HPC Service Check ===== End
# ===== [ web4 / web-hello ] - HPC Service Check ===== End
```

So now that we have great ways to check which nodes have diverged, let's deploy a bunch of them and see the result live.
We'll deploy web2 to web8.
```bash
./bin/deploy --node /web[2-8]/ --parallel
```

And then we check the result live:
```bash
~/hpc_tutorial/web_docker_image/test.bash
# =>
# Container web1: Hello Mars!
# Container web2: Hello World from web2.hpc_tutorial.org (<IP_ADDRESS>)
# Container web3: Hello World from web3.hpc_tutorial.org (<IP_ADDRESS>)
# Container web4: Hello World from web4.hpc_tutorial.org (<IP_ADDRESS>)
# Container web5: Hello World from web5.hpc_tutorial.org (<IP_ADDRESS>)
# Container web6: Hello World from web6.hpc_tutorial.org (<IP_ADDRESS>)
# Container web7: Hello World from web7.hpc_tutorial.org (<IP_ADDRESS>)
# Container web8: Hello World from web8.hpc_tutorial.org (<IP_ADDRESS>)
# Container web9: Hello World!
# Container web10: Hello World!
```
We see that only web services from 2 to 8 have been deployed.

By the way, remember the [`last_deploys` executable](/docs/executables/last_deploys.md)?
Time to check the new deployment there too.

```bash
./bin/last_deploys
# =>
# [...]
# +-------+---------------------+-------+------------+-------------------------------------------------------------------------------------------------------------------------------------+
# | Node  | Date                | Admin | Services   | Error                                                                                |
# +-------+---------------------+-------+------------+-------------------------------------------------------------------------------------------------------------------------------------+
# | web1  |                     |       |            | Error: failed_command                                                                                |
# |       |                     |       |            | /bin/bash: line 1: cd: /var/log/deployments: No such file or directory                                                              |
# |       |                     |       |            | Command '/tmp/hpc_ssh/platforms_ssh_5120020210428-1741-15weu1h/ssh hpc.web1 /bin/bash <<'EOF'' returned error code 1 (expected 0).  |
# | web10 |                     |       |            | Error: failed_command                                                                                |
# |       |                     |       |            | /bin/bash: line 1: cd: /var/log/deployments: No such file or directory                                                              |
# |       |                     |       |            | Command '/tmp/hpc_ssh/platforms_ssh_5120020210428-1741-15weu1h/ssh hpc.web10 /bin/bash <<'EOF'' returned error code 1 (expected 0). |
# | web9  |                     |       |            | Error: failed_command                                                                                |
# |       |                     |       |            | /bin/bash: line 1: cd: /var/log/deployments: No such file or directory                                                              |
# |       |                     |       |            | Command '/tmp/hpc_ssh/platforms_ssh_5120020210428-1741-15weu1h/ssh hpc.web9 /bin/bash <<'EOF'' returned error code 1 (expected 0).  |
# | local | 2021-04-27 17:09:44 | root  | my-service |                                                                                |
# | web2  | 2021-04-28 16:58:06 | root  | web-hello  |                                                                                |
# | web3  | 2021-04-28 16:58:06 | root  | web-hello  |                                                                                |
# | web4  | 2021-04-28 16:58:06 | root  | web-hello  |                                                                                |
# | web5  | 2021-04-28 16:58:06 | root  | web-hello  |                                                                                |
# | web6  | 2021-04-28 16:58:06 | root  | web-hello  |                                                                                |
# | web7  | 2021-04-28 16:58:06 | root  | web-hello  |                                                                                |
# | web8  | 2021-04-28 16:58:06 | root  | web-hello  |                                                                                |
# +-------+---------------------+-------+------------+-------------------------------------------------------------------------------------------------------------------------------------+
```
Some nodes haven't been deployed yet, so they return errors when trying to read deployment logs.
However we can clearly see on the other nodes that deployment was done a few seconds ago.

### Check and deploy all our nodes with various services at once

Now it's time to get even further: we also want to make some web services nodes implement our very first service `my-service` (remember the service configuring the file `~/hpc_tutorial/node/my-service.conf`).

This operation will only require to alter our inventory: we add the services we want on the nodes we want, with the metadata we want.
No need to change any configuration code.

Let's say we want the `my-service` service to be implemented in web services from web1 to web5, and that we want to assign different planets in those web services as well.
We will change our inventory file that will look like that in the end (see the various services and metadata changed):
```bash
cat <<EOF >~/hpc_tutorial/my-service-conf-repo/inventory.yaml
---
local:
  metadata:
    description: The local environment
    local_node: true
    service_port: 1107
    service_timeout: 60
  services:
    - my-service
web1:
  metadata:
    description: Web service nbr 1
    hostname: web1.hpc_tutorial.org
    planet: Mercury
    service_port: 1201
    service_timeout: 60
  services:
    - web-hello
    - my-service
web2:
  metadata:
    description: Web service nbr 2
    hostname: web2.hpc_tutorial.org
    planet: Venus
    service_port: 1202
    service_timeout: 60
  services:
    - web-hello
    - my-service
web3:
  metadata:
    description: Web service nbr 3
    hostname: web3.hpc_tutorial.org
    planet: Earth
    service_port: 1203
    service_timeout: 60
  services:
    - web-hello
    - my-service
web4:
  metadata:
    description: Web service nbr 4
    hostname: web4.hpc_tutorial.org
    planet: Mars
    service_port: 1204
    service_timeout: 60
  services:
    - web-hello
    - my-service
web5:
  metadata:
    description: Web service nbr 5
    hostname: web5.hpc_tutorial.org
    planet: Jupiter
    service_port: 1205
    service_timeout: 60
  services:
    - web-hello
    - my-service
web6:
  metadata:
    description: Web service nbr 6
    hostname: web6.hpc_tutorial.org
  services:
    - web-hello
web7:
  metadata:
    description: Web service nbr 7
    hostname: web7.hpc_tutorial.org
  services:
    - web-hello
web8:
  metadata:
    description: Web service nbr 8
    hostname: web8.hpc_tutorial.org
  services:
    - web-hello
web9:
  metadata:
    description: Web service nbr 9
    hostname: web9.hpc_tutorial.org
  services:
    - web-hello
web10:
  metadata:
    description: Web service nbr 10
    hostname: web10.hpc_tutorial.org
  # The list of service names this node should have
  services:
    - web-hello
EOF
```

We can check that services are assigned correctly using [`report`](/docs/executables/report.md):
```bash
./bin/report
# =>
# +-------+----------------------+------------------------+-------------+-----------+----+-----------------------+-----------------------+
# | Node  | Platform             | Host name              | IP          | Physical? | OS | Description           | Services              |
# +-------+----------------------+------------------------+-------------+-----------+----+-----------------------+-----------------------+
# | local | my-service-conf-repo |                        |             | No        |    | The local environment | my-service            |
# | web1  | my-service-conf-repo | web1.hpc_tutorial.org  | <IP_ADDRESS>  | No        |    | Web service nbr 1     | my-service, web-hello |
# | web10 | my-service-conf-repo | web10.hpc_tutorial.org | <IP_ADDRESS> | No        |    | Web service nbr 10    | web-hello             |
# | web2  | my-service-conf-repo | web2.hpc_tutorial.org  | <IP_ADDRESS>  | No        |    | Web service nbr 2     | my-service, web-hello |
# | web3  | my-service-conf-repo | web3.hpc_tutorial.org  | <IP_ADDRESS>  | No        |    | Web service nbr 3     | my-service, web-hello |
# | web4  | my-service-conf-repo | web4.hpc_tutorial.org  | <IP_ADDRESS>  | No        |    | Web service nbr 4     | my-service, web-hello |
# | web5  | my-service-conf-repo | web5.hpc_tutorial.org  | <IP_ADDRESS>  | No        |    | Web service nbr 5     | my-service, web-hello |
# | web6  | my-service-conf-repo | web6.hpc_tutorial.org  | <IP_ADDRESS>  | No        |    | Web service nbr 6     | web-hello             |
# | web7  | my-service-conf-repo | web7.hpc_tutorial.org  | <IP_ADDRESS> | No        |    | Web service nbr 7     | web-hello             |
# | web8  | my-service-conf-repo | web8.hpc_tutorial.org  | <IP_ADDRESS> | No        |    | Web service nbr 8     | web-hello             |
# | web9  | my-service-conf-repo | web9.hpc_tutorial.org  | <IP_ADDRESS> | No        |    | Web service nbr 9     | web-hello             |
# +-------+----------------------+------------------------+-------------+-----------+----+-----------------------+-----------------------+
```
Some nodes are having several services.

Without changing anything else, we can check all those services on all nodes with 1 command line:
```bash
./bin/deploy --why-run --all --parallel
# =>
# ===== Packaging deployment ==== Begin...
# ===== Packaging deployment ==== ...End
#
# ===== Checking on 11 nodes ==== Begin...
# Executing actions [100%] - |                                                                                C| - [ Queue: 0 - Processing: 0 - Done: 11 - Total: 11 ]
# Executing actions [100%] - |                                                                                C| - [ Queue: 0 - Processing: 0 - Done: 11 - Total: 11 ]
# ===== Checking on 11 nodes ==== ...End
#

# Check the diffs and files creations from run_logs
ls run_logs/* | xargs grep -e Create -e '>'
# =>
# run_logs/web1.stdout:> Hello Mercury from web1.hpc_tutorial.org (<IP_ADDRESS>)
# run_logs/web1.stdout:Create file from scratch
# run_logs/web10.stdout:> Hello World from web10.hpc_tutorial.org (<IP_ADDRESS>)
# run_logs/web2.stdout:> Hello Venus from web2.hpc_tutorial.org (<IP_ADDRESS>)
# run_logs/web2.stdout:Create file from scratch
# run_logs/web3.stdout:> Hello Earth from web3.hpc_tutorial.org (<IP_ADDRESS>)
# run_logs/web3.stdout:Create file from scratch
# run_logs/web4.stdout:> Hello Mars from web4.hpc_tutorial.org (<IP_ADDRESS>)
# run_logs/web4.stdout:Create file from scratch
# run_logs/web5.stdout:> Hello Jupiter from web5.hpc_tutorial.org (<IP_ADDRESS>)
# run_logs/web5.stdout:Create file from scratch
# run_logs/web9.stdout:> Hello World from web9.hpc_tutorial.org (<IP_ADDRESS>)
```
We see that `my-service` file will be created from scratch on `web1-5`, and the Hello World message of `web-hello` service will be corrected on `web1-5`, `web9` and `web10`.

This is typically the kind of situation that can occur often when the nodes are being maintained manually, or have suffered from local and temporary modifications by operators to cope with urgencies.

Now that we are happy with the review of those changes, we can align all our services on all our nodes:
```bash
./bin/deploy --all --parallel
# =>
# ===== Packaging deployment ==== Begin...
# ===== Packaging deployment ==== ...End
#
# ===== Deploying on 11 nodes ==== Begin...
# Executing actions [100%] - |                                                                                C| - [ Queue: 0 - Processing: 0 - Done: 11 - Total: 11 ]
# Executing actions [100%] - |                                                                                C| - [ Queue: 0 - Processing: 0 - Done: 11 - Total: 11 ]
#   ===== Saving deployment logs for 11 nodes ==== Begin...
# Executing actions [100%] - |                                                                                C| - [ Queue: 0 - Processing: 0 - Done: 11 - Total: 11 ]
#   ===== Saving deployment logs for 11 nodes ==== ...End
  #
# ===== Deploying on 11 nodes ==== ...End
```

And we can perform all the checks to make sure the deployment went smoothly:
```bash
./bin/last_deploys
# =>
# +-------+---------------------+-------+-----------------------+-------+
# | Node  | Date                | Admin | Services              | Error |
# +-------+---------------------+-------+-----------------------+-------+
# | local | 2021-04-28 17:34:17 | root  | my-service            |       |
# | web1  | 2021-04-28 17:34:17 | root  | web-hello, my-service |       |
# | web10 | 2021-04-28 17:34:17 | root  | web-hello             |       |
# | web2  | 2021-04-28 17:34:17 | root  | web-hello, my-service |       |
# | web3  | 2021-04-28 17:34:17 | root  | web-hello, my-service |       |
# | web4  | 2021-04-28 17:34:17 | root  | web-hello, my-service |       |
# | web5  | 2021-04-28 17:34:17 | root  | web-hello, my-service |       |
# | web6  | 2021-04-28 17:34:17 | root  | web-hello             |       |
# | web7  | 2021-04-28 17:34:17 | root  | web-hello             |       |
# | web8  | 2021-04-28 17:34:17 | root  | web-hello             |       |
# | web9  | 2021-04-28 17:34:17 | root  | web-hello             |       |
# +-------+---------------------+-------+-----------------------+-------+

# Check the configuration of web-hello services
~/hpc_tutorial/web_docker_image/test.bash
# =>
# Container web1: Hello Mercury from web1.hpc_tutorial.org (<IP_ADDRESS>)
# Container web2: Hello Venus from web2.hpc_tutorial.org (<IP_ADDRESS>)
# Container web3: Hello Earth from web3.hpc_tutorial.org (<IP_ADDRESS>)
# Container web4: Hello Mars from web4.hpc_tutorial.org (<IP_ADDRESS>)
# Container web5: Hello Jupiter from web5.hpc_tutorial.org (<IP_ADDRESS>)
# Container web6: Hello World from web6.hpc_tutorial.org (<IP_ADDRESS>)
# Container web7: Hello World from web7.hpc_tutorial.org (<IP_ADDRESS>)
# Container web8: Hello World from web8.hpc_tutorial.org (<IP_ADDRESS>)
# Container web9: Hello World from web9.hpc_tutorial.org (<IP_ADDRESS>)
# Container web10: Hello World from web10.hpc_tutorial.org (<IP_ADDRESS>)

# Check which node has the my-service file configured
./bin/run --all --command "echo \$(hostname) - \$(ls /root/hpc_tutorial/node/my-service.conf 2>/dev/null)"
# e8dddeb2ba25 - /root/hpc_tutorial/node/my-service.conf
# web1.hpc_tutorial.org - /root/hpc_tutorial/node/my-service.conf
# web10.hpc_tutorial.org -
# web2.hpc_tutorial.org - /root/hpc_tutorial/node/my-service.conf
# web3.hpc_tutorial.org - /root/hpc_tutorial/node/my-service.conf
# web4.hpc_tutorial.org - /root/hpc_tutorial/node/my-service.conf
# web5.hpc_tutorial.org - /root/hpc_tutorial/node/my-service.conf
# web6.hpc_tutorial.org -
# web7.hpc_tutorial.org -
# web8.hpc_tutorial.org -
# web9.hpc_tutorial.org -

# Check that there is nothing else to be changed on all our nodes
./bin/deploy --why-run --all --parallel
ls run_logs/* | xargs grep -e Create -e '>'
# =>
```

**Woot!**
You managed to:
* configure **different services on several nodes using different technologies** (local connection, using SSH, on Docker),
* **running commands, checking and deploying configuration on those nodes using the same simple 1-liners interface**, despite those nodes running on different technologies,
* **change the distribution and configuration** of your services and nodes only by editing your inventory and metadata,
* **track and check easily** all your deployments.

This already gives you powerful tools to **manage heterogeneous platforms and environments at scale** and empower the agility of your DevOps processes.

Next steps are about testing your processes and configurations so that you have very simple ways to monitor when they break, and integrate those tests in CI/CD-like workflows.

*Files that have been covered in this section can be checked in [this example tutorial folder](/examples/tutorial/03_scale).*

**[Next >> Test your processes and platforms](/docs/tutorial/04_test.md)**

---
**
Tutorial navigation
**

| [Introduction](/docs/tutorial.md)                                 | [1. Installation and first-time setup](/docs/tutorial/01_installation.md)                      | [2. Deploy and check a first node](/docs/tutorial/02_first_node.md)                                              | <nobr>👇You are here👇</nobr>
[3. Scale your processes](/docs/tutorial/03_scale.md)                                                                | [4. Testing your processes and platforms](/docs/tutorial/04_test.md)                              | [5. Extend Hybrid Platforms Conductor with your own requirements](/docs/tutorial/05_extend_with_plugins.md)                |
| ---------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------- |
| **[Use-case](/docs/tutorial.md#use-case)**             | **[Dependencies installation](/docs/tutorial/01_installation.md#hpc-dependencies)** | **[Add your first node and its platform repository](/docs/tutorial/02_first_node.md#add-first-node)** | **[Provision our web services platform](/docs/tutorial/03_scale.md#provision)**                           | **[Hello test framework](/docs/tutorial/04_test.md#framework)**                        | **[Create your plugins' repository](/docs/tutorial/05_extend_with_plugins.md#plugins-repo)**                    |
| **[Prerequisites](/docs/tutorial.md#prerequisites)**   | **[Our platforms' main repository](/docs/tutorial/01_installation.md#main-repo)**   | **[Check and deploy services on this node](/docs/tutorial/02_first_node.md#check-deploy)**            | **[Run commands on our new web services](/docs/tutorial/03_scale.md#run)**                                | **[Testing your nodes](/docs/tutorial/04_test.md#nodes-tests)**                        | **[Your own platform handler](/docs/tutorial/05_extend_with_plugins.md#platform-handler)**                      |
| **[Tutorial setup](/docs/tutorial.md#tutorial-setup)** |                                                                                | **[Updating the configuration](/docs/tutorial/02_first_node.md#update)**                              | **[Check and deploy our web services on several nodes at once](/docs/tutorial/03_scale.md#check-deploy)** | **[Testing your platforms' configuration](/docs/tutorial/04_test.md#platforms-tests)** | **[Write your own tests](/docs/tutorial/05_extend_with_plugins.md#test)**                                       |
|                                                                              |                                                                                |                                                                                |                                                                                | **[Other kinds of tests](/docs/tutorial/04_test.md#other-tests)**                      | **[Enough of stdout, we want to report to other tools](/docs/tutorial/05_extend_with_plugins.md#report)**       |
|                                                                              |                                                                                |                                                                                |                                                                                |                                                                                | **[What next?](/docs/tutorial/05_extend_with_plugins.md#what-next)**                                            |


text:
# vega_datasets

[![build status](http://img.shields.io/travis/altair-viz/vega_datasets/master.svg?style=flat)](https://travis-ci.org/altair-viz/vega_datasets)

A Python package for offline access to [vega datasets](https://github.com/vega/vega-datasets).

This package has several goals:

- Provide straightforward access in Python to the datasets made available at [vega-datasets](https://github.com/vega/vega-datasets).
- return the results in the form of a Pandas dataframe.
- wherever dataset size and/or license constraints make it possible, bundle the dataset with the package so that datasets can be loaded in the absence of a web connection.

Currently the package bundles a half-dozen datasets, and falls back to using HTTP requests for the others.

## Installation

```
$ pip install vega_datasets
```

## Usage

The main object in this library is ``data``:

```python
>>> from vega_datasets import data
```

It contains attributes that access all available datasets, locally if
available. For example, here is the well-known iris dataset:

```python
>>> df = data.iris()
>>> df.head()
   petalLength  petalWidth  sepalLength  sepalWidth species
0          1.4         0.2          5.1         3.5  setosa
1          1.4         0.2          4.9         3.0  setosa
2          1.3         0.2          4.7         3.2  setosa
3          1.5         0.2          4.6         3.1  setosa
4          1.4         0.2          5.0         3.6  setosa
```

If you're curious about the source data, you can access the URL for any of the available datasets:

```python
>>> data.iris.url
'https://vega.github.io/vega-datasets/data/iris.json'
```

For datasets bundled with the package, you can also find their location on disk:

```python
>>> data.iris.filepath
'/lib/python3.6/site-packages/vega_datasets/data/iris.json'
```

## Available Datasets

To list all the available datsets, use ``list_datsets``:

```python
>>> data.list_datasets()
['7zip', 'airports', 'anscombe', 'barley', 'birdstrikes', 'budget', 'budgets', 'burtin', 'cars', 'climate', 'co2-concentration', 'countries', 'crimea', 'disasters', 'driving', 'earthquakes', 'ffox', 'flare', 'flare-dependencies', 'flights-10k', 'flights-200k', 'flights-20k', 'flights-2k', 'flights-3m', 'flights-5k', 'flights-airport', 'gapminder', 'gapminder-health-income', 'gimp', 'github', 'graticule', 'income', 'iris', 'jobs', 'londonBoroughs', 'londonCentroids', 'londonTubeLines', 'lookup_groups', 'lookup_people', 'miserables', 'monarchs', 'movies', 'normal-2d', 'obesity', 'points', 'population', 'population_engineers_hurricanes', 'seattle-temps', 'seattle-weather', 'sf-temps', 'sp500', 'stocks', 'udistrict', 'unemployment', 'unemployment-across-industries', 'us-10m', 'us-state-capitals', 'weather', 'weball26', 'wheat', 'world-110m', 'zipcodes']
```

To list local datasets (i.e. those that are bundled with the package and can be used without a web connection), use the ``local_data`` object instead:

```python
>>> from vega_datasets import local_data
>>> local_data.list_datasets()

['airports', 'anscombe', 'barley', 'burtin', 'cars', 'crimea', 'driving', 'iowa-electricity', 'iris', 'seattle-temps', 'seattle-weather', 'sf-temps', 'stocks']
```

We plan to add more local datasets in the future, subject to size and licensing constraints. See the [local datasets issue](https://github.com/altair-viz/vega_datasets/issues/1) if you would like to help with this.

## Dataset Information

If you want more information about any dataset, you can use the ``description`` property:

```python
>>> data.iris.description
'This classic dataset contains lengths and widths of petals and sepals for 150 iris flowers, drawn from three species. It was introduced by R.A. Fisher in 1936 [1]_.'
```

This information is also part of the ``data.iris`` doc string.
Descriptions are not yet included for all the datasets in the package; we hope to add more information on this in the future.

license: MIT

branch_name: refs/heads/master

extension: md

license_type: permissive


text:
# Car Plate Box Dectection
Project developed for the discipline of Signais and systems at UFPE,

## Main dependencies:
  -   numpy
  -   openCV
  -   easyocr
  -   pytorch
  -   imutils
  -   glob

Tested using Anaconda Environment with Python 3.8.5

## Running the PlateBox Detect:
Run the code ```plateBoxDetect.py ``` for find car plate Box with fourier transform in images dataset located in  ```images/ ```, for more details of the operation see the report.
## Running the Basic openCV implementation:
Run the code ```openCVDetect.py ``` for find car plate Box with openCV Canny Edge Detection algoritms in images dataset located in ```images/ ```, for more details of the operation see the report.

repository: nogsp/CarPlateBoxDectection

branch_name: refs/heads/main

repo_name: nogsp/CarPlateBoxDectection


text:
Bali
====

[![License](https://img.shields.io/github/license/zsr2531/Bali?style=for-the-badge&logo=internet-archive)](https://github.com/zsr2531/Bali/tree/master/LICENSE)
[![Docs](https://img.shields.io/badge/docs-Docfx-blueviolet?style=for-the-badge&logo=read-the-docs)](https://zsr2531.github.io/Bali/api)
[![Nuget (with prereleases)](https://img.shields.io/nuget/vpre/Bali?style=for-the-badge&logo=nuget)](https://www.nuget.org/packages/Bali)
[![Nuget](https://img.shields.io/nuget/dt/Bali?color=blue&style=for-the-badge&logo=nuget)](https://www.nuget.org/packages/Bali)

A fast and lightweight .Net library to read and write Java .class files.

Features
--------

- Low memory footprint, the library doesn't allocate much from the heap.
- Provides low level access to the constant pool.
- Handles Java's "modified UTF-8" encoding correctly, unlike many other libraries out there.
- High level representation of the class file. *(todo)*
- Ability to read .jar files. *(todo)*

Why `Bali`?
-----------

> The island of Bali lies 3.2 km (2.0 mi) east of Java.

...

Want to contribute?
-------------------

If you wish to contribute, I will gladly appreciate it. You can read the contributing guidelines and a few small tips [here](CONTRIBUTING.md).

External libraries used
-----------------------

- [CodeGenHelpers](https://github.com/dansiegel/CodeGenHelpers) (MIT License): Used only for aiding source generation.

Disclaimer
----------

The library is still under heavy development.


text:
# HackYeah 2019
Project for HackYeah 2019 - Team K3MP-B

## Members
* Mathias Åsberg - [@Mindgames](https://github.com/Mindgames)
* Maciej Baranek - [@MAC3R](https://github.com/MAC3R)
* Bartek Pacia - [@bartekpacia](https://github.com/bartekpacia)
* Koppány Kondricz - [@kondricz](https://github.com/kondricz)
* Paweł Lubczyński - [@lubek1983](https://github.com/lubek1983)
* Bence Szegvári - [@bencesz](https://github.com/bencesz)

## Technology
### Frontend
* Angular

### Backend

##### Quiz provider microservice

- Express JS
- Node JS
- Typescript (optional)

Backend service which handles providing quiz question, and the interactions around them. Main responsabilities:

1. Provide a list of quiz questions when game starts based on categories and other query parameters

2. Handle incoming POST requests from users, when they are marking answers in questions (1v1 VS aggregate)

3. Replying to users if they have had the correct answers. Upon missed / hit answers trigger PUBSUB or google functions which are updating other, connected databases.

4. Handling a game close.

### Other
* Google Cloud Platform
* GitHub

gha_license_id: MIT


text:
# AVA: A large-scale database for aesthetic visual analysis

## Brief Information

### Description

一个大规模图像美学数据集。

### Scale

250,000+

### Annotation

aesthetic scores for each image, semantic labels for over 60 categories as well as labels related to photographic style

### Project Link

### Paper Link

http://refbase.cvc.uab.es/files/MMP2012a.pdf

## Note

XXXXXXXX

repository: harrywang/design-ai-resources

repo_name: harrywang/design-ai-resources


text:
# ng-scaffolding

*An opinionated boilerplate for AngularJS + Bootstrap*

## Table of Contents

- [Overview](#overview)
    - [Purpose](#purpose)
    - [Philosophy](#philosophy)
    - [Components](#components)
    - [Browser Support](#browser-support)
- [Quickstart](#quickstart)
- [Learn](#learn)
    - [Overall Directory Structure](#overall-directory-structure)
    - [Naming Conventions](#file-naming-conventions)
        - [Files](#files)
        - [ng-specific](#angular-specific)
    - [Tooling](#tooling)
        - [Node](#node)
        - [Gulp](#gulp)
    - [The Build System](#the-build-system)
        - [Gulp Commands](#gulp-commands)
        - [Master Config File](#master-config-file)
    - [Resources](#resources)

## Overview

Kickstart your [AngularJS](http://angularjs.org/) apps with a solid foundation.

It is based on and inspired by a number of boilerplate projects already available out there on the web,
and borrows heavily from the following list including but not limited to:

- [ngBoilerplate](http://bit.ly/ngBoilerplate)
- [ng-launchpad](http://www.samora.me/ng-launchpad)
- [Web Starter Kit](https://developers.google.com/web/starter-kit/)

##### Notable Differences

`ngScaffolding`, like `ngBoilerplate`, is meant to make life easy by providing a basic framework with which to kickstart AngularJS projects.
However, `ngScaffolding` is simpler and differs in a few ways:

- [gulp.js](http://gulpjs.com), instead of [Grunt](http://gruntjs.com), as a build/management tool.
- Integrated development server.
- [Protractor](https://github.com/angular/protractor) integration for end-to-end (e2e) tests.
- Development build has a similar directory structure to production. However the files are not minified and compressed.
- No planned support for CoffeeScript, it's all in plain old vanilla JavaScript (and always will be).

`ngScaffolding` is a simpler than `ngBoilerplate` on purpose, yet keeps the same basic principles and ideas.

##### Key Features

- Modular application structure to encourage component re-use.
- Automagically minify, compress, uglify and package files for production.
- Built-in development server which reloads the page on file changes.
- Automagically re-run tests on file changes.
- Use Less to simplify your CSS.

### Purpose

`ngScaffolding` is designed to make life easy by providing a basic framework with which to kickstart AngularJS projects.
It contains a best-practice directory structure to ensure code reusability and maximum scalability.

### Philosophy

The principal goal of `ngScaffolding` is to set projects up for long-term success.
So `ngScaffolding` tries to follow best practices everywhere it can.
These are:

- Properly orchestrated modules to encourage drag-and-drop component re-use.
- Tests exist alongside the component they are testing with no separate `test` directory required;
  the build process should be sophisticated enough to handle this.
- Speaking of which, the build system should work automagically, without involvement from the developer.
  It should do what needs to be done, while staying out of the way.
  Components should end up tested, linted, compiled, and minified, ready for use in a production environment.
- Integration with popular tools like Bower, Karma, and LESS.
- *Encourages* test-driven development. It's the only way to code.
- A directory structure that is cogent, meaningful to new team members, and supporting of the above points.
- Well-documented, to show new developers *why* things are set up the way they are.
- It should be responsive to evidence. Community feedback is therefore crucial to the success of `ngScaffolding`.

But `ngScaffolding` is not an example of an AngularJS app: this is a kickstarter.
If you're looking for an example of what a complete, non-trivial AngularJS app that does something real looks like,
complete with a REST backend and authentication and authorization,
then take a look at [`angular-app`](http://github.com/angular-app/angular-app),
which does just that - and does it well.

### Components

`ngScaffolding` also comes prepackaged with the most popular design frameworks around.
The following have already been added because I believe they are the bare minimum components you may need.
Add or remove according to your requirements.

- [AngularJS](http://angularjs.org)
- [Twitter Bootstrap](http://getbootstrap.com/)
- [AngularStrap](http://mcgrea.github.io/angularstrap)
- [Font Awesome](http://fontawesome.io)
- [LESS](http://lesscss.org)

Lastly, it contains a sophisticated [Gulp](http://gulpjs.com)-based build system to ensure maximum productivity.

### Browser Support

At present, I officially aim to support the following browsers:

* IE10, IE11, IE Mobile 10
* FF 30, 31
* Chrome 34, 35
* Safari 7, 8
* Opera 23, 24
* iOS Safari 7, 8
* Opera Coast
* Android / Chrome 4.4, 4.4.3
* BlackBerry 10

This is not to say that ngScaffolding cannot be used in browsers older than those reflected,
but merely that my focus will be on ensuring that the project works great in the above.

## Quickstart

[Download](https://github.com/doshprompt/ng-scaffolding/archive/master.zip) the project or clone this repository and build on what is already included in the `app` directory.

Pre-requisites: You will need to have installed Node.js and Git.

Then run the following commands in a terminal window:

```sh
$ git clone git://github.com/doshprompt/ng-scaffolding
$ cd ng-scaffolding
$ sudo npm install -g gulp bower
$ npm install
$ gulp
```

This should automatically open up a new tab/window in your default browser.

happy hacking, enjoy!

## Learn

### Overall Directory Structure

At a high level, the structure looks roughly something like this:

```
myApp
├── app
│   └── <development sandbox>
├── build
│   ├── karma.conf.js
│   ├── protractor.conf.js
│   └── ...
├── bower.json
├── gulpfile.js
├── package.json
├── .bowerrc
└── .gitignore
```

- `app` - the app's root directory.
- `build` - customizable build settings, including test runners.
- `tests` - all unit and end-to-end test suites go in here.
- `bower.json` - this is the project configuration for Bower and it contains the list of Bower dependencies that are needed.
- `gulpfile.js` - build script.
- `package.json` - metadata about the app, used by NPM and the build script. NPM dependencies are also listed here.
- `.bowerrc` - the Bower configuration file. This tells Bower to install bower_components into the `vendor/` directory instead.
- `.gitignore` - top level .gitignore file

### Naming Conventions

#### Files

- Configs: `{*.conf.js|*.conf.json}`
- Controllers: `*.ctrl.js`
- Services: `*.svc.js`
- Routing: `*.route.js`
- Templates: `*.part.html`
- Resource Files: `*.lang.json`
- Unit Tests: `*.spec.js`
- E2E Tests: `*.scenario.js`
- Mocks: `*.mock.js`
- LESS: `*.less`

#### Angular-specific

- Controller Names: `*Control`
- Module Names: `TODO`
- Services: `TODO`
- Models: `TODO`

### Tooling

If you would like to use the tooling provided as is,
make sure your system has [Node.js](http://nodejs.org) and [gulp.js](http://gulpjs.com) installed.

#### Node

Let's check to see if you already have Node installed.
Bring up a terminal and type `node --version`.
If Node responds, and if it shows a version at or above v0.10.x, proceed to checking if you have Gulp installed too.
If you require Node, go to [nodejs.org](http://nodejs.org/) and click on the big green Install button.

#### Gulp

Bring up a terminal and type `gulp --version`.
If Gulp is installed it should return a version number at or above 3.5.x.
If you don't see any errors, proceed to the Gulp commands section.
If you need to install Gulp, open up a terminal and type in the following:

```sh
$ npm install --global gulp
```

This will install Gulp globally.
Depending on your user account, you may need to gain elevated permissions using `sudo` (i.e `sudo npm install --global gulp`).
Next, install the local dependencies `ngScaffolding` requires:

```sh
$ npm install
```

That's it! You should now have everything needed to use the Gulp tools in ngScaffolding.

### The Build System

#### Gulp Commands

You can now use Gulp with the following commands to stay productive during development:

##### Watch For Changes & Refresh Across Devices

```sh
$ gulp server
```

This outputs an IP address you can use to locally test and another that can be used on devices connected to your network.

##### Compile

```sh
$ gulp
```

Build and optimize the current project, ready for deployment.
This includes linting as well as image, script, stylesheet and HTML optimization and minification.
Akin to running `make`

### Resources

**Read This**

- [Scalable Code Organization in AngularJS](https://medium.com/opinionated-angularjs/9f01b594bf06)

**Styleguides**

- [Airbnb JavaScript Style Guide](https://github.com/airbnb/javascript)
- [An AngularJS Style Guide for Closure Users at Google](https://google-styleguide.googlecode.com/svn/trunk/angularjs-google-style.html)

**Further Reading**

- [Best Practices](https://github.com/angular/angular.js/wiki/Best-Practices)
- [Anti Patterns](https://github.com/angular/angular.js/wiki/Anti-Patterns)
- [AngularJS Git Commit Message Conventions](https://docs.google.com/document/d/1QrDFcIiPjSLDn3EL15IJygNPiHORgU1_OOAqWjiDU5Y/edit#heading=h.uyo6cb12dt6w)
- [Best Practice Recommendations for Angular App Structure](https://docs.google.com/document/d/1XXMvReO8-Awi1EZXAXS4PzDzdNvV6pGcuaF4Q9821Es/pub)

**Other Resources**

- [Community-driven set of best practices for AngularJS application development](https://github.com/mgechev/angularjs-style-guide)

repository: doshprompt/ng-scaffolding

repo_name: doshprompt/ng-scaffolding


text:
---
layout: post
title: GETFLIX
subtitle: Not-Netflix :p ... A movie recommendation system
gh-repo: asmitks/GETFLIX
gh-badge: [star, fork, follow]
tags: [Recommendation system, Python, Mongo, Flask]
comments: true
---

# GETFLIX
This is a :movie_camera: movie recommendation system based on collaborative learning (user-user and item-item) which was applied on the IMDB
database.The recommendation system asks for ratings on 4 or more movies out of approx. 500 movies and returns 4 recommendations each
for User-User and Item-Item collaborative learning algorithms.
![alt text](https://raw.githubusercontent.com/asmitks/images/master/a.png)

## Tools Used
- Python 3.5
- Mongo DB
- Docker
- Heroku
- mlab
- PyMongo
- Flask
- HTML/CSS
- Pandas
- Beautiful soup

## Sources Referred
- **Collective Intelligence** by *Toby Segaran*

## How to Use?

You can find the hosted app here [GETFLIX](https://get1flix.herokuapp.com/)

### How to use the app?

At the above link you will see a page with approx. 500 movies.You need to rate any 4(k) or more movies and press submit at the bottom of the page
to get your recommendations

you can rate the movies by moving the slider thumb to 5 possible positions from (1 to 5) initially all sliders are at 0th position

![alt text](https://raw.githubusercontent.com/asmitks/images/master/123.png)

You will be presented 4 recommendations each of user-user and item-item collaborative learning.
> It is possible that user-user algo doesn't return 4 movies due to its dependency on the data of user ratings.

## Explanation
for explanation behind the algos and a brief report refer to `Report.pdf`

## Code Details
Brief description of folder/files in the repo.

## app.py
The main python3 flask file that contains the main algorithms and logic
#### how to run
```bash
python3 app.py
```
## templates
#### score.html
Main template that is returned as the home page
#### recommend.html
Template that is returned when both algos return answers
#### recommend1.html
Template is returned when user-user algo does not return answer.
## static
contains the css file style.css
## Other
It contains all raw data taken from movie lense.
### mongs.py
Used for creating mongo db database after data cleaning
``` bash
python3 other/mongs.py
```
### scrape.py
Used for scraping thumbnails from dedicated sites of imdb
```bash
python3 other/scrape.py
```
### change.py
Used for printing required tags that need to be inserted into HTML file.
```
bash python3 other/change.py
```
## Dockerfile
Used to run using docker
commands to run using docker
To run the dockerfile:
```bash
docker build -t flask-getflix:latest .
```
To run the app:
```bash
 docker run flask-getflix
```
## How to access mongoDB collections using mongolab.
The mongoDB collections that I used can be accessed thorough
#### imdb database
``` bash
MONGODB_URI='mongodb://asmit:Mama123456@ds043062.mlab.com:43062/imdb'
```
#### data_imdb
``` bash
MONGODB_URI='mongodb://asmit:Mama123456@ds127825.mlab.com:27825/data_imdb'
```


text:
# cs-go-market-watcher
Check if the value of weapons and other market items changes.

**[🔴]VERSION CONTAINS INTENTIONAL BUG! DO NOT USE![🔴]**

#### Edit the weapons and settings in the settings.json file

## How do I get the right weapon name?
Open up the item you want to track in the Steam Market. After that you copy the Name of the weapon from the url as shown below.
![Image of Yaktocat](https://imgur.com/1z375YO.png)

After that you go to https://urldecoder.org and paste the copied text into the first textbox. Then you can get the final name for the weapon from the textbox below. (Shown below)
![Image of Yaktocat](https://imgur.com/nI5EnZR.png)

## Currencies:
-  0: USD
-  2: Pounds
-  3: Euro
-  4: Swiss franc
-  5: Russian ruble
-  6: Złoty
-  7: Brazilian real
-  8: yen
-  9: Danish crown
-  10: Indonesian rupiah
-  11: Malaysian ringgit

repository: Matix-Media/cs-go-market-watcher

repo_name: Matix-Media/cs-go-market-watcher


text:
# Lenguaje de Programación

## Rust

<https://www.rust-lang.org/>

## Rust Foundation

<https://foundation.rust-lang.org/>

Miembros: AWS, Huawei, Google, Microsoft y Mozilla

<https://foundation.rust-lang.org/members/>

### Empresas utilizando Rust

* Mozilla
* Dropbox
* NPM
* Figma
* Microsoft
* Cloudflare
* Facebook
* Braintree
*

"Rust has become increasingly popular as a safer alternative to C and C++ for system development. It's become one of the most favored languages for system development. Rust is even being used now in the Linux operating system."

<https://searchcloudcomputing.techtarget.com/tip/Rust-rises-in-popularity-for-cloud-native-apps-and-environments>

[Computer Scientist proves safety claims of the programming language Rust](https://www.eurekalert.org/news-releases/610682)

[Disertacion](https://people.mpi-sws.org/~jung/thesis.html)

<https://coresumo.com/benefits-of-using-rust-programming-language/>

# C/C++

* MISRA-C++
* JSF C++

~~~
"The use of C++ for safety-critical systems is possible, provided that an appropriate coding
standard such as MISRA-C++ or JSF C++ is followed, and appropriate tools that
automatically check the conformance of all source code are used.  ...
Any programmer intending to write
safety-critical code in C++ should familiarize himself with both coding standards, as they give
a great insight into the potential hazards one has to deal with when using C++ in critical
code."
~~~
<https://www.appinf.com/download/SafetyCriticalC++.pdf>

# Conclusión

Es probable que al fin se implementa la solución en una mescla de lenguajes de programación.


text:
# SFD-640x640-3CH-MobileNetV2-CaffeSSD-Float32-Dense Model

__Input__
+ 640x640
+ 3 channels RGB

__Architecture__
+ MobileNet-V2
+ [MobileNetV2 Architecture - Pretrained Model COCO from tensorflow](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md)
+ the tensorflow weights need to be transformed to caffe. Follw the instructions [here](https://github.com/chuanqi305/MobileNetv2-SSDLite)
+ Features Layers : conv_13/expand, Conv_1, layer_19_2_2, layer_19_2_3, layer_19_2_4

__Training__
+ Caffe SSD
+ Float32
+ Dense
+ Mean values: [104, 117, 123]

__Files__
+ [deploy](deploy.prototxt)
+ [train](train.prototxt)
+ [test](test.prototxt)
+ [trained model 30K iterations](https://drive.google.com/file/d/1ph0mumCcEKXtaupl_TYRc8Q7rWsEt_KD/view?usp=sharing)
+ [trained model 100K iterations](https://drive.google.com/file/d/1t3KQ-gZifDi68zi5Rjy2tYCbZm_-W8hw/view?usp=sharing)

__Results[30K iterations]__
+ Easy: %
+ Medium: %
+ hard: %

__Results[100K iterations]__
+ Easy: %
+ Medium: %
+ hard: %

### Notes

##### Training
+ MobileNetV2 model has too many differences from MobileNetV1, and they use different parameters and different solver for training.
+ The training of the model is very slow, and it takes too much memory. We used batch size of 2 images with iter_size=10, so that at the end the training is done with batch size = 20.
+ To solve the memory issue, these files nedd to be replaced in Caffe src/caffe/layers with these [files](https://github.com/chuanqi305/MobileNetv2-SSDLite/tree/master/src). Don't forget to rebuild caffe again.
+ To start training, the learning rate needs to be very small, otherwise the fluctuations in training loss and validation accuracy will be very big. I found that learning rate 0.00005 is the best depending on my observations for first 3k (validation accuracy is calculated each 1K).
+ During training, the behaviour of the training was normal until ~ 30K, then the fluctuations in validatidation accuracy curve started to increase, and at the end the ccuracy was decreasing!! Next figure shows this behaviour.

![Alt text](SSD640x640-MobileNetV2-CaffeSSD-wider_vlidation_accuracy.png)

+ To solve this problem, we can use the snapshot of the model at 30K iterations (early stopping), to validate this solution we can also test the model at the end (100K iterations) to see the difference.

##### Testing
+ For testing, you should first modify the files in the src/ folder as explained in training notes. This should decrease the memory usage. After that, the testing should be done as usual.
+ I tested the model for WIDERA_FACE, but faced "out of memory" issue after event:25 num:3. Note that we use GPU with 16GB of RAM!
+ I tried to use the deploy model without batch_norm layer to decrease the memory usage, but I got some errors!
+ To be able to test the model, the direct solution is to use GPU with more memory if possible. But also the testing codes could be modified so that it tests each crop of the image separately and free the memory after that.

repository: bonseyes/SFD

gha_language: Python

repo_name: bonseyes/SFD


text:
# Trivia_Game
Multiple Choice JS Quiz
Run against time
5 MCQ about JS are added in the game. Acccording to your response, a random gif shows up.

## How it Looks
![game](https://github.com/bilalsarimeseli/Trivia_Game/blob/master/ASSETS/images/Screen%20Shot%202020-02-24%20at%2011.10.29%20PM.png?raw=true)

## Technologies Utilized
* HTML5
* CSS3
* JavaScript
+ JQuery

## Game Time:
[Trivia](https://bilalsarimeseli.github.io/Trivia_Game/)


text:
---
layout: post
title:  How Scraping NBA Stats is cooler than Michael Jordan
author: Ashwin
categories: [ tutorial , NBA]
image: assets/images/NBA.jpg
imagecaption : "NBA data Scraping"
description: "How to Scrap NBA website for advanced statistic and data"
featured: False
hidden: false
rating:
---
This summer I picked up a new hobby of following the NBA, as a data enthusiast, I wanted to understand how this season of NBA was different on paper from the previous ones as this was played inside the bubble without any fans.

For acquiring relevant data I started by utilizing the python library beautiful soup. However, to my surprise, the data wasn’t stored on the HTML Source page. After some digging, I discovered that the NBA stats website was built with an AngularJS, which means that the site is rendered client-side instead of server-side.

### What is Client-side rendering

The HTML which is rendered is only a template and it doesn’t hold any data, The Javascript in the server response fetches the data from an API and uses it to create the page client-side.

Basically, when you view the pages source code, you wouldn’t find the data but just a template of the webpage.

![[NBA States Website](https://stats.nba.com/players/traditional/?SeasonType=Regular%20Season&sort=PTS&dir=-1&Season=2019-20)](https://cdn-images-1.medium.com/max/2640/1*6ay35LCsJn_9UJr6OuSsHg.png)

![Ctrl + U takes you to the Page source](https://cdn-images-1.medium.com/max/2640/1*pGIksJJN5pAu6wTg2_hVtQ.png)

## Let’s Get Started

In this article, we will be scraping the NBA stats website for the [League player’s stats.](https://stats.nba.com/players/traditional/?SeasonType=Regular%20Season&sort=PTS&dir=-1&Season=2019-20) After hours of researching, I decided to adopt a process that was a lot simpler than Beautiful soup.

### Finding the API endpoint from the website

The first step is to open the webpage you want to scrape on your web browser (preferably google chrome or firefox) and open the developer tools. To do this just right click select inspect.

![Right Click Followed by Inspect](https://cdn-images-1.medium.com/max/2000/1*jG_Fmj6ERa_r_joh1bWceg.jpeg)

This would open a panel either to the right or the bottom of the page, Select network and XHR, and reload the page.

![Inspect panel](https://cdn-images-1.medium.com/max/3600/1*ex45rW_LWZVAG7fu2YehIg.jpeg)

Once we reload the page, all the requests from the page will be visible. At this point, you should do some digging to find the request you want. Most probably the endpoint would be named after the webpage you are looking at.

Since we are looking at the league players’ stats page, the endpoint might be named something similar. Select every option and preview the results to find the correct endpoint.

![Select and Preview](https://cdn-images-1.medium.com/max/3600/1*sRbHnULuPeJjQ2EM0CEGkQ.jpeg)

Once you find the right endpoint you are all set to move to the next step.

### Calling the API endpoint to get the data

In order to call the API, we would make use of the [requests](https://requests.readthedocs.io/en/master/user/quickstart/#custom-headers) python package. In order to do so, We need 3 components as part of the request syntax below.

The first part is the URL, in our case since we are accessing the league player stats, we can get it from the last step.

Under the Header tab, select general and copy the first part of the request URL.

![Request URL](https://cdn-images-1.medium.com/max/3600/1*fxWr9AiFQkdXS6Y3p0eKRw.jpeg)

Next, we need the request headers, Which can also be found under the same Header tab but under “Request Headers” Subsection.

![Request Header](https://cdn-images-1.medium.com/max/3600/1*FWkHO5B8A_0yf9Xwswp_ng.jpeg)

Header as a dictionary

The final component we need is the parameters, which can be found under the “Query String Parameter” subsection under the Header tab.

![Parameters](https://cdn-images-1.medium.com/max/3600/1*JESvDtNXW7-MsbLGyLSkwQ.jpeg)

Parameters

Now that we have all three parts, it’s simple to call the API. The response can then be manipulated into a data frame for analysis.

Get request

The final request will look something like this,

![Dataframe](https://cdn-images-1.medium.com/max/2730/1*HDoJGRdnXbEsrI1PwuDmiw.jpeg)

## Thank You !!

Congratulations 👏 !! we have successfully scraped the NBA stats website.

PS: This process definitely works for [stats.nba.com](https://stats.nba.com/). This might also work for any other website which was built with a client-side web framework using languages such as AngularJS. If the website you are targeting is built with the server-side framework and languages like Django or Ruby on Rails then our friend Beautiful Soup will give you a hand.

Good luck with your web scraping journey! I hope this post is helpful.

Feel free to reach out to me on [Twitter](https://twitter.com/AshwinMuru) or [Linkedin](https://www.linkedin.com/in/ashwinmuthiah/) if you may have any questions.

extension: markdown

gha_language: JavaScript


text:
# Comunica Context Entries

[![npm version](https://badge.fury.io/js/%40comunica%2Fcontext-entries.svg)](https://www.npmjs.com/package/@comunica/context-entries)

A collection of reusable Comunica context key definitions.

This module is part of the [Comunica framework](https://github.com/comunica/comunica),
and should only be used by [developers that want to build their own query engine](https://comunica.dev/docs/modify/).

[Click here if you just want to query with Comunica](https://comunica.dev/docs/query/).

## Install

```bash
$ yarn add @comunica/context-entries
```

## Usage

```typescript
import { KeysInitSparql } from '@comunica/context-entries';

// ...

const baseIRI = context.get(KeysInitSparql.baseIRI);
```

All available keys are available in [`Keys`](https://github.com/comunica/comunica/blob/master/packages/context-entries/lib/Keys.ts).

repository: comunica/comunica

gha_language: TypeScript

gha_license_id: NOASSERTION

repo_name: comunica/comunica


text:
---
title: Are US Citizenship and Immigration Services offices closed due to COVID-19?
category: rumors
layout: post
date: March 30, 2020
source: DHS
promoted: false
source_url: https://www.uscis.gov/news/alerts/uscis-temporary-office-closure-extended-until-least-april-7
excerpt: "Travel: Returning from travel"
---

On March 18, 2020 U.S. Citizenship and Immigration Services temporarily suspended in-person services at its field offices, asylum offices, and Application Support Centers (ASCs) to help slow the spread of Coronavirus (COVID-19).

USCIS offices will reopen on April 7, 2020 unless the public closures are extended further. Employees in these offices are continuing to perform mission-essential services that do not require face-to-face contact with the public.

USCIS will continue to provide limited emergency services.
Please call the contact center for assistance with emergency services.

repository: mayonissen/coronavirus-answers

repo_name: mayonissen/coronavirus-answers


text:
# keptn Architecture v0.2

![Architecture](./res/architecturev0.2.jpg)

keptn will run on Knative, which requires Kubernetes and Istio. The **Event Broker** will be implemented as a Kubernetes Service. We found that event timeouts, e.g. from GitHub webhooks, make the event delivery unreliable when using Knative services that scale up from 0 instances upon receiving an event. We'll continue to investigate this.

The **Channels** will be implemented using Knative Channels.

The **Services** will be implemented using Knative Services that subscribe to channels.

## Event Broker
Listens for and, if necessary, transforms third party events to keptn events and forwards them to the right channel.

## Channels
Keptn provides channels for events in a continuous delivery process. These channels should exclusively work with keptn events. Services can subscribe to channels to receive these events.

## Services
Services subscribe to channels and are triggered by events on said channel. The service executes a specific action based on the event type and its payload. This action can be executed directly in the service, e.g. sending a message to a Slack channel (fire-and-forget). Alternatively, the service sends an event to a third party tool that executes the action and notifies keptn of the outcome.

repository: nnganesha/keptn

repo_name: nnganesha/keptn


text:
# Create A New Branch With Git Switch

As of [Git 2.23](https://www.infoq.com/news/2019/08/git-2-23-switch-restore/),
there is a new command in town for change and creating branches --
`git-switch`.

As a git user, you may be used to using `git checkout -b my_branch` to create
and switch to a new branch called `my_branch`. The `git-checkout` command can
do that and a lot more. In order to reduce some confusion and create a more
explicit command for this kind of action. That's what brought about
`git-switch`.

Create and change to a new branch with `git-switch` and the `-c` flag:

```bash
$ git switch -c my_new_branch
```

The `-c` flag is short for `--create` and the docs describe it as "a convenient
shortcut for:"

```bash
$ git branch <new-branch>
$ git switch <new-branch>
```

See `man git-switch` for more details.

repository: jbranchaud/til

gha_language: Vim Script

repo_name: jbranchaud/til


text:
---
title: Print Theme for Tumblr
description: Print Theme is my first publicly released theme for the Tumblr blogging platform, inspired by the CMYK colour test blocks on printed material
tags: [web, tumblr, coding, html5]
alt: Print Theme for Tumblr blogging platform
---
{% include image.html image="printtheme.png" alt="Print Theme for Tumblr blogging platform" %}

Print Theme is my first publicly released theme for the Tumblr platform. It draws its inspiration from the <abbr title="Cyan Magenta Yellow Key(Black)">CMYK</abbr> colour blocks you see at the edge of printed material. I thought it would appeal to the creative bias of Tumblr users, with a graphical style I had not seen on other themes.

I noticed that many Tumblr themes sacrificed quality markup and accessibility for presentation. To ensure the best experince for all users I included HTML5 semantic markup, accessibility, international language support (and backwards compatibility to older browsers) while putting the content first and allowing theme customisation.

## Theme Features

*	HTML5 semantic markup
*	Typographic grid
*	Custom CSS support
*	Multi-language support
*	@media queries to handle print styles

The theme achieved 200+ installs within its first <time datetime="P7D">7 days</time> available and peaked at **450+ installs**.

For more info about the themes features and customisation options available, see my [Print Theme page](https://paulfosterdesign.wordpress.com/printtheme) for more details. I also posted a follow-up of the [theme progress](https://paulfosterdesign.wordpress.com/2011/07/28/print-theme-update/) and the lessons learned during its development, submission and release. All Print Theme posts are tagged [#printtheme](https://paulfosterdesign.wordpress.com/tag/printtheme/).

Print Theme can be found [on the Tumblr Theme Garden](https://www.tumblr.com/theme/30839).

Update: I have [migrated from Tumblr](https://paulfosterdesign.wordpress.com/2016/12/17/paulfosterdesign-on-wordpress-com/) to a WordPress.com blog of the same username.

gha_language: HTML


text:
# # ReceivingState

## Properties

Name | Type | Description | Notes
------------ | ------------- | ------------- | -------------
**updated_at** | **\DateTime** | The date and time when report was updated last time. Provided in [ISO 8601 format](link: https://en.wikipedia.org/wiki/ISO_8601). | [optional]
**stage** | **string** | Stage of Advance Ship Notice receiving | [optional]
**display_number** | **string** | A human friendly identifier of Advance Ship Notice. | [optional]
**content** | [**\Phobetor\Allegro\Model\ReceivingEntry[]**](ReceivingEntry.md) | The list of products with receiving status. | [optional]

[[Back to Model list]](../../README.md#models) [[Back to API list]](../../README.md#endpoints) [[Back to README]](../../README.md)

repository: Phobetor/allegro-php

repo_name: Phobetor/allegro-php


text:
# Jisho-Anki-Bridge

Extension to streamline the process of moving jisho.org search results to Anki deck cards, since at the time of writing this there does not seem to be any better tools to do this efficently. (Also this works as a nice exercise to learn about chrome extensions since I haven't implemented these before)

Requirements: You need to be signed in ankiweb.net and probably have the latest chrome installed.

repository: Shipuli/short-chrome-extensions

repo_name: Shipuli/short-chrome-extensions


text:
# Additional IP addresses example

This example shows how to add additional IP addresses to the certifcate using the module.

```hcl
terraform {
  required_version = ">= 0.12.31"
  required_providers {
    google = {
      version = "~> 3.69"
    }
  }
}

provider "google" {
  project = var.project_id
  region  = var.project_region
}

data "google_project" "project" {}

resource "google_service_account" "test" {
  account_id = "test-account"
}

resource "google_compute_address" "test" {
  name         = "test-ip"
  address_type = "EXTERNAL"
}

module "tls_cert" {
  source = "devops-rob/tls/gcp"

  project_id            = var.project_id
  region                = var.project_region
  service_account_email = google_service_account.test.email
  tls_bucket            = "test-tls-bucket"
  tls_cert_name         = "devopsrob"

  ip_addresses = [
    google_compute_address.test.address,
    "<IP_ADDRESS>",
  ]
}
```


text:
# Changelog
## 6.0.2 (2018-12-29)
* Updated `lesshint` to `6.3.0`. ([51360ce](https://github.com/lesshint/atom-linter-lesshint/commit/51360ce5cd942ffb6660665e14a86d5eb63f2948))

## 6.0.1 (2018-12-26)
* Updated `lesshint` to `6.2.0`. ([55f90e2](https://github.com/lesshint/atom-linter-lesshint/commit/55f90e2761d5ec43ab01794bd359fd3ee985b151))

## 6.0.0 (2018-12-19)
* Updated `lesshint` to `6.0.0`. ([43dab68](https://github.com/lesshint/atom-linter-lesshint/commit/43dab686a191cd28792ac26efc5793da9b7f47c1))

## 5.0.0 (2018-03-29)
* Updated `lesshint` to `5.0.0`. ([60d8d05](https://github.com/lesshint/atom-linter-lesshint/commit/60d8d05c0326412a039ffd09f7f4fa5483e8fe4e))

## 4.0.2 (2018-02-10)
* Updated `lesshint` to `4.6.4`. ([a70f1c3](https://github.com/lesshint/atom-linter-lesshint/commit/a70f1c35af064f4902598f7d67371ac2814e8dfb))

## 4.0.1 (2017-07-10)
* Updated `lesshint` to `4.0.1`. ([fcd7c40](https://github.com/lesshint/atom-linter-lesshint/commit/fcd7c40018fec1b875cd1f2f6c44c66b31c847c1))

## 4.0.0 (2017-07-05)
* Updated `lesshint` to `4.0.0`. ([2ce02fc](https://github.com/lesshint/atom-linter-lesshint/commit/2ce02fc3a5fbab05d4cfe708eca138aa6b2c80f3))
* Updated messages to the Standard Linter v2 format. ([cdb7595](https://github.com/lesshint/atom-linter-lesshint/commit/cdb7595fcfeb6f325cd231e519f68bada2389cec))
* Updated `eslint` to `4.0.0`. ([17bd20d](https://github.com/lesshint/atom-linter-lesshint/commit/17bd20dbfa5b96d62ed7686cdd109058dca0b63a))
* CI tests are run on Node 8. ([3072d93](https://github.com/lesshint/atom-linter-lesshint/commit/3072d93b8483879e61f9668c76f719a0a402bf65))

## 3.0.3 (2017-05-08)
* Updated `lesshint` to `3.3.1`. ([941ef3b](https://github.com/lesshint/atom-linter-lesshint/commit/941ef3be77f6aab47fb596b10a2996dbf8c4c9df))

## 3.0.2 (2017-04-25)
* Updated `lesshint` to `3.2.0`. ([879e7cf](https://github.com/lesshint/atom-linter-lesshint/commit/879e7cf9b91121822c35a0afe46851994bca603a))

## 3.0.1 (2017-02-16)
* Updated `lesshint` to `3.1.0`. ([762916c](https://github.com/lesshint/atom-linter-lesshint/commit/762916c337c91cb9758db16e5686d4d7124ad487))
* Updated `atom-linter` to `9.0.0`. ([796d70a](https://github.com/lesshint/atom-linter-lesshint/commit/796d70ac06aa64e366cc2bf56db718a8f7014fa1))

## 3.0.0 (2017-02-08)
* Updated `lesshint` to `3.0.0`. ([4b6ac25](https://github.com/lesshint/atom-linter-lesshint/commit/4b6ac25bb8cd27283167741cbc7feb6c16c0fc67))

## 2.2.0 (2016-11-23)
* Added a `globalConfig` option to provide a way to use global configs. ([d2bd9f0](https://github.com/lesshint/atom-linter-lesshint/commit/d2bd9f074144b9ff7735d012b5235251e1e7c729))
* Updated `lesshint` to `2.2.1`. ([6f5e88a](https://github.com/lesshint/atom-linter-lesshint/commit/6f5e88a60755965df5ede35fb8476154e8bfcd95))

## 2.1.0 (2016-10-03)
* Better reporting when `lesshint` can't check a file. ([f12d13f](https://github.com/lesshint/atom-linter-lesshint/commit/f12d13f6c178a760a6b9ed3dfa4f1eb81b5a92a2))
* Updated `lesshint` to `2.1.1`. ([8190277](https://github.com/lesshint/atom-linter-lesshint/commit/81902772585285073ffff1197ff8a4c44ce31207))
* Updated `atom-linter` to `8.0.0`. ([41907de](https://github.com/lesshint/atom-linter-lesshint/commit/41907de94f681965a5884fb348830937aa3441de))

## 2.0.2 (2016-07-31)
* Updated `lesshint` to `2.0.2` to pull in some critical bug fixes. ([1de86be](https://github.com/lesshint/atom-linter-lesshint/commit/1de86beb19f7c1cacb625b0a97dbba09903d62de))
* Updated `atom-linter` to `7.0.0`. ([4791fcf](https://github.com/lesshint/atom-linter-lesshint/commit/4791fcf9f39ce51e8212219a9874d3220d69fbce))

## 2.0.1 (2016-07-14)
* Fixed an issue where the plugin would crash on incomplete Less code. ([5868445](https://github.com/lesshint/atom-linter-lesshint/commit/586844511796d87cd0bab25382050fec945c770b))
* Updated specs for new `atom-linter` behaviour. ([f32b4f7](https://github.com/lesshint/atom-linter-lesshint/commit/f32b4f725b6ea472e3901696c5123428c697d9e6))

## 2.0.0 (2016-07-14)
* Updated `lesshint` to `2.0.0`. ([014e718](https://github.com/lesshint/atom-linter-lesshint/commit/014e7189628bd2cd6f78dbe79992a65f381723cb))
* Updated `atom-linter` to `6.0.0`. ([c5198f9](https://github.com/lesshint/atom-linter-lesshint/commit/c5198f9f196018189565c982eee01c955337d88c))
* Updated `eslint` to `3.0.1`. ([529ed73](https://github.com/lesshint/atom-linter-lesshint/commit/529ed7323d1b28f78fb832cce5adaaaecde84ae9))

## 1.5.0 (2016-05-12)
* Added specs, linting, and CI configurations. ([7eba5f7](https://github.com/lesshint/atom-linter-lesshint/commit/7eba5f7a634dbe94c3f28be25ea2c8522dcc8b03), [31c4e13](https://github.com/lesshint/atom-linter-lesshint/commit/31c4e1331d4f4bc5089cfea7190dfb5ca8dc6e39), [20d5901](https://github.com/lesshint/atom-linter-lesshint/commit/20d5901c023296c3820c3afedae00305f125679e))
* Bumped `lesshint` to `1.5.2`. ([0f4e79a](https://github.com/lesshint/atom-linter-lesshint/commit/0f4e79a7a4b7b649b023a690a457543a9d952532))
* Updated `atom-linter` to `5.x`. ([f48eecc](https://github.com/lesshint/atom-linter-lesshint/commit/f48eeccb955b3f6744648b5d9daba5fb20faec8c))

## 1.4.0 (2016-03-12)
* Updated `atom-linter` and `atom-package-deps` to `4.x` and utilizing new helper methods. ([b9cc24d](https://github.com/lesshint/atom-linter-lesshint/commit/b9cc24deedf5893571772fa169255cf4379a2cad))
* Fixed an issue where the correct column wasn't always reported. ([df59a07](https://github.com/lesshint/atom-linter-lesshint/commit/df59a0798e908d534fb91a8db6c8d60bca7e853a))

## 1.3.0 (2015-12-14)
* Updated to `lesshint` `1.2.1`. ([7301783](https://github.com/lesshint/atom-linter-lesshint/commit/73017834b87e8c52aab7d7309ef7b640554505b3))

## 1.2.0 (2015-10-26)
* Added the `name` property to the linter so it's shown in result output. ([26d108a](https://github.com/lesshint/atom-linter-lesshint/commit/26d108a9a0d4ddc0caba7d3196b6dafefb5b1b0b))
* Made sure that `lesshint` is `require()`d just once instead of on every `lint` call. ([60172c8](https://github.com/lesshint/atom-linter-lesshint/commit/60172c8478bf6e668f9acd8040bb3ce0ca1ea1e1))

## 1.1.1 (2015-10-23)
* Bump for new `lesshint` version with some critical bug fixes. ([7a118ca](https://github.com/lesshint/atom-linter-lesshint/commit/7a118cae32aef8fbfa321afb289b560679cab0bd))

## 1.1.0 (2015-10-20)
* Updated to `lesshint` `1.0.0`. ([37410b1](https://github.com/lesshint/atom-linter-lesshint/commit/37410b137c4f640692a811f5a8c18bbabe0532c1))

## 1.0.0 (2015-09-23)
* Initial release. ([1bb3a9f](https://github.com/lesshint/atom-linter-lesshint/commit/1bb3a9f8c8dcc79c49755500d3cf0385868b9db3))

repository: conradoTCK/atom-linter-lesshint

repo_name: conradoTCK/atom-linter-lesshint


text:
=====
1pass
=====

A command line interface (and Python library) for reading passwords from
[1Password](https://agilebits.com/onepassword).

Command line usage
==================

To get a password::

    1pass unlock mail.google.com

By default this will look in ``~/Dropbox/1Password.agilekeychain``. If that's
not where you keep your keychain::

    1pass unlock --path ~/whatever/1Password.agilekeychain mail.google.com

Or, you can set your keychain path as an enviornment variable::

    export ONEPASSWORD_KEYCHAIN=/path/to/keychain

    1pass unlock mail.google.com

By default, the name you pass on the command line must match the name of an
item in your 1Password keychain exactly. To avoid this, fuzzy matching is
made possible with the ``--fuzzy`` flag::

    1pass unlock --fuzzy mail.goog

If you don't want to be prompted for your password, you can use the
``--no-prompt`` flag and provide the password via standard input instead::

    emit_master_password | 1pass unlock --no-prompt mail.google.com

You can list the items you have in your keychain by using the subcommand `list`::

    1pass list

Python usage
============

The interface is very simple::

    from onepassword import Keychain

    my_keychain = Keychain(path="~/Dropbox/1Password.agilekeychain")
    my_keychain.unlock("my-master-password")
    my_keychain.item("An item's name").password

An example of real-world use
============================

I wrote this so I could add the following line to my ``.muttrc`` file::

    set imap_pass = "`1pass unlock 'Google: personal'`"

Now, whenever I start ``mutt``, I am prompted for my 1Password Master Password
and not my Gmail password.

The ``--no-prompt`` flag is very useful when configuring ``mutt`` and PGP.
``mutt`` passes the PGP passphrase via standard in, so by inserting ``1pass``
into this pipline I can use my 1Password master password when prompted for my
PGP keyphrase::

    set pgp_decrypt_command="1pass unlock --no-prompt pgp-passphrase | gpg --passphrase-fd 0 ..."

Contributors
============

* George Brocklehurst <https://github.com/georgebrock>
* Pip Taylor <https://github.com/pipt>
* Adam Coddington <https://github.com/latestrevision>
* Ash Berlin <https://github.com/ashb>
* Zach Allaun <https://github.com/zachallaun>
* Eric Mika <https://github.com/kitschpatrol>
* George Hickman <https://github.com/ghickman>
* Frazer McLean <https://github.com/RazerM>
* Rafael Caricio <https://github.com/rafaelcaricio>

License
=======

*1pass* is licensed under the MIT license. See the license file for details.

While it is designed to read ``.agilekeychain`` bundles created by 1Password,
*1pass* is **not** officially sanctioned or supported by
[AgileBits](https://agilebits.com/). I do hope they like it though.


text:
# CHANGE LOG

## 20210214

* Converted repo from `https` (deprecated) to `ssh` for version control
* modified names of `.sol` contract files and corresponding test `js` files
* Migration script - changed names of contracts, added more dummy addresses
* Package.json - bumped OpenZep and Truffle HD
* Truffle config - v6, changes to gas limit
* Created CHANGELOG file
* Minor edits to `README` and `.gitignore`

repository: fuguefoundation/ff-contracts

gha_language: Solidity

repo_name: fuguefoundation/ff-contracts