text:
# SignalFx Agent Ansible Role

This is the `signalfx-agent` Ansible role that will install and configure the
SignalFx Smart Agent on your remote hosts.  To install it, you can clone the
agent repo to your controller and add the `signalfx-agent` role directory to
its `roles_path` in your `ansible.cfg`, or use this document's directory as
your working directory.  The role is also available via Ansible Galaxy:

```
ansible-galaxy install signalfx.smart_agent
```

To use this role, simply include the `signalfx-agent` role invocation in your
playbook (or `signalfx.smart_agent` if installed via Galaxy).  An
ansible-playbook call using the provided example playbook and variable files
would be similar to:

```
ansible-playbook -i your_inventory_file -e @example-config.yml example-playbook.yml
```

This role sources the following variables:

 - `sfx_agent_config`: A mapping that gets converted to the Smart Agent
   configuration YAML file. See the [Agent Config
   Schema](https://github.com/signalfx/signalfx-agent/blob/main/docs/config-schema.md)
   for a full list of acceptable options and their default values.  The only
   required key-value pair is `signalFxAccessToken`.

    Here is a basic `sfx_agent_config` that will monitor a basic set of host-level components:

    ```yaml
    sfx_agent_config:
      signalFxAccessToken: MY-TOKEN  # Required
      monitors:
        - type: cpu
        - type: filesystems
        - type: disk-io
        - type: net-io
        - type: load
        - type: memory
        - type: vmem
        - type: host-metadata
        - type: processlist
    ```

	It is suggested to keep this mapping in a variable file in the respective
	`group_vars` or `host_vars` directory for your target remote hosts, or to
	pass it in via the `-e @path/to/variable_file` ansible-playbook extra vars
	option for a "global" configuration.

 - `sfx_conf_file_path`: The target path for the Smart Agent configuration
   file generated from `sfx_agent_config` (**default:**
   '/etc/signalfx/agent.yaml'). See `sfx_win_conf_file_path` for Windows platforms.

 - `sfx_repo_base_url`: The url provided to yum/apt for obtaining the SignalFx Smart Agent
   (**default:** `https://splunk.jfrog.io/splunk`)

 - `sfx_package_stage`: The package repo stage to use: `release`, `beta`, or `test`
   (**default:** 'release')

 - `sfx_version`: The agent package version.  This is of the form `<agent
   version>-<package revision>` (e.g. package version `3.0.1-1` is the first
   package revision that contains the agent version `3.0.1`).  Releases with
   package revision > 1 contain changes to some aspect of the packaging scripts
   (e.g. init scripts) but contain the same agent bundle. (**default:**
   'latest')

 - `sfx_service_user` and `sfx_service_group`: Set the user/group ownership for the
   signalfx-agent service. The user/group will be created if they do not exist.
   Requires agent package version 5.1.0 or newer. (**default:** 'signalfx-agent'). Not used
   in Windows platforms.

 - `sfx_service_state`: Set the state of signalfx-agent service. (**default:** 'started')

 - `sfx_win_install_path`: The Windows installation path. (**default:**
   'C:\\Program Files\\SignalFxAgent')

 - `sfx_win_conf_file_path`: The Windows agent configuration file path. (**default:**
   '\\ProgramData\\SignalFxAgent\\agent.yaml')

 - `sfx_win_dl_base_url`: The base URL for Windows agent downloading (**default:**
   'https://dl.signalfx.com')

 - `sfx_skip_repo` (Linux only): If installing the agent from a custom or self-hosted
   apt/yum repo, set to `true` to skip the installation of the default repo
   (**default:** `false`)

**Note**: On Linux, after the `signalfx-agent` role is deployed, Ansible will manage the
`signalfx-agent` service via the Ansible core `service` module.  This module
will automatically determine the host's init system for starting/stopping the
`signalfx-agent` service, with a preference for systemd (`systemctl`).

repository: signalfx/signalfx-agent

branch_name: refs/heads/main

repo_name: signalfx/signalfx-agent


text:
# Overview
This is the sample for the [Azure DSC Extension 2.8 & How to map downloads of the extension dependencies to your own location.](http://blogs.msdn.com/b/powershell/archive/2015/10/21/azure-dsc-extension-2-2-amp-how-to-map-downloads-of-the-extension-dependencies-to-your-own-location.aspx)

# How to use the sample
* Download all the files in the folder to the same folder
* Update the settings near the end of `DownloadMappings.ps1`
* Run `DownloadMappings.ps1`

license: MIT

branch_name: refs/heads/master

extension: md

license_type: permissive


text:
---
layout: shared/narrow
title: "On the server: sending notifications"
description: "A service running on a server is responsible for taking the data provided in the push message and actually forwarding it to the user as a notification."
authors:
  - dgash
published_on: 2015-10-01
updated_on: 2015-10-01
order: 7
---

<p class="intro">
  A service running on the server is responsible for taking the
  data provided in the push message and actually forwarding it to the user as a notification.
</p>

{% include shared/toc.liquid %}

The [GCM documentation](https://developer.android.com/google/gcm/index.html)
has a reference for the [HTTP syntax](https://developers.google.com/cloud-messaging/http-server-ref)
used to pass messages from your app server to client apps. The
[XMPP server protocol](https://developers.google.com/cloud-messaging/xmpp-server-ref)
serves a similar purpose.

## Message content

In the service worker's push handler, code is executed that establishes
the message information, including the `title`, `body`, `icon`, and `tag`. It
is the service worker's job to take the pushed data and turn it into a
sendable notification.

{% highlight javascript %}
self.addEventListener('push', function(event) {
  console.log('Push message', event);
  var title = 'Push message';
  event.waitUntil(
    self.registration.showNotification(title, {
      body: 'The Message',
      icon: 'images/icon.png',
      tag: 'my-tag'
    })
  );
});
{% endhighlight %}

## Notification test

[Earlier](notifications), we discussed creating a project in Google Developer
Console, and retaining some information for later use in the manifest file.
That information is also useful in a simple notification test.

For a quick check of your service worker, you can use
[cURL](https://en.wikipedia.org/wiki/CURL) to send a push message to your
browser, provided that you have whitelisted your IP address for your local
machine in the Google Developer Console.

extension: markdown


text:
# 1.0.1 (August 10, 2018)

## Bug fixes / Improvements

* [[`dcbef27350`](https://github.com/webhintio/hint/commit/dcbef27350558eb17980a3c494c07ec1d03a2454)] - Docs: Add missing commas (by [`Bruno Vinicius Figueiredo dos Santos`](https://github.com/IAmHopp)).

# 1.0.0 (August 6, 2018)

✨

repository: jamierytlewski/hint

gha_language: TypeScript

repo_name: jamierytlewski/hint


text:
---
title: Secure web applications with docker container
date: 2017-01-06 22:10:00 +0000
categories: [technology]
tags: [docker]
author: Chun
---

Secure Web Applications with Docker Container
============================================

Many open source web applications doesn't provide out-of-the-box authentication and authorization features. For example, [Elasticsearch](https://www.elastic.co/products/elasticsearch) expose HTTP service on port 9200/9300 by default and allows anyone to access the service through these ports.

Traditional solutions to this problem include:
- Implement cusomized code within the application to support authentication and authorization.
- Update firewall configuration to block remote access to the public ports, and put a reverse proxy with authentication and authorization over the application.

These approaches both involve a fair amount of work and are intrusive/not flexible.

With Docker, we now have more elegant way to address security requirements of web applications in a none intrusive / resuable manner:

> Run web application with Docker container in private virtual network.
> Expose service through port mapping on a reverse proxy container.

![Docker Network](https://docs.docker.com/engine/userguide/networking/images/network_access.png)

Let's take a step by step look at this approach.

Prerequisites:
-------------

- A linux VM which has Docker engine installed and access to internet
- Able to download Docker images from DockerHub
- An Github account (for using Github OAuth feature)

Create named docker network
---------------------------

Docker use to provide `--link` option for connecting one container to another, allowing applications running inside the 2 containers to communicate with each other through network. With recent upgrade, this feature is now replaced by the **user-defined network**. Hence, let's first create a named network in which we will run all the collaborating containers in his example.

```
$ docker network create --driver bridge dockernetwork1
f72b24259f4a773f43a664e2f30a2283e6a97405ecc4e38aa72a0ec727cab2c4

$ docker network inspect dockernetwork1
[
    {
        "Name": "dockernetwork1",
        "Id": "f72b24259f4a773f43a664e2f30a2283e6a97405ecc4e38aa72a0ec727cab2c4",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "<IP_ADDRESS>/16",
                    "Gateway": "<IP_ADDRESS>"
                }
            ]
        },
        "Internal": false,
        "Containers": {},
        "Options": {},
        "Labels": {}
    }
]
```
Here we are using the `bridge` driver which allows port mapping to the docker host's physical network.

Run target application in Docker network
----------------------------------------

[portainer](http://portainer.io/) is a web based docker management tool. It exposes HTTP service on default port 9000. We are going to use it as an example and see how we can protect it from public access.


text:
# lite-id
[![Build Status](https://travis-ci.org/imatveev/lite-id.svg?branch=master)](https://travis-ci.org/imatveev/lite-id)<br />
A simple nodejs module for generate unique string id.
## Installation
    npm install lite-id
## When should I use lite-id?
You need to generate a simple unique id in code.
## API
    var id = require('lite-id');
    console.log(id());
    //Result: J6YgVpwp1929
### id([length])
Generates an id. Optional [length] is set resulting id length. Default length is 10.
## Tests

Lite-id have a mocha test to find collisions on generated id's. Test for 3,000,000 unique id's with default length
and consist is passed and processed without conflicts.

    npm test //standart test with 60K instances
    npm run extended_test //extended test with 3M instances
## Examples

With default length

    var id = require('lite-id');
    console.log(id());
    //Result: J6YgVpwp1929

With custom length

    var id = require('lite-id');
    console.log(id(20));
    //Result: jXf315M_G10SQKa2786RG

repository: imatveev/lite-id

gha_language: JavaScript

repo_name: imatveev/lite-id


text:
**MIT License**

To build on GNU/Linux install developer packages for SDL2. You can find corresponding libraries in `CmakeLists.txt`.
Create build folder and run next commands from it:

```
cmake ../source/
make
```

Run game with `./Tetris`
___

repository: menteralgo/tetris

repo_name: menteralgo/tetris


text:
# Tools
If you want to process 3D data, [Meshlab](http://www.meshlab.net/) and [Blender](https://www.blender.org/) is your good friend.

1. obj2ply_vc_convertor.py
=> Convert sequence .obj  to .ply and with the vertex color info.
      ex: python obj2ply_vc_convertor.py -- -i path/to/obj_folder -o path/to/ply_folder -m path/to/Transfer_texture_to_Vertex_Color.mlx

      Don't forget to change the meshlabServer path to your install path

====================================================================================================

repository: BigTsung/Tools

repo_name: BigTsung/Tools


text:
## 2.4.4 (15-06-2020)
  * Add explicit support for net48 and netcoreapp3.1
## 2.4.3 (24-12-2018)
  * Allow longer console input than the default 254 character limitation
## 2.4.2 (08-11-2017)
  * Allow TimeSpan arguments
## 2.4.1 (26-10-2017)
  * Navigatable commands in command line mode now get executed
## 2.4.0 (06-09-2017)
  * Update to netstandard20
## 2.3.2 (15-08-2017)
  * Add inner exception stack trace when executing command
## 2.3.1 (15-08-2017)
  * Ignore get/set methods on public type scanning
## 2.3.0 (15-08-2017)
  * Methods with similar call structure are now fixed (#2)
  * Auto up after commands is now available (#3)
  * Async void commands will show an error message and won't execute
  * Ability to scan public types, allowing modes where dependant libraries won't need a reference to commanr runner to still offer commands (#7)

repository: janpieterz/CommandRunner

gha_language: C#

gha_license_id: MIT

repo_name: janpieterz/CommandRunner


text:
# 1134. Find Duplicate File in System

**Description**

Given a list of directory info including directory path, and all the files with contents in this directory, you need to find out all the groups of duplicate files in the file system in terms of their paths.

A group of duplicate files consists of at least two files that have exactly the same content.

A single directory info string in the input list has the following format:

`"root/d1/d2/.../dm f1.txt(f1_content) f2.txt(f2_content) ... fn.txt(fn_content)"`

It means there are `n` files (`f1.txt`, `f2.txt` ... `fn.txt` with content `f1_content`, `f2_content` ... `fn_content`, respectively) in directory `root/d1/d2/.../dm`. Note that `n >= 1` and `m >= 0`. If `m = 0`, it means the directory is just the root directory.

The output is a list of group of duplicate file paths. For each group, it contains all the file paths of the files that have the same content. A file path is a string that has the following format:

`"directory_path/file_name.txt"`

```
1.No order is required for the final output.
2.You may assume the directory name, file name and file content only has letters and digits, and the length of file content is in the range of [1,50].
3.The number of files given is in the range of [1,20000].
4.You may assume no files or directories share the same name in the same directory.
5.You may assume each given directory info represents a unique directory. Directory path and file info are separated by a single blank space.
```

**Example**

Example 1

```
Input:
["root/a 1.txt(abcd) 2.txt(efgh)", "root/c 3.txt(abcd)", "root/c/d 4.txt(efgh)", "root 4.txt(efgh)"]
Output:
[["root/a/2.txt","root/c/d/4.txt","root/4.txt"],["root/a/1.txt","root/c/3.txt"]]
```

Example 2

```
Input:
["root/a 1.txt(abcd) 2.txt(efgh)"]
Output:
[]
```

**Challenge**

1. Imagine you are given a real file system, how will you search files? DFS or BFS?
2. If the file content is very large (GB level), how will you modify your solution?
3. If you can only read the file by 1kb each time, how will you modify your solution?
4. What is the time complexity of your modified solution? What is the most time-consuming part and memory consuming part of it? How to optimize?
5. How to make sure the duplicated files you find are not false positive?

https://www.lintcode.com/problem/find-duplicate-file-in-system/note/192626

```
Imagine you are given a real file system, how will you search files? DFS or BFS ?
In general, BFS will use more memory then DFS. However BFS can take advantage of the locality of files in inside directories, and therefore will probably be faster

If the file content is very large (GB level), how will you modify your solution?
In a real life solution we will not hash the entire file content, since it’s not practical. Instead we will first map all the files according to size. Files with different sizes are guaranteed to be different. We will than hash a small part of the files with equal sizes (using MD5 for example). Only if the md5 is the same, we will compare the files byte by byte

If you can only read the file by 1kb each time, how will you modify your solution?
This won't change the solution. We can create the hash from the 1kb chunks, and then read the entire file if a full byte by byte comparison is required.

What is the time complexity of your modified solution? What is the most time consuming part and memory consuming part of it? How to optimize?
Time complexity is O(n^2 * k) since in worse case we might need to compare every file to all others. k is the file size

How to make sure the duplicated files you find are not false positive?
We will use several filters to compare: File size, Hash and byte by byte comparisons.
```

```python
from collections import defaultdict

class Solution:
    """
    @param paths: a list of string
    @return: all the groups of duplicate files in the file system in terms of their paths
    """
    def findDuplicate(self, paths):
        # Write your code here
        if not paths or len(paths) == 0:
            return []

        res = []
        # map from content to directory
        mapping = defaultdict(list)

        for path in paths:
            # path = "root/a 1.txt(abcd) 2.txt(efgh)"
            splitted_path = path.split(' ')
            prefix_path = splitted_path[0]
            for idx in range(1, len(splitted_path)):
                file_name, content = splitted_path[idx].split('(')
                content = content[:len(content) - 1] # remove ')'

                mapping[content].append(prefix_path + "/" + file_name)

        for _, filelist in mapping.items():
            if len(filelist) > 1:
                res.append(filelist)

        return res
```

repository: jxhangithub/lintcode

path: /Interviews/goldmansachs/1134.Find Duplicate File in System/README.md

repo_name: jxhangithub/lintcode


text:
## How to contribute to Spokestack Android
Interested in contributing to Spokestack Android? We appreciate all kinds of help!

### Pull Requests
We gladly welcome pull requests.

Before making any changes, we recommend opening an issue (if it doesn’t
already exist) and discussing your proposed changes. This will let us give
you advice on the proposed changes. If the changes are minor, then feel free
to make them without discussion.

### Development Process
To make changes, fork the repo. Write code in this fork, ensuring it passes the following
checks (it might be useful to put these in a pre-commit hook with `|| exit 1`
at the end of each line):

```bash
mvn checkstyle:check
mvn verify
```

These checks are performed automatically by CI whenever
a branch is pushed and must pass before a PR will be
merged.

#### Test Coverage
We strive to keep test coverage as high as possible using `jacoco`. The `pom.xml`
file includes a minumum coverage ratio; the previously mentioned checks will not
pass unless this minimum is met.

#### Native Code
This package requires the [Android NDK](https://developer.android.com/ndk/guides/index.html)
to be installed and the `ANDROID_HOME` and `ANDROID_NDK_HOME` variables to be
set. On OSX, `ANDROID_HOME` is usually set to `~/Library/Android/sdk` and
`ANDROID_NDK_HOME` is usually set to `~/Library/Android/sdk/ndk/<version>`.

`ANDROID_NDK_HOME` can also be specified in your local Maven `settings.xml` file
as the `android.ndk.path` property.

repository: spokestack/spokestack-android

gha_language: Java

repo_name: spokestack/spokestack-android


text:
---
templateKey: strategy-post
title: Value Proposition Canvas
date: 2021-03-14T17:41:43.943Z
description: >
  The Value Proposition Canvas is a quick and structured way to note your
  customer’s needs, and compare them to the values your product is expected to
  create.

  Through the Value Proposition Canvas, you should have greater visibility on how well your product features map to your customer’s pain-points and nice-to-haves, providing you with an indication of how valuable your product will be to users.
link: https://docs.google.com/presentation/d/1k6xB4cPPHMIy_U5Lzw-lSyI75Go_vgN3hWCinU9Kxvk/edit#slide=id.gb70c0c9545_0_165
tags:
  - GTM
---

1. Start on the right side of the canvas by noting all the jobs your customer’s need to satisfy / solve under “Customer Jobs”.
2. Given the noted jobs, derive pain-points they currently face and note them under “Pains”.
3. Similarly, derive nice-to-haves (experiences that make them happy) and note them under “Gains”.
4. When done, move to the left side of the canvas and note your planned products / services under “Products & Services”.
5. Next, consider how your products solve customer pains and note them under “Pain Relievers”.
6. Finally, consider how your products create customer gains and note them under “Gain Creators”.


text:
---
layout: post
status: publish
published: true
title: Postfix+Amavis+ClamAV+SpamAssassin setup in FreeBSD
date: 2007-11-12 01:16:00.000000000 +01:00
categories:
- hacks
tags: []
---
A friend's sysadmin went missing so was left with a broken mail server that did not deliver anything. There are plenty of articles on doing this in Linux, the challenge was doing it in FreeBSD 5.4, when the latest stable version is 6.2.

An excellent first reading:

[Linux Journal - Adding Clam Antivirus to Your Postfix Server](http://www.linuxjournal.com/article/7778)

Although an article for Linux, it explains well the setup steps that need to be done. I applied similar steps in the FreeBSD system, with the differences below.

1. Stopped everything first.

```
/usr/local/etc/rc.d/postfix.sh stop
/usr/local/etc/rc.d/amavisd.sh stop
```

2. Saved a backup of `/etc/amavisd.conf`, copied `/usr/local/etc/amavisd.conf` over it, and edited the file according to the article and the original file. (Created users, directories, changed permissions, etc.)

3. Saved a backup of `/usr/local/etc/clamd.conf`. Actually, upon configuring ClamAV it turned out that the version is just too old. When starting up, it warned that not only the virus database needs to be up to date, but the software itself too. Reluctantly, I downloaded the latest source from here:

[http://www.clamav.net/download/sources](http://www.clamav.net/download/sources)

The build went through without issue. Before installing, had to remove the ClamAV FreeBSD package with `pkg_deinstall` or `pkg_delete`. BUT! Before you do that, make sure you save the rc script, as the source tarball does not have it.

After the build, configured according to the article and the original file. (Created users, directories, changed permissions, etc.)

4. With the above setup, things were still not perfect, ClamAV's logs had frequent errors about SpamAssassin, and spam was not filtered well. Even though it was the latest SpamAssassin package for the system, it just wasn't quite good enough. So had to install this from source too, from CPAN:

[http://search.cpan.org/](http://search.cpan.org/)

5. With all the above setup, started running the services, the order doesn't really matter. This should be done after system startup:

```
/usr/local/etc/rc.d/amavisd.sh start
/usr/local/etc/rc.d/clamav-clamd.sh start
/usr/local/etc/rc.d/clamav-freshclam.sh start
```

To have these automatically started, look at the individual rc scripts, they mention what variable needs to be set in `/etc/rc.conf` to have them automatically start.

With the above setup, `postqueue -p` showed decreasing number of unprocessed messages, from over 1500 down to just dozens. My friend confirmed proper mail delivery and spam filtering.

To conclude, things were setup the right way, and not deviating much from the "FreeBSD way". ClamAV just *had* to be setup from source, there was no other way. SpamAssassin likewise, being a dependency of ClamAV. Amavis and Postfix are still from the original FreeBSD packages, but they work fine so are not a problem right now.

repository: janosgyerik/homepage

branch_name: refs/heads/gh-pages

gha_language: HTML

repo_name: janosgyerik/homepage


text:
# Flask_Face_liveness_Detectioin_Webapp
Simple Facial Recognition Web app based on flask framework and OpenCV.

### Prerequisites:
- Python3.6

- OpenCV

- Flask

### Installation

* Open Terminal Using ctrl+alt+t  Then Type ` $ git clone https://github.com/data-peace/datapeace-junkyard.git ` .

* Get Into Web App Directory
```sh
$ cd  datapeace-junkyard/projects/face_spoofing_demo/
```

* Make Virtual Envirment
```sh
$ python3.6 -m venv .venv
```
*  Activate Virtual Enviroment
```sh
$ source .venv/bin/activate
```

* Install requirements
```sh
$ pip install -r requirements.txt
```

# Heroku Deployment
`Using Heroku CLI`
`Aptfile`
`Procfile`
`supervisor.conf`
```sh
$ git init
$ git add .
$ git commit -m "commit changes"
$ heroku login
$ heroku apps  <choose Your app >
$ heroku git:remote -a <appname>
$ heroku create face-spoofing-demo <appname>
$ git push heroku master
```

# Using Supervisor Utility
- ` gunicorn and supervisor is succesfulle installed in virtual enviroment`
- ` add app root in .env `
```sh
$ supervisord -c supervisor.conf
```
- ` Use supervisorctl to control server `
```sh
$ supervisorctl -c supervisor.conf
$ supervisor > status
`It Shows Cureent running app status`
$ supervisor > stop app
`to stop running app by name`
$ supervisor > exit
```

# Manual Run
- `All Requirements Are Succesfully Installed  in Virtual Eniviromnet `

* Running Server
```sh
$python3.6 app.py
```

- ` Now Flask Server start on Localhost With Url [http://localhost:5555/] `

# Feature

* For Webcam Capturing Visit

- `http://localhost:5555/`

* For Image Uplod Visit

- `http://localhost:5555/image/`

# click

[Face Detection]https://face-spoofing-demo.herokuapp.com/

repository: bytesbit/Face-Detection-

repo_name: bytesbit/Face-Detection-


text:
---
layout: post
title: 'Blood and Honey'
categories:
 - balkans
---

If you were a little overwhelmed with yesterday's Bosnian linkfest, do at least check out this site. Amazing photos from a journalist who was there during the entire war.

the site for the book, Blood and Honey (on photoarts)

the book (I just bought the 9 clam used copy, sorry)

"First I was a Yugoslav. Then, I was a Bosnian. Now I'm becoming a Muslim. It's not my choice. I don't even believe in God. But after two hundred thousand dead, what do you want me to do?" --A resident of Sarajevo

I was/am profoundly impacted just by this website. This photographer captured more than words could possibly say in just a few images. I do hope you take the time to explore this photoessay and to think about what happened in Bosnia, and about the effects of war, "just" or no.

repository: dealingwith/poole

gha_language: CSS

repo_name: dealingwith/poole


text:
# SkyDB

SkyDB is a framework that allows Skynet users to create decentralized accounts
and store mutable files in those accounts. It is built on top of the [Registry](#registry), where every SkyDB entry in the Registry is a skylink that contains the full data.

<aside class="warning">
These functions have only been implemented for Browser JS at the moment.
</aside>

## Getting Data From SkyDB

```shell--curl
# There are no API endpoints for SkyDB -- you must use the registry directly.
```

```javascript--browser
import { SkynetClient, genKeyPairFromSeed } from "skynet-js";

const client = new SkynetClient();
const { publicKey } = genKeyPairFromSeed("this seed should be fairly long for security");

const dataKey = "myApp";

async function getJSONExample() {
  try {
    const { data, revision } = await client.db.getJSON(publicKey, dataKey);
  } catch (error) {
    console.log(error);
  }
}
```

### Function

`getJSON`

### Parameters

Field | Type | Description
----- | ---- | -----------
`publicKey` | `string` | User's public key as a hex-encoded string. Can be generated with the `genKeyPairFromSeed` function.
`dataKey` | `string` | The key of the data to fetch for the given user.

### Response

```javascript-browser
{
  data: {
    example: "This is some example JSON data."
  },
  revision: 11
}
```

## Setting Data On SkyDB

```shell--curl
# There are no API endpoints for SkyDB -- you must use the registry directly.
```

```javascript--browser
import { SkynetClient, genKeyPairFromSeed } from "skynet-js";

const client = new SkynetClient();
const { privateKey } = genKeyPairFromSeed("this seed should be fairly long for security");

const dataKey = "myApp";
const json = { example: "This is some example JSON data." };

async function setJSONExample() {
  try {
    await client.db.setJSON(privateKey, dataKey, json);
  } catch (error) {
    console.log(error);
  }
}
```

### Function

`setJSON`

### Parameters

Field | Type | Description
----- | ---- | -----------
`privateKey` | `string` | User's private key as a hex-encoded string. Can be generated with the `genKeyPairFromSeed` function.
`dataKey` | `string` | The key of the data to fetch for the given user.
`json` | `object` | The JSON object to set for the given private key and data key.

### Response

Empty on success.

## Getting The Entry URL

```javascript--browser
import { SkynetClient, genKeyPairFromSeed } from "skynet-js";

const client = new SkynetClient();
const { publicKey } = genKeyPairFromSeed("this seed should be fairly long for security");

const dataKey = "foo";

function getEntryExample() {
  try {
    const url = client.registry.getEntryUrl(publicKey, dataKey);
  } catch (error) {
    console.log(error);
  }
}
```

### Function

`getEntryUrl`

### Parameters

See [Getting Data From The Registry](#getting-data-from-the-registry).

### Response

```javascript-browser
"https://siasky.net/skynet/registry?publickey=ed25519%3Ac1197e1275fbf570d21dde01a00af83ed4a743d1884e4a09cebce0dd21ae254c&datakey=7c96a0537ab2aaac9cfe0eca217732f4e10791625b4ab4c17e4d91c8078713b9"
```

repository: NebulousLabs/skynet-docs

repo_name: NebulousLabs/skynet-docs


text:
+++
author = "Göran Svensson"
date = "2016-03-09T14:28:30Z"
description = "Cloudinary is a cloud service that offers a solution to a web application’s entire image management pipeline. Easily upload images to the cloud. Automatically perform smart image resizing, cropping and conversion without installing any complex software."
draft = false
keywords = ["Digital Asset Management", "Cloudinary", "Appernetic Cloudinary setup", "static website"]
tags = ["Cloudinary", "Digital Asset Management", "Appernetic Service","Appernetic Static Site Generator", "Help", "static website"]
title = "Manage your web site images in the cloud"
topics = ["How-to"]
type = "post"

+++

## Cloudinary for image and video assets management in your static website
We no longer use GitHub to manage images in your Appernetic static website service account. You can still use GitHub and they have an easy to use image upload, but to use Appernetic's image upload you now have to first create a Cloudinary account. The advantages of using Cloudinary is so big that it was an easy decision to integrate it. Images are seamlessly delivered through Akamai's fast CDN. By not bogging down GitHub with large images you are also being a good netizen.

Cloudinary is a cloud service that offers a solution to a web application's entire image management pipeline. Easily upload images to the cloud. Automatically perform smart image resizing, cropping and conversion without installing any complex software. Integrate Facebook or Twitter profile image extraction in a snap, in any dimension and style to match your website’s graphics requirements.

When you upload your image to your article in the PageDown editor it will be sent to your Cloudinary account where you can use the web interface to interactively manage your media library.

## Setup your account
Start by signing up to [Cloudinary][1]. In the dashboard under the headline Account details, you have the **Cloud name**, that you need to add in the Appernetic user settings.

![Cloud name in the dashboard ][2]
Cloud name settings in the dashboard.

Now to create a **Upload preset**  click Settings down to the left in the dashboard, select the Upload tab and scroll down to Upload Presets and configure an unsigned upload preset.

![Configure upload presets][3]
Configure unsigned upload presets.

To be able to list images you also need to unmark resource list in security settings.

![Unmark resource list][4]
Unmark resource list in security settings.

Now all should be configured at Cloudinary! Don't forget to verify your account otherwise, it will be disabled. Now the last thing to do is add the information to your Appernetic settings. Click on your GitHub profile image up to the right to show your user settings.

![Appernetic cloudinary settings.][5]
Add your Cloudinary settings to your Appernetic account.

![enter image description here][6]
Continue with your content strategy, now with a new tool in your arsenal!

  [1]: https://cloudinary.com/invites/lpov9zyyucivvxsnalc5/csv1uzzgc8ei8ww3ijtf
  [2]: https://res.cloudinary.com/appernetic/v1457539317/mgwu3dyjasovzrq7risa
  [3]: https://res.cloudinary.com/appernetic/v1457538951/s4jzrkhlauoororjskki
  [4]: https://res.cloudinary.com/appernetic/v1457539163/ybd0y47g0ojzsshvdfd4
  [5]: https://res.cloudinary.com/appernetic/v1457539785/yglkvrt4wjhns4xtmzwi
  [6]: https://res.cloudinary.com/appernetic/v1457543046/bgcxapwc14ebqcvn1apy

repository: pukhalski/appernetic-hugo-project

repo_name: pukhalski/appernetic-hugo-project


text:
# Queen Attack

#### Epicodus Ruby: 06/26/2018

#### By Rita Bennett-Chew, Will Swanson

## Description

A method taking two arguments (the position of the queen in x, y formatted as an array [x,y], and the position of the opponent in the same format) and which checks the possibility of the queen attacking the opponent piece.

## Setup/Contribution Requirements

1. Clone the repo
1. Make a new branch
1. Commit and push your changes
1. Create a PR

## Technologies Used

* Bootstrap x.x.x
* jQuery 3.3.1
* Ruby 2.4.1

## Specs/Steps
* validate that the two pieces are not located in the same spot
 - Input: [1,1],[1,1]
 - Output: "The queen and the opponent piece cannot occupy the same position."
* check that the queen can attack a piece horizontal to her
 - Input: [1,1],[5,1]
 - Output: True
* check that the queen can attack a piece vertical to her
 - Input: [1,1],[1,4]
 - Output: True
* check that the queen can attack a piece diagonal to her
 - Input: [1,1],[4,4]
 - Output: True

## License

This software is licensed under the MIT license.

Copyright (c) 2018 **Rita Bennett-Chew, Will Swanson**

repository: ritabc/queen-attack

repo_name: ritabc/queen-attack


text:
---
layout: post
title:  "Grandma's Lemon Chicken"
excerpt:
date:   2016-01-12 17:12:11 +0000
categories: recipes
---

This recipe was invented by my Grandma, so all the credit will have to go to her. She used to make this when my family visited her, and I can vivdly remember sitting at the dining table in her kitchen, eating lemon chicken, carrots, and green beans. It's a really delicious recipe.

Servings: 3-4

Time: 20 minutes prep, 90 minutes cooking

Cost: €4.96, €1.65 per serving (please note the chicken I got was on offer)

Calories: 506.4 kcal, 168 kcal per serving

![Picture 1]({{site.url}}/images/img_06_01.jpg "Ingredients.")

Ingredients:

  * 3 chicken breasts (500g or so)
  * 1 chicken stock cube
  * Juice of 1 lemon or 5 tbsps bottled lemon juice
  * Flour - 10g needed to coat the chicken, but they're easier to coat if you pour out 50g
  * sage - fresh or dried, to taste (I use 2 tbsps)

Equipment:

  * Frying pan
  * Baking tray

Preheat the oven to 150C fan.  Coat each piece in flour and bung them into the frying pan.

![Picture 2]({{site.url}}/images/img_06_02.jpg "Chicken in the pan")

Cook the chicken so that you can't see any pink raw meat. Don't cook it on too high heat though, because you don't want to brown it either. While that's cooking you can make up your stock. You'll need around 400ml. Add the lemon juice to the stock.

![Picture 3]({{site.url}}/images/img_06_03.jpg "Stock")

When there is no more raw chicken exposed, pop them into a large baking tray, preferebly glass. Pour the stock over the chicken. If the stock isn't covering the chicken you can add some hot water. Then add your fresh or dried sage. I'm using dried because I don't have any herb pots currently!

![Picture 4]({{site.url}}/images/img_06_04.jpg "Ready to go in the oven")

Put in the oven for 90 minutes at 150C (fan oven). I recommend pairing with carrots and beans, but this goes well with rice, potatoes, pasta, and pretty much any veg.

![Picture 5]({{site.url}}/images/img_06_05.jpg "Finished!")

Enjoy!


text:
---
layout: post
title: "Discovering Latent Knowledge Without Supervision"
description: "This post is a tutorial of the code for the research paper Discovering Latent Knowledge in Language Models Without Supervision."
feature-img: "assets/img/rainbow.jpg"
thumbnail: "assets/img/tasmanian_devils.jpg"
tags: [Python, Unsupervised Learning]
---

This post walks through recent work on [Discovering Latent Knowledge in Language Models Without Supervision](https://arxiv.org/abs/2212.03827) by Burns et al. The paper uses latent knowledge in the model's activations to train the model. Their method answers yes-no questions accurately by identifying a direction in the activation space that adheres to logical consistency properties, such as having opposite truth values for a statement and its negation.

This post walks through the code to demonstrate the result. Much of it was taken directly from the [author's repo](https://github.com/collin-burns/discovering_latent_knowledge/tree/main).

<b>Table of Contents</b>
* TOC
{:toc}

```python
import copy
import os

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import plotly.express as px
import torch
import torch.nn as nn
import umap
from datasets import load_dataset
from matplotlib import pyplot as plt
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.manifold import TSNE
from sklearn.model_selection import train_test_split
from torchinfo import summary
from tqdm import tqdm
from transformers import AutoModelForMaskedLM, AutoTokenizer
```

```python
np.random.seed(42)
```

# Data

The authors tested their approach on various datasets and found that it worked. In this post, we'll use the [amazon polarity](https://huggingface.co/datasets/amazon_polarity) dataset for simplicity.

```python
data = load_dataset("amazon_polarity")["test"]
```

    Found cached dataset amazon_polarity (/home/julius/.cache/huggingface/datasets/amazon_polarity/amazon_polarity/3.0.0/a27b32b7e7b88eb274a8fa8ba0f654f1fe998a87c22547557317793b5d2772dc)

      0%|          | 0/2 [00:00<?, ?it/s]

Let's take a look at the dataset.

```python
for i in range(5):
    print(data[i])
    print()
```

    {'label': 1, 'title': 'Great CD', 'content': 'My lovely Pat has one of the GREAT voices of her generation. I have listened to this CD for YEARS and I still LOVE IT. When I\'m in a good mood it makes me feel better. A bad mood just evaporates like sugar in the rain. This CD just oozes LIFE. Vocals are jusat STUUNNING and lyrics just kill. One of life\'s hidden gems. This is a desert isle CD in my book. Why she never made it big is just beyond me. Everytime I play this, no matter black, white, young, old, male, female EVERYBODY says one thing "Who was that singing ?"'}

    {'label': 1, 'title': "One of the best game music soundtracks - for a game I didn't really play", 'content': "Despite the fact that I have only played a small portion of the game, the music I heard (plus the connection to Chrono Trigger which was great as well) led me to purchase the soundtrack, and it remains one of my favorite albums. There is an incredible mix of fun, epic, and emotional songs. Those sad and beautiful tracks I especially like, as there's not too many of those kinds of songs in my other video game soundtracks. I must admit that one of the songs (Life-A Distant Promise) has brought tears to my eyes on many occasions.My one complaint about this soundtrack is that they use guitar fretting effects in many of the songs, which I find distracting. But even if those weren't included I would still consider the collection worth it."}

    {'label': 0, 'title': 'Batteries died within a year ...', 'content': 'I bought this charger in Jul 2003 and it worked OK for a while. The design is nice and convenient. However, after about a year, the batteries would not hold a charge. Might as well just get alkaline disposables, or look elsewhere for a charger that comes with batteries that have better staying power.'}

    {'label': 1, 'title': 'works fine, but Maha Energy is better', 'content': "Check out Maha Energy's website. Their Powerex MH-C204F charger works in 100 minutes for rapid charge, with option for slower charge (better for batteries). And they have 2200 mAh batteries."}

    {'label': 1, 'title': 'Great for the non-audiophile', 'content': "Reviewed quite a bit of the combo players and was hesitant due to unfavorable reviews and size of machines. I am weaning off my VHS collection, but don't want to replace them with DVD's. This unit is well built, easy to setup and resolution and special effects (no progressive scan for HDTV owners) suitable for many people looking for a versatile product.Cons- No universal remote."}

The labels correspond to the sentiment of the review. `1` indicates a positive review and `0` indicates a negative review.

Now let's take a closer look at the text.

```python
text = data[0]["content"]
```

```python
text
```

    'My lovely Pat has one of the GREAT voices of her generation. I have listened to this CD for YEARS and I still LOVE IT. When I\'m in a good mood it makes me feel better. A bad mood just evaporates like sugar in the rain. This CD just oozes LIFE. Vocals are jusat STUUNNING and lyrics just kill. One of life\'s hidden gems. This is a desert isle CD in my book. Why she never made it big is just beyond me. Everytime I play this, no matter black, white, young, old, male, female EVERYBODY says one thing "Who was that singing ?"'

Before it goes into the model we'll need to convert the text into statements that are either true or false. To do that, we can create a new statement saying that the review is positive, and another to say that it's negative. That way we'll be guaranteed to have one true and one false statement.

```python
def create_prompt_review(text: str, label: int) -> str:
    """
    Given a review (text) and its corresponding sentiment label (0 for negative or 1 for positive),
    creates a zero-shot prompt for the review by including the sentiment label as the answer.
    """
    sentiment = ["negative", "positive"][label]
    prompt = f"The following review expresses a {sentiment} sentiment:\n{text}"
    return prompt
```

```python
create_prompt_review(text, 0)
```

    'The following review expresses a negative sentiment:\nMy lovely Pat has one of the GREAT voices of her generation. I have listened to this CD for YEARS and I still LOVE IT. When I\'m in a good mood it makes me feel better. A bad mood just evaporates like sugar in the rain. This CD just oozes LIFE. Vocals are jusat STUUNNING and lyrics just kill. One of life\'s hidden gems. This is a desert isle CD in my book. Why she never made it big is just beyond me. Everytime I play this, no matter black, white, young, old, male, female EVERYBODY says one thing "Who was that singing ?"'

```python
create_prompt_review(text, 1)
```

    'The following review expresses a positive sentiment:\nMy lovely Pat has one of the GREAT voices of her generation. I have listened to this CD for YEARS and I still LOVE IT. When I\'m in a good mood it makes me feel better. A bad mood just evaporates like sugar in the rain. This CD just oozes LIFE. Vocals are jusat STUUNNING and lyrics just kill. One of life\'s hidden gems. This is a desert isle CD in my book. Why she never made it big is just beyond me. Everytime I play this, no matter black, white, young, old, male, female EVERYBODY says one thing "Who was that singing ?"'

Now we have statements that are either true or false. Let's get the hidden states from them.

## Extract hidden states

To do this, we'll need to tokenize the text. Then we'll run it through a model.

We'll need to grab a tokenizer that's right for our model. Fortunately, this is easy to do with `AutoTokenizer`.

```python
# specify where to cache the model weights
cache_dir = os.getenv('MODELS')

# Set the model to the desired device (e.g., CPU or GPU)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
```

We'll use the `deberta-v2-xxlarge` model, so here we'll grab the associated tokenizer.

```python
model_type = "encoder"
tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-v2-xxlarge", cache_dir=cache_dir)
```

    Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
    Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.

Let's take a look at some of the attributes of the tokenizer.

```python
print(f"Tokenizer Type: {type(tokenizer)}")
print(f"Model Type: {tokenizer.model_input_names}")
print(f"Vocabulary Size: {tokenizer.vocab_size}")
print(f"Max Model Input Sizes: {tokenizer.model_max_length}")
print(f"Padding Side: {tokenizer.padding_side}")
print(f"Special Tokens: {tokenizer.special_tokens_map}")
```

    Tokenizer Type: <class 'transformers.models.deberta_v2.tokenization_deberta_v2_fast.DebertaV2TokenizerFast'>
    Model Type: ['input_ids', 'token_type_ids', 'attention_mask']
    Vocabulary Size: 128000
    Max Model Input Sizes: 512
    Padding Side: right
    Special Tokens: {'bos_token': '[CLS]', 'eos_token': '[SEP]', 'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'mask_token': '[MASK]'}

Some of these attributes are worth going into a little more. "Vocabulary Size" refers to the number of unique tokens in the tokenizer's vocabulary. "Max Model Input Sizes" represents the maximum number of tokens the model can process in a single input sequence.

Now let's tokenize a piece to text to get a look at it.

```python
encoder_text_ids = tokenizer(text, truncation=True, return_tensors="pt").input_ids.to(device)
# encoder_text_ids = tokenizer(text, truncation=True, return_tensors="pt").input_ids
```

```python
encoder_text_ids
```

    tensor([[     1,    295,   2096,   6141,     45,     54,      9,      5,  15226,
               6270,      9,     86,   2084,      4,     16,     30,   7585,      8,
                 32,   2927,     14,  32370,      7,     16,    188,   6607,   1935,
                  4,    220,     16,     25,     99,     11,     10,    134,   4148,
                 22,    401,     89,    286,    233,      4,     78,    680,   4148,
                 87,  67775,     72,   2226,     11,      5,   2687,      4,     69,
               2927,     87,  69353,  19273,      4,  30225,     12,     24,  62271,
               2013,  72503,  18615,  51284,      7,   6639,     87,   2777,      4,
                330,      9,    173,     25,     12,   3246,  13547,      4,     69,
                 13,     10,   6460,     13,   2354,   2927,     11,     50,    276,
                  4,    846,    121,    252,    158,     22,    352,     13,     87,
               1287,     89,      4,  74299,     16,    368,     32,      6,    104,
                644,    668,      6,    602,      6,    607,      6,    323,      6,
               2582,      6,   2176, 114957,    464,     54,    309,     55,  11553,
                 28,     15,   4510,    817,    109,      2]], device='cuda:0')

This looks good. We see the start token (`1`) at the beginning and the end token (`2`) at the end.

# Hidden States

#### Model

To get the hidden states we'll have to create our model.

```python
model = AutoModelForMaskedLM.from_pretrained("microsoft/deberta-v2-xxlarge", cache_dir=cache_dir)
```

    Some weights of the model checkpoint at microsoft/deberta-v2-xxlarge were not used when initializing DebertaV2ForMaskedLM: ['lm_predictions.lm_head.dense.weight', 'lm_predictions.lm_head.dense.bias', 'lm_predictions.lm_head.bias', 'deberta.embeddings.position_embeddings.weight', 'lm_predictions.lm_head.LayerNorm.bias', 'lm_predictions.lm_head.LayerNorm.weight']
    - This IS expected if you are initializing DebertaV2ForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
    - This IS NOT expected if you are initializing DebertaV2ForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
    Some weights of DebertaV2ForMaskedLM were not initialized from the model checkpoint at microsoft/deberta-v2-xxlarge and are newly initialized: ['cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.bias']
    You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

```python
type(model)
```

    transformers.models.deberta_v2.modeling_deberta_v2.DebertaV2ForMaskedLM

Let's move it to the appropriate device and set it to eval mode.

```python
model = model.to(device)
model.eval();
```

OK, now we've got our model. We can look at a summary of it.

```python
input_size = (1, 256)
dummy_input = torch.randint(0, model.config.vocab_size, size=input_size, dtype=torch.long, device=device)
summary(model, input_data=dummy_input)
```

    ===================================================================================================================
    Layer (type:depth-idx)                                            Output Shape              Param #
    ===================================================================================================================
    DebertaV2ForMaskedLM                                              [1, 256, 128100]          --
    ├─DebertaV2Model: 1-1                                             [1, 256, 1536]            --
    │    └─DebertaV2Embeddings: 2-1                                   [1, 256, 1536]            --
    │    │    └─Embedding: 3-1                                        [1, 256, 1536]            196,761,600
    │    │    └─LayerNorm: 3-2                                        [1, 256, 1536]            3,072
    │    │    └─StableDropout: 3-3                                    [1, 256, 1536]            --
    │    └─DebertaV2Encoder: 2-2                                      [1, 256, 1536]            786,432
    │    │    └─LayerNorm: 3-4                                        [512, 1536]               3,072
    │    │    └─ModuleList: 3-7                                       --                        (recursive)
    │    │    └─ConvLayer: 3-6                                        [1, 256, 1536]            7,082,496
    │    │    └─ModuleList: 3-7                                       --                        (recursive)
    ├─DebertaV2OnlyMLMHead: 1-2                                       [1, 256, 128100]          --
    │    └─DebertaV2LMPredictionHead: 2-3                             [1, 256, 128100]          --
    │    │    └─DebertaV2PredictionHeadTransform: 3-8                 [1, 256, 1536]            2,363,904
    │    │    └─Linear: 3-9                                           [1, 256, 128100]          196,889,700
    ===================================================================================================================
    Total params: 1,763,803,236
    Trainable params: 1,763,803,236
    Non-trainable params: 0
    Total mult-adds (G): 3.80
    ===================================================================================================================
    Input size (MB): 0.00
    Forward/backward pass size (MB): 2552.44
    Params size (MB): 7052.07
    Estimated Total Size (MB): 9604.51
    ===================================================================================================================

We can also take a look at the configuration of the model.

```python
model.config
```

    DebertaV2Config {
      "_name_or_path": "microsoft/deberta-v2-xxlarge",
      "attention_head_size": 64,
      "attention_probs_dropout_prob": 0.1,
      "conv_act": "gelu",
      "conv_kernel_size": 3,
      "hidden_act": "gelu",
      "hidden_dropout_prob": 0.1,
      "hidden_size": 1536,
      "initializer_range": 0.02,
      "intermediate_size": 6144,
      "layer_norm_eps": 1e-07,
      "max_position_embeddings": 512,
      "max_relative_positions": -1,
      "model_type": "deberta-v2",
      "norm_rel_ebd": "layer_norm",
      "num_attention_heads": 24,
      "num_hidden_layers": 48,
      "pad_token_id": 0,
      "pooler_dropout": 0,
      "pooler_hidden_act": "gelu",
      "pooler_hidden_size": 1536,
      "pos_att_type": [
        "p2c",
        "c2p"
      ],
      "position_biased_input": false,
      "position_buckets": 256,
      "relative_attention": true,
      "share_att_key": true,
      "transformers_version": "4.28.1",
      "type_vocab_size": 0,
      "vocab_size": 128100
    }

We can see that the size of each attention head is 64 and there are 24 attention heads, thus the transformer has a hidden size of 64 * 24 = 1536. This means that there are 1536 intermediate representations of the data within that layer, which includes contextual information and information about the relationship between the tokens.

We can also see that there are 48 hidden layers. This means that there are 48 transformer layers in the model.

Now let's do a simple forward pass through the model.

```python
with torch.no_grad():
    output = model(encoder_text_ids, output_hidden_states=True)
```

We had 48 hidden layers, but we should expect one more hidden state because of the embedding layer. Let's check that.

```python
len(output.hidden_states)
```

    49

Looks good!

## Visualizing Activations

Visualizations of activations are not always easy to make sense of, but let's give it a try. We'll look at the last layer.

```python
layer = -1  # last layer

hs_tuple = output["hidden_states"]
hidden_states = hs_tuple[layer].cpu().numpy().squeeze()

# Apply t-SNE to reduce dimensionality
tsne = TSNE(n_components=2, random_state=42)
hidden_states_tsne = tsne.fit_transform(hidden_states)

# Apply PCA to reduce dimensionality
pca = PCA(n_components=2)
hidden_states_pca = pca.fit_transform(hidden_states)

# Apply UMAP to reduce dimensionality
umap_reducer = umap.UMAP(n_components=2, random_state=42)
hidden_states_umap = umap_reducer.fit_transform(hidden_states)

# Create a function to plot the reduced hidden states
def plot_hidden_states(hidden_states, title):
    plt.figure()
    plt.scatter(hidden_states[:, 0], hidden_states[:, 1])
    plt.xlabel("Component 1")
    plt.ylabel("Component 2")
    plt.title(title)
    plt.show()

# Plot the t-SNE, PCA, and UMAP reduced hidden states
plot_hidden_states(hidden_states_tsne, "t-SNE Visualization of Hidden States")
plot_hidden_states(hidden_states_pca, "PCA Visualization of Hidden States")
plot_hidden_states(hidden_states_umap, "UMAP Visualization of Hidden States")
```

    huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
    To disable this warning, you can either:
    	- Avoid using `tokenizers` before the fork if possible
    	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)

![png](2023-05-08-discovering-latent-knowledge-without-supervision_files/2023-05-08-discovering-latent-knowledge-without-supervision_51_1.png)

![png](2023-05-08-discovering-latent-knowledge-without-supervision_files/2023-05-08-discovering-latent-knowledge-without-supervision_51_2.png)

![png](2023-05-08-discovering-latent-knowledge-without-supervision_files/2023-05-08-discovering-latent-knowledge-without-supervision_51_3.png)

It's always hard to know what this means. Let's look at it in 3D (commented out on the blog).

```python
pca_3d = PCA(n_components=3)
hidden_states_pca_3d = pca_3d.fit_transform(hidden_states)

df = pd.DataFrame(hidden_states_pca_3d, columns=["Component 1", "Component 2", "Component 3"])

fig = px.scatter_3d(
    df,
    x="Component 1",
    y="Component 2",
    z="Component 3",
    color="Component 1",
    color_continuous_scale=px.colors.sequential.Plasma,
    size_max=10,
    opacity=0.8,
    template="plotly_dark",
    title="3D PCA Visualization of Hidden States",
    labels={
        "Component 1": "Component 1",
        "Component 2": "Component 2",
        "Component 3": "Component 3",
    },
    width=800,
    height=600,
)

# fig.show()
```

# Doing it at Scale

## Getting Hidden States

Now, we're going to get all the hidden states. We'll get the ones with positive sentiment and negative sentiment.

```python
def get_encoder_hidden_states(model, tokenizer, input_text, layer=-1):
    """
    Given an encoder model and some text, gets the encoder hidden states (in a given layer, by default the last)
    on that input text (where the full text is given to the encoder).

    Returns a numpy array of shape (hidden_dim,)
    """
    # tokenize
    encoder_text_ids = tokenizer(input_text, truncation=True, return_tensors="pt").input_ids.to(model.device)

    # forward pass
    with torch.no_grad():
        output = model(encoder_text_ids, output_hidden_states=True)

    # get the appropriate hidden states
    hs_tuple = output["hidden_states"]

    hs = hs_tuple[layer][0, -1].detach().cpu().numpy()

    return hs
```

```python
model.eval()
all_neg_hs, all_pos_hs, all_gt_labels = [], [], []
n = 100

for _ in tqdm(range(n)):
    # for simplicity, sample a random example until we find one that's a reasonable length
    # (most examples should be a reasonable length, so this is just to make sure)
    while True:
        idx = np.random.randint(len(data))
        text, true_label = data[idx]["content"], data[idx]["label"]
        # the actual formatted input will be longer, so include a bit of a marign
        if len(tokenizer(text)) < 400:
            break

    # get hidden states
    neg_hs = get_encoder_hidden_states(model, tokenizer, create_prompt_review(text, 0))
    pos_hs = get_encoder_hidden_states(model, tokenizer, create_prompt_review(text, 1))

    # collect
    all_neg_hs.append(neg_hs)
    all_pos_hs.append(pos_hs)
    all_gt_labels.append(true_label)

X_neg = np.stack(all_neg_hs)
X_pos = np.stack(all_pos_hs)
y = np.stack(all_gt_labels)
```

    100%|█████████████████████████████████████████| 100/100 [00:16<00:00,  5.93it/s]

Now we have our hidden states. Let's split them into train and test sets.

```python
X_pos_train, X_pos_test, X_neg_train, X_neg_test, y_train, y_test = train_test_split(
    X_pos, X_neg, y, random_state=42
)
```

## Test Model Representations

The question is, do we have good representations of the data? The way to check this is to make a simple model. If logistic regression can't use the hidden states to classify the review, they're not any good.

We want to combine the positive and negative together and there are a few ways we could do that. The simplest way is just to subtract them.

```python
X_train = X_neg_train - X_pos_train
X_test = X_neg_test - X_pos_test
```

```python
lr = LogisticRegression(class_weight="balanced")
lr.fit(X_train, y_train)
print(f"Logistic regression accuracy: {lr.score(X_test, y_test)}")
```

    Logistic regression accuracy: 0.92

Yay! We've built good representations.

## Contrast-Consistent Search

Now let's use at Contrast-Consistent Search (CCS). You can learn more about it in the paper linked at the top.

The first step is to prepare the data by normalizing it and turning it into a Pytorch tensor.

```python
def normalize(x: np.ndarray) -> np.ndarray:
    normalized_x = x - x.mean(axis=0, keepdims=True)
    return normalized_x
```

```python
# Prepare data
X_neg_train_normed = normalize(X_neg_train)
X_pos_train_normed = normalize(X_pos_train)
```

```python
def convert_to_tensors(x0: np.ndarray, x1: np.ndarray, device: str) -> tuple[torch.Tensor, torch.Tensor]:
    x0 = torch.tensor(x0, dtype=torch.float, device=device)
    x1 = torch.tensor(x1, dtype=torch.float, device=device)
    return x0, x1
```

```python
# Convert data to tensors
X_neg_train_tensor, X_pos_train_tensor = convert_to_tensors(X_neg_train_normed, X_pos_train_normed, device="cuda")

```

Now we can train with CCS. Note that we don't need any labels for this.

To do this, we'll need to create a probe. A probe is a simple model used in self-supervised learning to learn a task-specific representation from the features extracted by an unsupervised model. In this case,  the probe is a binary classifier that distinguishes between the two sets of samples.

```python
def create_probe(input_size: int, device: str) -> nn.Module:
    probe = nn.Sequential(nn.Linear(input_size, 1), nn.Sigmoid())
    probe.to(device)
    return probe
```

```python
probe = create_probe(X_neg_train_tensor.shape[-1], "cuda")
```

The loss function is a crucial component of the CCS algorithm because it serves as the primary driving force behind the optimization of the model's parameters. In this specific case, the CCS algorithm aims to learn a binary classifier that can distinguish between two sets of samples.

The CCS loss function comprises two key components: the informative loss and the consistent loss. The informative loss encourages the model to minimize the squared difference between the probabilities assigned to the samples from both sets. This component of the loss function aims to make the model more discriminative, pushing it to assign higher probabilities to one set of samples and lower probabilities to the other set. In doing so, the model is encouraged to identify patterns or features that separate the two sets of samples, thus maximizing the information it can learn from the unlabeled data.

The consistent loss, on the other hand, focuses on ensuring that the model remains consistent in its predictions. This is achieved by penalizing the squared difference between the probabilities assigned to samples from x0 and the complement of the probabilities assigned to the corresponding samples from x1. By minimizing this component of the loss function, the model is encouraged to consistently assign complementary probabilities to samples from the two sets. This consistency is crucial for ensuring the model's robustness and preventing it from overfitting to spurious correlations in the training data.

```python
def get_loss(p0: torch.Tensor, p1: torch.Tensor) -> torch.Tensor:
    informative_loss = (torch.min(p0, p1) ** 2).mean(0)
    consistent_loss = ((p0 - (1 - p1)) ** 2).mean(0)
    return informative_loss + consistent_loss
```

We'll write a function that finds the best probe and returns it.

```python
def train(probe: nn.Module,
          x0: torch.Tensor,
          x1: torch.Tensor,
          num_epochs: int,
          lr: float,
          weight_decay: float,
          device: str) -> float:

    # Shuffle the input data
    permutation = torch.randperm(len(x0))
    x0, x1 = x0[permutation], x1[permutation]

    # Initialize the optimizer
    optimizer = torch.optim.AdamW(probe.parameters(), lr=lr, weight_decay=weight_decay)

    # Set batch size and number of batches
    batch_size = len(x0)
    nbatches = len(x0) // batch_size

    # Training loop
    for _ in range(num_epochs):
        for j in range(nbatches):
            # Create input batches
            x0_batch = x0[j * batch_size : (j + 1) * batch_size]
            x1_batch = x1[j * batch_size : (j + 1) * batch_size]

            # Forward pass through the probe
            p0, p1 = probe(x0_batch), probe(x1_batch)

            # Calculate loss
            loss = get_loss(p0, p1)

            # Update the probe's parameters
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

    return loss.detach().cpu().item()
```

```python
def find_best_probe(
    probe: nn.Module,
    X_neg: torch.Tensor,
    X_pos: torch.Tensor,
    num_epochs: int,
    ntries: int,
    lr: float,
    weight_decay: float,
    device: str,
) -> tuple[nn.Module, float]:
    """
    Train the probe for multiple attempts and return the best probe and its loss.

    Parameters:
    probe: Initial probe model to train.
    X_neg: Negative class data.
    X_pos: Positive class data.
    num_epochs: Number of training epochs for each try.
    ntries: Number of attempts to find the best probe.
    lr: Learning rate for the optimizer.
    weight_decay: Weight decay for the optimizer.
    batch_size: Batch size for training, set to -1 for full batch training.
    device: Device to use for training, e.g. "cuda" or "cpu".

    Returns the best probe found and its corresponding loss.
    """
    best_loss = np.inf
    for _ in range(ntries):
        # Create and train a new probe for each attempt
        probe = create_probe(X_neg.shape[-1], device)
        loss = train(probe, X_neg, X_pos, num_epochs, lr, weight_decay, device)

        # Update the best probe if the current loss is lower
        if loss < best_loss:
            best_probe = copy.deepcopy(probe)
            best_loss = loss

    return best_probe, best_loss
```

```python
best_probe, best_loss = find_best_probe(probe, X_neg_train_tensor, X_pos_train_tensor, num_epochs=1000, ntries=10, lr=1e-3,
                        weight_decay=0.01, device="cuda")
```

Now we can evaluate our results.

```python
def get_acc(probe: nn.Module, X_neg_test: np.ndarray, X_pos_test: np.ndarray, y_test: np.ndarray, device: str) -> float:
    """
    Compute the accuracy of the probe on the given test data.

    Parameters:
    probe: The trained probe model.
    X_neg_test: Negative class test data.
    X_pos_test: Positive class test data.
    y_test: Test labels.
    device: Device to use for evaluation, e.g. "cuda" or "cpu".

    Returns the accuracy of the probe on the test data.
    """
    # Normalize the test data
    x0_test_normalized = normalize(X_neg_test)
    x1_test_normalized = normalize(X_pos_test)

    # Convert the test data to tensors
    x0_test, x1_test = convert_to_tensors(x0_test_normalized, x1_test_normalized, device)

    # Compute probabilities with the best probe
    with torch.no_grad():
        p0, p1 = probe(x0_test), probe(x1_test)

    # Compute average confidence and predictions
    avg_confidence = 0.5 * (p0 + (1 - p1))
    predictions = (avg_confidence.detach().cpu().numpy() < 0.5).astype(int)[:, 0]

    # Calculate the accuracy
    acc = (predictions == y_test).mean()
    acc = max(acc, 1 - acc)

    return acc
```

```python
ccs_acc = get_acc(best_probe, X_neg_test, X_pos_test, y_test, device="cuda")
print(f"CCS accuracy: {ccs_acc}")
```

    CCS accuracy: 0.84

The final accuracy is pretty good! This indicates that the algorithm is able to learn meaningful patterns from the given data without the need for explicit labels.


text:
---
title: jQuery DOM Manipulation test (bug?)
author: Damien
layout: post
comments: true
permalink: /2012/01/21/jquery-dom-manipulation-test-bug/
categories:
  - Javascript
  - web development
tags:
  - JQuery
---

    I discovered something really weird when I needed to manipulate the DOM with jQuery (1.7.1)….
 check the log and the source with Firebug and you will see what I mean!

If you are interested to analyze the script, here is a [separate file][1] with the complete test. Maybe you will find an answer for me :)

 [1]: http://damien.antipa.at/wp-content/uploads/2012/01/jquery_test.html "jQuery DOM manipulation test with weird behavior"

repository: visiongeist/blog

repo_name: visiongeist/blog


text:
# [akka](http://akka.io/)

---

> Akka is a toolkit and runtime for building highly concurrent, distributed, and fault tolerant event-driven applications on the JVM.

---

* actors
* remoting
* supervision

---

> Actors er concurrent prosesser som kommuniserer via meldinger

---

```scala
import akka.actors._

case class Message(msg: String)

val system = ActorSystem()
// ActorRef er en nettverkstransparent referanse til en actor
val client: ActorRef = system.actorOf(Props[Client])

// ! sender melding til en actor
client ! Message("hi")

class Client extends Actor {
  // implementer all message handling i receive m/pattern matching
  def receive = {
    case Message(msg) => println(s"got $msg")
  }
}

```

---

> 50 million msg/sec on a single machine.
> Small memory footprint; ~2.5 million actors per GB of heap.

---

```scala
// instansier
system.actorOf(Props[SomeActor], name = "some-actor")

system.actorOf(Props(new SomeActor))

// lookup
system.actorSelection("/user/some-actor")
```

---

## supervision

```scala
class Supervisor extends Actor {
  override val supervisorStrategy =
    OneForOneStrategy(maxNrOfRetries = 10, withinTimeRange = 1 minute) {
      case _: ArithmeticException      ⇒ Resume
      case _: NullPointerException     ⇒ Restart
      case _: Exception                ⇒ Escalate
    }

  val worker = context.actorOf(Props[Worker])

  def receive = {
    case n: Int => worker forward n
  }
}
```

repository: oyvindberg/scala-kurs-oppgaver

repo_name: oyvindberg/scala-kurs-oppgaver


text:
---
slug: /blog/2023-06-08/todo_trek
title: Some observations about Chris McCord's new LiveView demo project todo_trek
description:
draft: true
date: 2023-06-08T12:00:00.000-07:00
---

I always find myself reading the Phoenix LiveView changelog with baited breath. Chris McCord, the author of Phoenix, has done a stellar job steering the ecosystem. Whenever the Phoenix team releases new features, I'm on the lookout for opportunities to learn and improve my mastery of Elixir and Phoenix. As of Phoenix LiveView 0.19, Chris also released a [reference codebase][gc] to help demo the new features for nested streams and form changeset apis.

Here's a short list of some learnings that I've identified while perusing the [todo_trek][gc] code.

## Using structs to annotate Phoenix Component attribute types

Throughout documentation I've seen usages of `:map` but hadn't seen the specific type specified. I don't doubt that this appeared in more recent boilerplate generated by `mix phx.new`.

```elixir
  #...
  # Here JS is being used to denote the type vs using a "plain" :map
  attr :on_cancel, JS, default: %JS{}

  def modal(assigns) do
  #...
```

## Passing streams down

Another pattern that I thought was interesting was the way that streams are passed down as the source of truth throughout the code for the todo list.

## Event pattern

This pattern isn't necessarily new. McCord's live_beats app also used this pattern, but seeing it a second time

## References

[gc]: https://github.com/chrismccord/todo_trek

- [GitHub - chrismccord/todo_trek][gc]


text:
# ReCub
Code to calculate unique permutations of a 3x3x3 Rubik's Cube.

Here is a code I made that calculates unique permutations of a 3x3x3 cube.
The program starts with a solved cube and makes one move at a time and checks if the new found permutation of cube has already occured, and if not, it saves it in a file to be checked after each move.
Permutation stored in file is stored as a concatenated string of moves performed on cube and resultant cube permutation.

# Factors to check if the permutation is unique
1. If the cube permutation is obtained before in the file.
    check_repeat() is function which checks if current cube permutation is already available in perm.dat file.
2. If the every move made to get current permutation was done in another orientation of cube, is it still unique.
    check_rotate() is function which checks if moves made on solved cube to obtain current cube permutation were done having some other cube orientation obtains another cube permutation; this newly obtained permutation is then checked in perm.dat file.
3. If all the moves were mirrored.
    check_mirror() is function performs moves made to get current permutation in a mirrored way to obtain new permutation. This new permutation is checked with perm.dat file and also checked through check_rotate() function to get all possible orientations.
    Here mirroring moves means :
      Right -> Left,
      Up -> UpInverted,
      Front -> FrontInverted and so on.

# Saving results
When a permutation passes all the checks and is unique in above mentioned ways, it is stored in 'perm.dat' named file.

# Notes
In the Code, MaxMoves indicates till how many moves does the program has to check the permutations. I've set it to 20 because it was previously observed as all permutations beyond 20 moves repeat themselves.

While storing results, I've used Alphabets A - L to determine cube moves.
A : Up,
B : UpInverted,
C : Down,
D : DownInverted,
E : Left,
F : LeftInverted,
G : Right,
H : RightInverted,
I : Front,
J : FrontInverted,
K : Back,
L : BackInverted

In perm.dat,
W : White,
R : Red,
B : Blue,
O : Orange,
G : Green,
Y : Yellow

This is first time I've wrote such a big code so please ignore wrong naming conventions, wrong coding conventions. Thank you.


text:
# Custom implementation

## Track pageview

Sends a page view event , by default. Specific information regarding the page is automatically collected and sent (if unspecified and detected from the page).  You can call this method once the window._swat object is initialized.

This API can be called in below scenarios,

- A product page is loaded

- A product is loaded in a quick view popup

- A product variant is changed, if you want more granularity in tracking, at a variant level in addition to the above two scenarios

```javascript
window._swat.trackPageview({
  "epi": 123,
  "du": "https://www.understandfrance.org/AsterixObelix",
  "empi": 1234,
  "iu" : "//www.understandfrance.org/Images/AsterixObelix.jpg",
  "pr": 50,
  "stk": 100,
  "variants": [{"Blue / XL": 123}]
});
```

Argument | Type | Description
--------- | ------- | -----------
epi | int/string | External product unique id (variant level if applicable)
empi | int/string | External product master id (if there is a group product id with different variant ids)
iu | string | Image-uri - Without protocol so protocol can be decided while rendering to http or https
pr | float | Price


text:
# slothbear-issues

https://bharding2.github.io/slothbear-issues/

This was an experiment to launch a blogging platform using the github issues api in the least number of lines of code possible.  There version using angular js ended up at ~60 lines of code, and the completely vanilla version ended up at ~100 lines of code.


text:
---
templateKey: blog-post
title: >-
  August 2019 Meetup: How to optimize your EC2 costs by upto 90% & scale your
  application throughput by upto 10x
date: 2019-08-28T09:12:58.958Z
description: Aug 2019 meetup features Amazon EC2 Spot Instances
tags:
  - meetup
  - August 2019
  - '2019'
---
After a very successful [AWS Community Day 2019](https://www.awsugblr.in/blog/2019-08-01-aws-community-day-bengaluru-2019-the-hatrick-made-a-history/), we are pretty excited to bring back our monthly meetups with a super good session. The event will be held on 31st August 2019 at Locus in 80 ft road, Koramangala.

We have [Chakravarthy Nagarajan](https://www.linkedin.com/in/chakravarthy-nagarajan-7653311a/) (popularly known as Chakra), the Spot Specialist SA from AWS talking about Amazon EC2 Spot Instances. You will learn to optimize your EC2 costs by upto 90%, scale your application throughput by upto 10x and build a resilient service.

Amazon EC2 Spot instances are compute instances provided from the spare pooled capacity of On-Demand EC2 infrastructure, offered with low, predictable prices - discounted upto 90% on the On-Demand instance prices - with a caveat of these instances being "interruptible" after a 2 min notification, if Amazon needs to pull back the Spot instances capacity to serve On-Demand customers.

In this session delivered by Spot Specialist teams from Amazon, learn how customers are using EC2 Spot instances - not just as an attractive low cost purchase model for Cloud computing resources - but as an architectural approach to run applications with highly scalable compute resources, while working with instance interruptions to improve the resilience of their application services.

 Here you go with the agenda

* Conceptual Overview Presentation for EC2 Spot instances - 1 hr
* Hands on Demo of Spot Fleet and ASG - (15 mins)
* Workload specific Spot integrations - (Auto-Scaling, EMR and Containers)  - (30 mins)
* Pop Quiz, prizes announcement and Wrap up (15 mins)

### About the speaker:

[Chakravarthy Nagarajan](https://www.linkedin.com/in/chakravarthy-nagarajan-7653311a/) (popularly known as Chakra) is a Spot Specialist SA at AWS since Aug’18. Chakra comes with 18+ years of overall experience - across areas of expertise including HPC and Bigdata. He has  implemented and designed over 150+ HPC/BigData clusters in APAC across Industry segments. He has published some interesting white papers on HPC storage and Big Data earlier in his career. On the academic front, he is an International MBA graduate from United Business Institute, Brussels.

### Venue

Locus \
2nd Floor, Opposite Regional Passport Office, \
80 Feet Rd, Koramangala 8th Block, \
Bengaluru, \
Karnataka 560095

### Register for the meetup

[Please fill up this form](https://lnkd.in/ffCtzzt) to register and help us plan for better logistics

## Join the community

Join the AWSUGBLR community - you can find us on our [About](https://www.awsugblr.in/about) page. We now have a presence on [Slack](http://go.awsugblr.in/slack) as well

repository: awsugblr/awsugblr

repo_name: awsugblr/awsugblr


text:
Markdown
========

You can write codebase documentation in the popular Markdown or ReST format;
Pants eases generating readable versions and publishing them so your users can read it.

Markdown to HTML
----------------

To tell Pants about your Markdown or ReST file, use a
`page`
target in a `BUILD` file as in this excerpt from
[examples/src/java/org/pantsbuild/example/hello/main/BUILD](https://github.com/pantsbuild/pants/blob/master/examples/src/java/org/pantsbuild/example/hello/main/BUILD):

!inc[start-after=README page](hello/main/BUILD)

To render the page as HTML, use the
markdown goal. For example, to view
`examples/src/java/org/pantsbuild/example/hello/main/README.md` as HTML in
your browser,

    :::bash
    $ ./pants markdown --open examples/src/java/org/pantsbuild/example/hello/main:readme

Pants generates the HTML files in the `dist/markdown/` directory tree.

Markdown Syntax
---------------

Pants uses the Python `Markdown` module; thus, in addition to the usual
Gruber `Markdown` syntax, there are [other
features](http://pythonhosted.org/Markdown/) Pants uses Python
Markdown's `codehilite`, `extra`, `tables`, and `toc` extensions.

### Link to Another `page`

One `page` can link to another. Regular Markdown-link syntax works for
regular links; but if you use `page`s to generate both `.html` files and
wiki pages, it's not clear what to link to: the `.html` file or the wiki
address. You can use a Pants-specific syntax to make links that work
with generated HTML or wiki pages.

To set up a page `source.md` that contains a link to `dest.md`, you make
a `dependencies` relation and use the special `pants(...)` syntax in the
markdown.

In the `BUILD` file:

    :::python
    page(name='source',
      source='source.md',
      dependencies=[':dest'], # enables linking
      provides=[...publishing info...],
    )

    page(name='dest',
      source='dest.md',
      provides=[...publishing info...],
    )

To set up the links in `source.md` that point to `dest.md` or an anchor
therein:

    For more information about this fascinating topic,
    please see [[Destinations|pants('path/to:dest')]],
    especially the
    [[Addendum section|pants('path/to:dest)'#addendum]],

Pants replaces the `pants('path/to:dest')` with the appropriate link.

### Include a File Snippet

Sometimes the best way to explain `HelloWorld.java` is to show an
excerpt from `HelloWorld.java`. You can use the `!inc` markdown to do
this. Specify a file to include and (optionally) regexps at which to
start copying or stop copying. For example, to include an excerpt from
the file `HelloMain.java`, starting with the first line matching the
pattern `void main` and stopping before a subsequent line matching
`private HelloMain`:

    !inc[start-at=void main&end-before=private HelloMain](HelloMain.java)

To include *all* of `HelloMain.java`:

    !inc(HelloMain.java)

To include most of `HelloMain.java`, starting after license boilerplate:

    !inc[start-after=Licensed under the Apache](HelloMain.java)

It accepts the following optional parameters, separated by ampersands
(&):

start-at=*substring*

When excerpting the file to include, start at the first line containing
*substring*.

start-after=*substring*

When excerpting the file to include, start after the first line
containing *substring*.

end-before=*substring*

When excerpting the file to include, stop before a line containing
*substring*.

end-at=*substring*

When excerpting the file to include, stop at a line containing
*substring*.

ReStructedText Syntax
---------------------

Pants can generate web content from
[docutils reStructuredText](http://docutils.sourceforge.net/rst.html)-formatted text.

To tell Pants that your `page` target's source is in reStructuredText format, you can either

* give the `source` file an `.rst` file extension, or
* pass `format='rst'` to the `page` target.

Publishing
----------

You can tell Pants to publish a page. So far, there's only one way to
publish: as a page in an Atlassian Confluence wiki. (You can add other
doc-publish backends to Pants;
[[send us a patch|pants('src/docs:howto_contribute')]]!)

To specify the "address" to which to publish a page, give it a
`provides` parameter.

Once you've done this, you can publish the page by invoking the goal
set up in your workspace. For example, if
the goal was set up with the name "confluence", you invoke:

    :::bash
    $ ./pants confluence examples/src/java/org/pantsbuild/example/hello/main:readme

To specify that a page should be published to a Confluence wiki page,
set its `provides` to something like:

    :::python
    page(...
      provides=[
        wiki_artifact(wiki=confluence,
          space='ENG',
          title='Pants Hello World Example',
          parent='Examples',
        )
      ],)

...assuming your workspace is set up for confluence publishing with a `Wiki` symbol named
`confluence` set up as part of a plugin as described below:

**Setting up your workspace for Confluence publish**

That `wiki` specifies some information about your wiki server. So far, the only kind of thing you
can publish to is a Confluence wiki. To set up and register this symbol, set up a
[[Pants plugin|pants('src/docs:howto_plugin')]] if your workspace doesn't already
have one. In the plugin, define a `Wiki` and register it:

    :::python
    import urllib

    from pants.backend.docgen.targets.doc import Wiki

    def confluence_url_builder(page, config):
      title = config['title']
      return title, 'https://wiki.archie.org/display/%s/%s' % (
        config['space'],
        urllib.quote_plus(title))

    confluence_wiki = Wiki(name='confluence', url_builder=confluence_url_builder)

    # in register.py:
    def build_file_aliases():
      return BuildFileAliases(
        # ...
        objects={
          # ...
          'confluence', confluence_wiki},
      )

You need to install a goal to enable publishing a doc to confluence. To do this, in your Pants
plugin, install a goal that subclasses `ConfluencePublish`:

    :::python
    from pants.backend.docgen.tasks.confluence_publish import ConfluencePublish

    class ArchieConfluence(ConfluencePublish):
      def wiki(self):
        return confluence_wiki
      def api(self):
        return 'confluence2'

    # in register.py:
    def register_goals():
      # ...
      task(name='confluence', action=ArchieConfluence, dependencies=['markdown']).install()

In your `pants.ini` file, add a section with the url of your wiki server. E.g., if your server
is at wiki.archie.org, it would look like:

    [confluence]
    url: https://wiki.archie.org

repository: foursquare/pants

gha_language: Python

repo_name: foursquare/pants


text:
a view template is a resource.

dependencies are either server-side or client-side, runtime and loadtime.

a view template may have the following dependencies that are created by a template author:

css -> <link /> client-side, loadtime
javascript -> <script /> client-side, loadtime
javascript -> controller client-side
data sources -> server-side, runtime

a data source is any resource
data source URLs may be created dynamically

webcomponents may have two types of controllers:

component controller - called implicitly when accessing the view of a component, contains lifecycle methods.
view controller - a view controller is either executed client- or server-side, based on the render mode.

repository: juxtapos/Rain

repo_name: juxtapos/Rain


text:
---
title: About - Meet our expert
menuTitle: About
---

|
Need dummy content here to make the image appear bigger
 | Experience                                                                                |
| --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ![Anupreksha's Profile Photo](./profile.jpg)                                                  |
My name is **_Dt. Anupreksha Rastogi_**. I am professionally a dietician with over 7 years of experience in Therapeutic Diets and weight management.
 |

Due to my regular interest in food, I prefered to pursue my studies in the subject. So, I did my Masters in Food Science from Banaras Hindu University in 2012. From there my journey began by working with the organizations like Fortis Healthcare, VLCC, Corporate offices, many chains and in the Trinity Hospital, Zirakpur Mohali previously.

She now works out of a clinic mentioned below.
![Anupreksha's Clinic Announcement](./announcement.jpeg)
Being a qualified dietician and a life member of Indian Dietetics Association, I believe in implementing safe, healthy eating habits not just thr numbers in weighing scale.
All this is taught throughout the journey of our programs.

### Expertise in:

- Therapeutics Diets
- Weight Management
- Diabetes Care
- Sports Nutrition for athletes
- Corporate Food Service Management.

#### Qualifications:

M.Sc. (Food & Nutrition)
CFN, DNHE
UGC-NET-June2012
Diabetes Educator

### Contact:

Reach out to Dt. Anupreksha on her email and/or her mobile number for a consultation.

**Email**<EMAIL_ADDRESS>
**Mobile**: +91-8196857331
**Mobile**: +91-8557088040

**Smart Binge Clinic Address**: Spectra Healthcare, SCO-74, Sector-20c, Tribune Road, Chandigarh

repository: divyamrastogi/smart-binge

repo_name: divyamrastogi/smart-binge


text:
## Deploy etcd in Docker
Please read the [issues](https://github.com/gradywang/notesofyongqiao/tree/master/etcd/issues) and [notes](https://github.com/gradywang/notesofyongqiao/blob/master/etcd/notes.md) before installation carefully.

The following steps are refer from the [official document](https://coreos.com/etcd/docs/latest/docker_guide.html).

### ETCD official docker image

Pull the latest official ETCD image:

```
# docker pull quay.io/coreos/etcd
```

All supported versions are listed in [tags](https://quay.io/repository/coreos/etcd?tab=tags) , you can specify the image version if needed. For example:
```
# docker pull quay.io/coreos/etcd:v3.0.4
```

### 2.x Image version
#### Run etcd in standalone mode
Please check wherther a etcd service has be started on this machine.
```
# export HostIP="<IP_ADDRESS>"
# export ETCD_IMAGE="quay.io/coreos/etcd:v2.3.7"

# docker run -d \
 --name etcd \
 --restart always \
 --net host ${ETCD_IMAGE} \
 --name etcd \
 --advertise-client-urls http://${HostIP}:2379,http://${HostIP}:4001 \
 --listen-client-urls http://<IP_ADDRESS>:2379,http://<IP_ADDRESS>:4001
```

Refer the [deploy.md](https://github.com/gradywang/notesofyongqiao/blob/master/etcd/deploy/deploy.md) to install etcd and use `etcdctl` to do verification:

```
# etcdctl cluster-health
member ce2a822cea30bfca is healthy: got healthy result from http://<IP_ADDRESS>:2379
cluster is healthy

# etcdctl member list
ce2a822cea30bfca: name=etcd peerURLs=http://localhost:2380,http://localhost:7001 clientURLs=http://<IP_ADDRESS>:2379,http://<IP_ADDRESS>:4001 isLeader=true
```

#### Run a etcd cluster with three nodes

```
# export ETCD_NODE1="<IP_ADDRESS>" && export ETCD_NODE2="<IP_ADDRESS>" && export ETCD_NODE3="<IP_ADDRESS>"
# export ETCD_NAME1="etcd1" && export ETCD_NAME2="etcd2" && export ETCD_NAME3="etcd3"
# export ETCD_CLUSTER_TOKEN="etcd-cluster"
# export ETCD_IMAGE="quay.io/coreos/etcd:v2.3.7"

# docker run -d \
 --name etcd \
 --restart always \
 --net host ${ETCD_IMAGE} \
 -name ${ETCD_NAME1} \
 -advertise-client-urls http://${ETCD_NODE1}:2379,http://${ETCD_NODE1}:4001 \
 -listen-client-urls http://<IP_ADDRESS>:2379,http://<IP_ADDRESS>:4001 \
 -initial-advertise-peer-urls http://${ETCD_NODE1}:2380 \
 -listen-peer-urls http://<IP_ADDRESS>:2380 \
 -initial-cluster-token etcd-cluster-1 \
 -initial-cluster ${ETCD_NAME1}=http://${ETCD_NODE1}:2380,${ETCD_NAME2}=http://${ETCD_NODE2}:2380,${ETCD_NAME3}=http://${ETCD_NODE3}:2380 \
 -initial-cluster-state new

# docker run -d \
 --name etcd \
 --restart always \
 --net host ${ETCD_IMAGE} \
 -name ${ETCD_NAME2} \
 -advertise-client-urls http://${ETCD_NODE2}:2379,http://${ETCD_NODE2}:4001 \
 -listen-client-urls http://<IP_ADDRESS>:2379,http://<IP_ADDRESS>:4001 \
 -initial-advertise-peer-urls http://${ETCD_NODE2}:2380 \
 -listen-peer-urls http://<IP_ADDRESS>:2380 \
 -initial-cluster-token etcd-cluster-1 \
 -initial-cluster ${ETCD_NAME1}=http://${ETCD_NODE1}:2380,${ETCD_NAME2}=http://${ETCD_NODE2}:2380,${ETCD_NAME3}=http://${ETCD_NODE3}:2380 \
 -initial-cluster-state new

# docker run -d \
 --name etcd \
 --restart always \
 --net host ${ETCD_IMAGE} \
 -name ${ETCD_NAME3} \
 -advertise-client-urls http://${ETCD_NODE3}:2379,http://${ETCD_NODE3}:4001 \
 -listen-client-urls http://<IP_ADDRESS>:2379,http://<IP_ADDRESS>:4001 \
 -initial-advertise-peer-urls http://${ETCD_NODE3}:2380 \
 -listen-peer-urls http://<IP_ADDRESS>:2380 \
 -initial-cluster-token etcd-cluster-1 \
 -initial-cluster ${ETCD_NAME1}=http://${ETCD_NODE1}:2380,${ETCD_NAME2}=http://${ETCD_NODE2}:2380,${ETCD_NAME3}=http://${ETCD_NODE3}:2380 \
 -initial-cluster-state new
```

Refer the [deploy.md](https://github.com/gradywang/notesofyongqiao/blob/master/etcd/deploy/deploy.md) to install etcd and use `etcdctl` to do verification:

```
# etcdctl cluster-health
member 4a2c4ac424ea87c2 is healthy: got healthy result from http://<IP_ADDRESS>:2379
member 9f564d76c2326b1a is healthy: got healthy result from http://<IP_ADDRESS>:2379
member f45d70d240a351bc is healthy: got healthy result from http://<IP_ADDRESS>:2379
cluster is healthy

# etcdctl member list
4a2c4ac424ea87c2: name=etcd1 peerURLs=http://<IP_ADDRESS>:2380 clientURLs=http://<IP_ADDRESS>:2379,http://<IP_ADDRESS>:4001 isLeader=false
9f564d76c2326b1a: name=etcd0 peerURLs=http://<IP_ADDRESS>:2380 clientURLs=http://<IP_ADDRESS>:2379,http://<IP_ADDRESS>:4001 isLeader=true
f45d70d240a351bc: name=etcd2 peerURLs=http://<IP_ADDRESS>:2380 clientURLs=http://<IP_ADDRESS>:2379,http://<IP_ADDRESS>:4001 isLeader=false
```

### >3.x Image version
In the >3.x etcd official docker image,  the docker `CMD` has be changed to support etcdctl with environment variable.

#### Run etcd in standalone mode
Please check wherther a etcd service has be started on this machine.
```
# export ETCD_NAME="etcd"
# export ETCD_IMAGE="quay.io/coreos/etcd:v3.0.4"

# docker run -d \
 --name etcd \
 --restart always \
 --net host ${ETCD_IMAGE} /usr/local/bin/etcd \
 --name ${ETCD_NAME} \
 --advertise-client-urls http://$(hostname -i):2379,http://$(hostname -i):4001 \
 --listen-client-urls http://<IP_ADDRESS>:2379,http://<IP_ADDRESS>:4001
```

Verification:
```
# docker exec etcd /bin/sh -c "/usr/local/bin/etcdctl cluster-health"
member 8e9e05c52164694d is healthy: got healthy result from http://<IP_ADDRESS>:2379
cluster is healthy

# docker exec etcd /bin/sh -c "/usr/local/bin/etcdctl member list"
8e9e05c52164694d: name=etcd peerURLs=http://localhost:2380 clientURLs=http://<IP_ADDRESS>:2379,http://<IP_ADDRESS>:4001 isLeader=true

User V3 API to do verification:
# docker exec etcd /bin/sh -c "export ETCDCTL_API=3 && /usr/local/bin/etcdctl endpoint health"
<IP_ADDRESS>:2379 is healthy: successfully committed proposal: took = 1.769565ms

# docker exec etcd /bin/sh -c "export ETCDCTL_API=3 && /usr/local/bin/etcdctl endpoint status"
<IP_ADDRESS>:2379, 8e9e05c52164694d, 3.0.4, 25 kB, true, 2, 219
```

#### Run a etcd cluster with three nodes

```
# export ETCD_NODE1="<IP_ADDRESS>" && export ETCD_NODE2="<IP_ADDRESS>" && export ETCD_NODE3="<IP_ADDRESS>"
# export ETCD_NAME1="etcd1" && export ETCD_NAME2="etcd2" && export ETCD_NAME3="etcd3"
# export ETCD_CLUSTER_TOKEN="etcd-cluster"
# export ETCD_IMAGE="quay.io/coreos/etcd:v3.0.4"

# docker run -d \
 --name etcd \
 --restart always \
 --net host ${ETCD_IMAGE} /usr/local/bin/etcd \
 -name ${ETCD_NAME1} \
 -advertise-client-urls http://${ETCD_NODE1}:2379,http://${ETCD_NODE1}:4001 \
 -listen-client-urls http://<IP_ADDRESS>:2379,http://<IP_ADDRESS>:4001 \
 -initial-advertise-peer-urls http://${ETCD_NODE1}:2380 \
 -listen-peer-urls http://<IP_ADDRESS>:2380 \
 -initial-cluster-token etcd-cluster-1 \
 -initial-cluster ${ETCD_NAME1}=http://${ETCD_NODE1}:2380,${ETCD_NAME2}=http://${ETCD_NODE2}:2380,${ETCD_NAME3}=http://${ETCD_NODE3}:2380 \
 -initial-cluster-state new

# docker run -d \
 --name etcd \
 --restart always \
 --net host ${ETCD_IMAGE} /usr/local/bin/etcd \
 -name ${ETCD_NAME2} \
 -advertise-client-urls http://${ETCD_NODE2}:2379,http://${ETCD_NODE2}:4001 \
 -listen-client-urls http://<IP_ADDRESS>:2379,http://<IP_ADDRESS>:4001 \
 -initial-advertise-peer-urls http://${ETCD_NODE2}:2380 \
 -listen-peer-urls http://<IP_ADDRESS>:2380 \
 -initial-cluster-token etcd-cluster-1 \
 -initial-cluster ${ETCD_NAME1}=http://${ETCD_NODE1}:2380,${ETCD_NAME2}=http://${ETCD_NODE2}:2380,${ETCD_NAME3}=http://${ETCD_NODE3}:2380 \
 -initial-cluster-state new

# docker run -d \
 --name etcd \
 --restart always \
 --net host ${ETCD_IMAGE} /usr/local/bin/etcd \
 -name ${ETCD_NAME3} \
 -advertise-client-urls http://${ETCD_NODE3}:2379,http://${ETCD_NODE3}:4001 \
 -listen-client-urls http://<IP_ADDRESS>:2379,http://<IP_ADDRESS>:4001 \
 -initial-advertise-peer-urls http://${ETCD_NODE3}:2380 \
 -listen-peer-urls http://<IP_ADDRESS>:2380 \
 -initial-cluster-token etcd-cluster-1 \
 -initial-cluster ${ETCD_NAME1}=http://${ETCD_NODE1}:2380,${ETCD_NAME2}=http://${ETCD_NODE2}:2380,${ETCD_NAME3}=http://${ETCD_NODE3}:2380 \
 -initial-cluster-state new
```

Verification:
```
# docker exec etcd /bin/sh -c "/usr/local/bin/etcdctl cluster-health"
member 8e9e05c52164694d is healthy: got healthy result from http://<IP_ADDRESS>:2379
cluster is healthy

# docker exec etcd /bin/sh -c "/usr/local/bin/etcdctl member list"
8e9e05c52164694d: name=etcd peerURLs=http://localhost:2380 clientURLs=http://<IP_ADDRESS>:2379,http://<IP_ADDRESS>:4001 isLeader=true

User V3 API to do verification:
# docker exec etcd /bin/sh -c "export ETCDCTL_API=3 && /usr/local/bin/etcdctl endpoint health"
<IP_ADDRESS>:2379 is healthy: successfully committed proposal: took = 1.769565ms

# docker exec etcd /bin/sh -c "export ETCDCTL_API=3 && /usr/local/bin/etcdctl endpoint status"
<IP_ADDRESS>:2379, 8e9e05c52164694d, 3.0.4, 25 kB, true, 2, 219
```

### Supportor
Wang Yong Qiao (Weichat: gradyYQwang / Email<EMAIL_ADDRESS>

repository: gradywang/notesofyongqiao

repo_name: gradywang/notesofyongqiao