text:
Change Log
==========

## gtools-1.10.1 (2022-12-05)

### Bug fixes

- `gstats transform [if] [in], replace nogreedy` not allowed (since nogreedy
  reads groups on the fly it can't initialize the entire variable).

## gtools-1.10.0 (2022-12-04)

### Features

- `noinit` option for `gcollapse, merge`, `gegen`, `gstats` (selected),
  `gregress` (and co.) to prevent targets from being emptied out
  with `replace`. Prints warning!

### Enhancements

- `gtop` prints the number of levels in Other and Missing rows by
  default. (With missing it only does it if there's more than
  one type of missing value.)

- `greshape` tries to detect repeated stubs and suggests this possibility
  to the user when a stub matches multiple variables.

### Bug fixes

- Closes #87: For OSX, make now compiles x86_64 and arm64 separately
  then combines via `lipo`.

- `geomean` (`gcollapse`, `gegen`, `gstats`) no longer exhibits
  inconsistent behavior with zeros and negative values. A negative value
  results in a missing value evne if there are zeros present (internally,
  if a zero is encountered the loop just checks for negative values and
  returns 0 if it finds none).

- `gstats winsor`, exits with error if replace and if/in are passed
  (the way it's set up it'd be a bit of a hassle to allow init/noinit).

- `gstats transform`, `gstats hdfe`, `gregress` (and co.) all now
  initialize their targets to be empty (missing values) with
  if in and replace.

- `gstats transform` now exits with error if excludeself is passed
  without range, and now prints a warning if it's passed with
  stats other than range.

- `gtop` no longer incorrectly replaces the display value if the
  numerical variable has a value label and no missing values. If there
  was a single value this would result in an error: `gtop` would think
  there was always at least one missing value to replace.

- `gcollapse` no longer fails when trying to label the collapsed output
  if the source labels are blank (this can happen for example with data
  transformed to `.dta` from other formats or programs).

- `gcollapse` no longer gives incorrect missing variables list when
  part of that list is called with varlist notation (e.g. `x* y`
  and `x*` exist but `y` does not).

- `gunique` no longer ignores if/in with `gen` and `replace`

## gtools-1.9.2 (2022-09-22)

### Bug fixes

- Fixed incorrect results with excludeself without specified range in `gstats transform`
  (if there were missing values the input buffer was not being copied, but replaced).

### Features

- Faster excludeself mean and sum without specified range in `gstats transform`.

## gtools-1.9.1 (2022-03-28)

### Features

- Adds various algorithms for projection (cg, squarem, it, map) to
  `gstats hdfe`, `gregress`, `givregress`, `gglm`.
- Some ancillary options (`tolerance()`, `traceiter`, `standardize`)
- Parallel execution of select functions can be enabled at compile
  time via `GTOOLSOMP`
- Typed (direct/non-hashed) radix sort to API internals

## gtools-1.9.0 (2022-03-14)

### Features

- Add `gstats hdfe` (alias `gstats residualize`) for residualizing
  variables (i.e. HDFE transform).

## gtools-1.8.4 (2022-02-02)

### Enhancements

- Move plugin compilation to github rather than travis.

### Bug Fixes

- `gregress` and accompanying commands were not parsing absorb variable types
  since version 1.8.2! The relevant code had been commented out. In effect
  all variables were being taken to be integers.
- `gregress` and accompanying commands with HDFE no longer exit prematurely if
  there are redundant absorbed FEs next to each other.
- Closes #85: Bug in `gegen` warning message causes errors in some fun calls.
- Closes #84, #83, #73: OSX compilation moved to github.

## gtools-1.8.3 (2021-11-03)

### Bug Fixes

- Closes #82: `cw` in `gcollapse` now working.

## gtools-1.8.2 (2021-08-29)

### Features

- Add option to save residuals and predict (the behavior of `predict()` is not
  consistent at the moment; use with extreme caution).

### Bug Fixes

- Closes #79: Adds disclaimer to benchmarks.

## gtools-1.8.1 (2021-08-27)

### Bug Fixes

- Closes #78: if now passed raw/in double-quotes throughout the pipeline
- Closes #75: gunique returns 0s in r() when there are no obs
- Closes #74: gstats transform parses abbreviated targets
- Closes #72: Warning for gegen expressions without by group
- Fixed GLM issues generating de-meaned variables
- Fixed gegen nunique with multiple inputs
- Fixed bug in `gpoisson` where internal weights not copied correctly in loop.
- Various fixes to the docs.

## gtools-1.8.0 (2021-06-15)

### Features

- Added `gglm` to estimate GLM models, including `logit`.

## gtools-1.7.5 (2020-04-18)

### Bug fixes

- Fixed bug where `gstats range (select)` returned `0` instead of
  missing when all values in the input vector were missing.

## gtools-1.7.4 (2020-04-14)

### Enhancements

- User must now specify global `GTOOLS_BETA` to use beta features.

### Bug fixes

- Fixed bug where the prefix in `gstats` was `stat_` instead of `stat|`

## gtools-1.7.3 (2020-01-30)

### Bug fixes

- `geomean` in `gcollapse`, `gegen`, and `gstats` now returns missing
  if source has any negative values (complex numbers not supprted and
  checking special cases when the function might be defined given
  possibly negative input was too complicated). Further, if any input
  is zero the function now returns zero (previous behavior was not
  well-defined).

## gtools-1.7.2 (2020-01-26)

### Features

- Closes #69. `greshape wide/spread` now allows `labelformat()` for
  custom variable labels (only when a single variable is passed to
  `key()/j()`). The default is `#keyvalue# #stublabel#`.  Available
  placeholders are `#stubname#`, `#stublabel#`, `#keyname#`,
  `#keylabel#`, `#keyvalue#`, and `#keyvaluelabel#`

### Enhancements

- Closes #68. `gegen` now allows `by:` prefix when calling a
  `gstats transform` function (this is only allowed because these calls
  already require single-variable input, so the `by:` prefix should not
  present an issue when calling the function).

## gtools-1.7.1 (2020-01-26)

### Bug Fixes

- In `gquantiles`, data was read incorrectly with `by()` and `weights`
  if `xtile` was not requested. In particular, the data was copied as if
  the target had only one column, but since weights need to be included,
  the target has two columns. This was fixed.

## gtools-1.7.0 (2020-01-24)

### Enhancements

- In `givgregress`
    - Values to missing if model is not identified (not enough instruments).
    - `hdfe` no longer saves projected vars.

### Features

- In `gregress` (including `givregress` and `gpoisson`) collinear
  columns are now dropped (LDL' decomposition).
- `gegen shift, shiftby(#)` and `gstats transform (shift #)` are available
  to compute lags (negative `#`) and leds (positive `#`).

## gtools-1.6.4 (2019-09-14)

### Enhancements

- Collinearity and singularity checks in regression models
- `colmajor_ix` and related functions deleted.

- Adds `cumsum`, `cumsum +/-`, and `cumsum +/- varname` to `gstats transform`;
  options can also be passed to `cumsum` globally via `cumby()`

### Bug fixes

- Fixes bug in `greshape, dropmiss` where if the number of remaining
  observations is lower than the observations in memory, `set obs` would
  be run, resulting in an error. Now `keep in` is run for that case.

### Enhancements

- Removed rowmajor option from `gregress`

## gtools-1.6.2 (2019-08-25)

### Features

- Adds `givregress` computed via 2SLS

### Bug fixes

- Fixed bug in `gf_regress_linalg_dgemm_colmajor()`, defined in
  `src/plugin/regress/linalg/colmajor.c`:  Matrix multiplication
  now correct when axes are of different size. This bug does not
  affect previous versions because this function was only used
  to multiply square, albeit non-symetric (but that was not the issue),
  matrices.

### Enhancements

- Splits `gregress`, `gpoisson`, and `givregress`. While they
  are all run by `sf_regress` internally, they are three distinct
  commands.

- Categorize documentation into "Data manupulation", "Statistics",
  and "Regression models".

## gtools-1.6.1 (2019-08-21)

### Features

- Adds `gpoisson` computed via IRLS
- Adds weights to `gregress` and `gpoisson`

### Bug fixes

- Fixed cluster regression bug where `X` was passed instead of `xptr`

### Enhancements

- Modularized the code base so that aliases are assigned to internal functions
  instead of the copy/paste if/else branching statements.

## gtools-1.6.0 (2019-08-18)

### Features

- Adds `gregress`  (alias `greg`) with regressions by group, including
  robust SE, cluster SE (one-way or nested only), and absorb several FE
  (including HDFE, although the algorithm is inefficient at the moment).

## gtools-1.5.11 (2019-08-04)

### Features

- Fixes #67; adds `gegen x = rank(varname) [wgt], by(varlist) ties(type)`
  via `gstats transform (rank) [wgt], by() ties()`. Weights are optional.

### Bug Fixes

- Fixed bug where a by variable being used as a source but not a
  target got renamed to the target and was no longer available as a by
  variable. Now a new variable should be created and the by variable
  remains unchanged.

## gtools-1.5.10 (2019-08-03)

### Bug Fixes

- Fixed memory leak where the C by variables were not cleared from memory
  of `st_into->output` was allocated because free code was upgraded from 6
  or 7 to 9. Conditional logic in place said that by variables should not
  be cleared if free code was greater than 7, but that was only meant to
  skip free code 8 and free code 9 in _some_ scripts, but not all. Code 8
  logic was deprecated and now by variables are allocated with code 8, so
  they are always clared if free code is 6 or higher.

## gtools-1.5.9 (2019-06-22)

### Bug Fixes

- `by: gegen` now generates variables using the `by` prefix.
  This would give incorrect answers if the expression inside
  egen assumed that it would be generated with `by`. For example
  `by var: gegen x = mean(max(y, y[1]))`

## gtools-1.5.8 (2019-06-11)

### Features

- `gstats transform` can compute rolling/range statistics via
  `(range stat #[statlow] #[stathigh] [var])` or `(range stat)`
  with the `interval(#[statlow] #[stathigh] [var])` option. Access
  via `gegen` using `range_stat` with the `interval()` option.

- `gstats range` and `gstats moving` are aliases for `gstats transform`.
  They assume that every stat specified is a range stat or a moving
  stat, respectively.

- `gcollapse`, `gegen`, `gstats tab` now allow function `geomean`
  for the geometric mean.

## gtools-1.5.7 (2019-06-09)

### Features

- `gstats transform, auto[()]` allows automagically naming
  targets based on the source variable's name and the statistic
  requested. Default is `#source#_#stat#`.

- `gstats transform (moving stat lower upper)` computes moving
  window statistics. `gegen x = moving_stat(y), window(lower upper)`
  is an alias.

## gtools-1.5.6 (2019-06-08)

### Features

- New function `gstats transform` which applies a transformation to
  a variable; that is `y_i = f(x_i)`. For example,

      gstats transform (demean) y = x, by(group)

  gives

      n_j  = sum_i 1{group_i = j}
      s_j  = sum_i 1{group_i = j} * x_i
      y_ij = x_i - s_j / n_j

  available:

    normalize, standardize: f(x) = (x - mean(x)) / sd(x)
    demean:                 f(x) = (x - mean(x))
    demedian:               f(x) = (x - median(x))

- Closes #63: `greshape wide/gather` allows `prefix(...)` for custom
  output names.

- Closes #62: New stats available in `gegen`:
    - winsor, winsorize call `gstats winsor`
    - standardize, normalize, demean, demedian call `gstats transform`

### Enhancements

- `gunique, detail` now uses `gstats sum, detail`

### Bug fixes

- Closes #64: Removes `head` command from `greshape` tests (done a few
  commits ago but someone noticed before the merge).

## gtools-1.5.5 (2019-04-24)

### Features

- Allows option `dropmiss` to use multiple output stubs in both `long`
  and `long, nochecks`.

## gtools-1.5.4 (2019-04-16)

### Features

- Adds option `dropmiss` to drop missing observations when reshaping long
  (via `long` or `gather`).

## gtools-1.5.3 (2019-04-04)

### Enhancements

- Allows the user to specify the temporary directory for files via
  `global GTOOLS_TEMPDIR`

## gtools-1.5.2 (2019-03-25)

### Features

- Closes #58; allows `uselabels[(varlist, [exclude])]` to optionally
  specify which variables to use labels for (default is all variables).
  The user can also specify the option `exclude` to specify which
  variables _not_ to do this for.

## gtools-1.5.1 (2019-03-24)

### Features

- `greshape` supports `@` syntax for wide and long. Change the string
  to be matched via `match()`

- `greshape` supports stata varlist syntax for long to wide (may not be
  combined with `@` within a stub).

- `greshape` does not support varlist syntax for wide to long, but can
  use `match(regex)` for complex wide to long matches (see examples).

## gtools-1.5.0 (2019-03-23)

### Features

- Closes #57

- `glevelsof, mata[(name)]` saves the levels to mata. The levels are _not_
  stored in `r(levels)` and option `local()` is not allowed. With `silent`,
  the levels are additionally not formatted.

- `glevelsof, mata numfmt()` requires `numfmt` to be a mata print format
  instead of a C print format.

- `gtop, ntop(.)` and `gtop, ntop(-.)` now allow printing all the levels
  from largest to smallest or the converse.

- `gtop, alpha` sorts the top levels in variable order. if `gtop -var, alpha`
  is passed then they are sorted in reverse order.

- `gtop, mata` uses temporary files on disk to read the levels from C
  via mata. Matrices and locals are not used, meaning `r(levels)`,
  `r(toplevels)`, and the resuls stored via the option -matrix()-,
  ``r(`matrix')``, are no longer available. The user can access each
  of these via the mata object `GtoolsByLevels` (the user can change
  the name of this object via `mata(name)`). The levels are stored raw
  in `GtoolsByLevels.charx` and `GtoolsByLevels.numx`; the levels are
  stored formatted in `GtoolsByLevels.printed`; the frequencies are
  stored in `GtoolsByLevels.toplevels`.

- `r(matalevels)` stores the name of the mata object with the levels and frequencies.

- `gtop` also stores `r(ntop)`, `r(nrows)`, and `r(alpha)` as return scalars,
  for the numbere of top levels (if `.`, this will be `r(J)`), the number of
  rows in the `toplevels` matrix (it may or not include a row for "other" and
  a row for "missing"), and whether the top levels are sorted by their values.

- `gtop, mata numfmt()` requires `numfmt` to be a mata print format instead of
  a C print format.

## gtools-1.4.2 (2019-02-25)

### Enhancements

- `gstats sum/tab` allow saving their results in a mata object via
  `matasave[(name)]`. Prior versions hard-coded the object's name.

## gtools-1.4.1 (2019-02-24)

### Features

- `gstats sum` and `gstats tab` (alias `gstats summarize` and
  `gstats tabstat`) are a fast, by-able alternative to `sum, detail`
  and `tabstat`

- `gstats sum` or `gstats tab` with option `matasave` stores the output and
  by levels in `GstatsOutput`, an object of class `GtoolsResults`.

- `gcollapse` and `gegen` now allow the stats:
    - `select#`, `#`th smallest value (`select1` is the same as `min`)
    - `select-#`, `#`th largest value (`select-1` is the same as `max`)
    - `rawselect#` and `rawselect-#`, ibid but ignoring weights.
    - `cv`, coefficient of variation, `sd/mean`
    - `variance`
    - `range`, `max` - `min`

### Bug fixes

- `gcollapse` no longer crashes when `rawstat` does not match any entries.
- Fixes #54; incorrect checking for `greshape wide/spread` blank keys
- Fixes #55; allows `uselabels` to use labels as balues in `greshape gather`

### Enhancements

- `lgtools.mlib` added with come pre-compiled mata functios.

## gtools-1.3.5 (2019-02-19)

### Enhancements

- In `greshape`, levels of `key()`/`j()` are now saved _before_ being
  converted to variable names, meaning the labels preserve the names
  of the levels (up to the maximum variable label length in Stata).

## gtools-1.3.4 (2019-02-17)

### Bug fixes

- Critical bug fix in `gtop`; version 1.2.5 introduced a bug
  in `gtop` where the levels were incorrectly given.

- Minor bug fix in `gtop`; inverted levels were not correctly
  sorted with weights. The levels themselves were OK, however.

## gtools-1.3.3 (2019-02-15)

### Enhancements

- Removed locale as a dependency; comma printing done manually.

## gtools-1.3.2 (2019-02-14)

### Testing

- Hopefully fixes #40; might be related to #53

## gtools-1.3.1 (2019-02-11)

### Enhancements

- `greshape`'s preferred syntax is now `by()` and `keys()` instead
  of `i()` and `j()`; the docs and most of the printouts reflect this.

### Bug fixes

- `gtop`, `glevelsof`, and `gcontract` parse wildcards before
  adding any temporary variables, ensuring the latter don't
  get included in internal function calls.

- `greshape` no longer crashes if there are missing values in
  `keys()`/`j()`; the mata function `strtoname` converted "" into "";
  now I force missing to be converted to `_`.

- `glevelsof` and `gtop` should handle embedded characters better.
  Printing is still a problem but they get copied to the return
  values properly.

## gtools-1.3.0 (2019-02-08)

### Features

- `greshape long` and `greshape wide` are a fast alternative to
  reshape. It additionally implements `greshape spread` and `greshape gather`,
  which are analogous to the `spread` and `gather` commands from R's `tidyr`.

### Enhancements

- Added docs, examples `greshape` and `gstats`

### Bug fixes

- Fixes #49

## gtools-1.2.7 (2019-01-23)

### Enhancements

- Deleted all ancillary files and code related to `spookyhash.dll`

## gtools-1.2.6 (2019-01-19)

### Enhancements

- SpookyHash code compiled directly as part of the plugin. Might fix #35

## gtools-1.2.5 (2019-01-19)

### Enhancements

- Faster hash sort with integer bijection (two-pass radix sorts for
  smaller integers; undocumented option `_ctolerance()` allows the user
  to force the regular counting sort).

- Faster index copy when every observation is read (simply assign the
  index pointer to `st_info->index`)

## gtools-1.2.4 (2019-01-05)

### Bug fixes

- Stata 14.0 no longer tries to load SPI version 3 (loads version 2).
- Fixes #51: `gstats winsor` no longer fails with percentiles not strictly
  between 0 and 100.

## gtools-1.2.3 (2018-12-22)

### Bug fixes

- Fixes #50: `gstats winsor` no longer fills in missing values as the
  high percentile.

## gtools-1.2.2 (2018-12-18)

### Features

- `gstats winsor` now takes weights.

## gtools-1.2.1 (2018-12-16)

### Bug fixes

- `gstats winsor` no longer crashes with multiple variables.
- `gstats winsor, trim` correctly copies the source to the target.

## gtools-1.2.0 (2018-12-16)

### Features

- `gstats` general-purpose wrapper for misc functions.
- `gstats winsor` for Winsorizing and trimming data.

## gtools-1.1.2 (2018-11-16)

### Enhancements

- Improved variable parsing in general.
- Error message when variable not found now explicit.
- If `-` is found, warning message noting the default is to
  interpret that as negative, not part of a varlist.
- `ds` and `nods` control parsing options for `-`

## gtools-1.1.1 (2018-11-14)

### Features

- `gcollapse, merge` and `gegen` now accept the undocumented option
  `_subtract` to subtract the result from the source variable. This is
  an undocumented option meant for advanced users, so no additional
  checks have been added. If you use it then you know what you're doing.

## gtools-1.1.0 (2018-11-02)

### Features

- `gcollapse (nmissing)` counts the number of missing values (weights
  allowed).

### Enhancements

- `gcollapse (nansum)` and `gcollapse (rawnansum)` preserve missing
  value (NaN) information: If all entries are missing, the output is also
  missing (instead of 0). This is a more flexible version of the previous
  implementation, `gcollapse, missing`.

### Bug fixes

- `gisid` no longer gives wrong results when the data is partially
  ordered. Partially (weakly) sorted data would be incorrectly counted
  as totally sorted.
- `gcollapse (rawsum)` gives 0 if all entries are missing.
- `gcollapse` and `gegen` correctly parse types with weights for counts
  and sums. This includes `gcollapse, sumcheck`
- Closes #45
- Closes #46
- Closes #47

## gtools-1.0.7 (2018-10-27)

### Features

- Added option `sumcheck` to create sum targers from integer source
  as the smallest type that reasonable given the total sum.
- Closes #44

### Enhancements

- Recast upgrade (bug fix from 1.0.5) now done per-variable.
- Added about section to readme, with some plugs (good idea? bad idea?)

## gtools-1.0.6 (2018-09-25)

### Enhancements

- All the help files (and the readme) ask the user to run `gtools, upgrade`

## gtools-1.0.5 (2018-09-20)

### Bug fixes

- `gcollapse` no longer gives wrong results when `count` is the first of
  multiple stats requested for a given source variable. Previous versions
  wrongly recast the source as long.

## gtools-1.0.4 (2018-09-16)

### Enhancements

* Typo fixes and improvements to gtools help file.

## gtools-1.0.3 (2018-08-18)

### Bug fixes

* Gtools exits with error if `_N > 2^31-1` and points the user to the
  pertinent bug report.

## gtools-1.0.2 (2018-08-08)

### Enhancements

* Specify that you can install via `ssc` and upgrade via `gtools, upgrade`
* Can selectively specify tests in `gtools_tests.do`

### Bug fixes

* Fixed minor bug in `gtools.ado` (`else exit {` is now `else {`)

## gtools-1.0.1 (2018-07-23)

### Bug fixes

* `gegen, replace` gave the wrong result if the source was also the target
  (`tag` and `group` worked, but all other stats did not).
* `gunique, by()` was affected by this bug and now works correctly.
* `gcollapse, merge` exits with error if there are no observations matching `if' `in'.

### Enhancements

* `gunique, by()` no longer prints the top unique levels by default unless
  the user asks for them via `detail`.

## gtools-1.0.0 (2018-07-21)

First official release! This will go on SSC once all the tests have
passed. Final TODOs:

[X] Tests from github 0.14.2 tag
    [X] Linux with Stata 13
    [X] Windows with Stata 13
    [X] Windows with Stata 15
    [X] Linux with Stata 15
    [X] OSX with Stata 14
[X] Read the documentation (Stata)
[X] Read the documentation (online)

### Enhancements

* `gtools, license v` prints the licenses of the open source projects
  used in gtools.

### Bug fixes

* Fixed typos in test script
* `gcollapse, merge` no longer replces targets without `replace`
* `glevelsof, gen(, replace)` no longer conflicts with `replace`

------------------------------------------------------------------------

## gtools-0.14.2 (2018-07-21)

Feature freeze!

### Features

* `gtop` (and `gtoplevelsof`) now accept weights.
* `glevelsof` has option `gen()` to store the levels in a variable.
* `glevelsof` has option `nolocal` to skip storing the levels in a local.

### Enhancements

* Added weights to the quick showcase in the README
* Added `gen()` to the quick `glevelsof' showcase
* Added `gen()` to the `glevelsof' usage page and examples.
* Added tests for `gtop [weight]` and `glevelsof, gen()`

### Bug Fixes

* `gegen()` with `replace` sets missing values outside `if in` range.

## gtools-0.14.1 (2018-07-19)

### Features

* `fasterxtile` and `gquantiles` now accept weights (including `by()`)
* Note that stata might not handle weights accurately in pctile
  and xtile (or, at the very least, it does not seem to follow its
  documented formulas in all cases).
* Option `compress` tries to recast strL variables as str#
* Option `forcestrl` ignores binary values check (use with caution).

### Bug Fixes

* `semean` returns missing with fewer than 2 observations.
* If `strL` contains binary data `gtools` functions now throw an error.
* `strL` missing values now read correctly.
* `strL` partial support for OSX (long strings only).
* Added `strL`-specific tests
* All tests passing in Stata 13 in Linux, Windows
* All tests passing in Stata 15 in Linux
* All tests passing in Stata 14 in OSX

## gtools-0.14.0 (2018-07-17)

### Bug Fixes

* Closes https://github.com/mcaceresb/stata-gtools/issues/39 `strL`
  partial support (OSX pending).
* Closes https://github.com/mcaceresb/stata-gtools/issues/41 `wild` with
  existing variables gets a warning.
* Fixes https://github.com/mcaceresb/stata-gtools/issues/42 `gunique` typo.

------------------------------------------------------------------------

## gtools-0.13.3 (2018-05-06)

### Features

* Adds `gduplicates` as a replacement of `duplicates`. This is
  basically a wrapper for `gegen tag` and `gegen count`.

### Enhancements

* `_gtools_internals.ado` exits when `GTOOLS_CALLER` is empty.

## gtools-0.13.2 (2018-05-03)

### Features

* Fixes https://github.com/mcaceresb/stata-gtools/issues/31
  (added option `mlast` to `hashsort` to imitate `gsort` when sorting
  numbers in inverse order and they contain missing values).

## gtools-0.13.1 (2018-05-02)

### Pending

- For now, throws error for strL variable: https://github.com/mcaceresb/stata-gtools/issues/39

Plugin version 2.0 makes no mention of strL variables; however, plugin
version 3.0 notes that they are actually not supported by the macros
that I use to access strings. I think I'll have to compile a sepparate
version of the plugin for Stata 13 and Stata 14 and above. The Stata
13 version will throw an error for strL variables. This is currently
the only version until I figure out where to test Stata 14 and how to
implement this switch.

### Bug fixes

- Fixes https://github.com/mcaceresb/stata-gtools/issues/38

When the quantile requested was N - 1 out of an array of length N, I
makde an exception so the function picked up the largest value, instead
of passing N - 1 to the selection algorithm. However, I made a mistake
and quantiles between 100 - 150 / N and 100 - 100 / N, left-inclusive
right-exclusive, would give the wrong quantile.

## gtools-0.13.0 (2018-04-24)

### Enhancements

- Added some basic debugging code and comments to the code base.

### Bug fixes

- `sd`, `semean` give the correct answer when the group is a singleton
   or when all observations are the same.
- `skew`, `kurt` give the correct answer when the group is a singleton
   or when all observations are the same.

------------------------------------------------------------------------

## gtools-0.12.8 (2018-04-23)

### Features

* Added `rawsum`
* Added option `rawstat()`; you can pass a list of targets for which
  weights will be ignored. `percent` cannot be called with `rawstat`.
  All targets must be named _**explicitly**_ (i.e. will not expand
  varlist notation). Fixes https://github.com/mcaceresb/stata-gtools/issues/37

Neither rawsum nor rawstat are very smart. If the user requests them
without weights, they will be ignored without warning. If the user
requests them with weights, the weighted version will still be called
(weighted internals are slower than unweighted internals).

### Bug fixes

- When not weighted, `skew` and `kurt` return missing when all
  observations are the same. When weighted, they may return -1 or 1
  due to numerical (im)precission problems. This issue is also present
  in Stata's implementation and should only come up when working with
  doubles rounded to arbitrary decimal places.

## gtools-0.12.7 (2018-04-05)

### Bug fixes

* Added OSX plugin in build.
* Enclosed various macros in `""' in case they contain quotations.

## gtools-0.12.6 (2018-03-31)

### Features

* Added skewness and kurtosis to `gcollapse` and `gegen`

### Bug fixes

* Fixed install issues in https://github.com/mcaceresb/stata-gtools/issues/36

## gtools-0.12.5 (2018-03-06)

### Bug fixes

* Fixed install issues in https://github.com/matthieugomez/benchmark-stata-r/issues/6
* Fixed `gegen` type issue in https://github.com/mcaceresb/stata-gtools/issues/34

## gtools-0.12.4 (2018-02-06)

### Enhancements

- `gegen` always created a temporary variable in an attempt
  to fix https://github.com/mcaceresb/stata-gtools/issues/33,
  which was unnecessary and it slowed it down.
- Updated typos in documentation.

## gtools-0.12.3 (2018-02-01)

### Enhancements

- Linux and Windows tests passing with updated weights and nunique
- Updated documentation.

### Bug fixes

- Weights used the incorrect rules for outputting missing values
  when computing `sd` or `se*`; this was fixed.
- Fixes https://github.com/mcaceresb/stata-gtools/issues/33 and now
  tries to parse and expression first and a varlist second.

## gtools-0.12.2 (2018-02-01)

### Enhancements

- OSX version of gcollapse, gegen with weights and nunique
- Travis-compiled; I won't commit to master w/o passing tests

## gtools-0.12.1 (2018-01-22)

### Features

- gcontract supports fweights (Unix only; tests pending)
- gcollapse and gegen support nunique (Unix only; tests pending)
- gcollapse and gegen now support weights in Windows and Unix
  (OSX pending). gegen does not support weights with multiple
  sources (e.g. `sum(x y z)`).

------------------------------------------------------------------------

## gtools-0.11.5 (2018-01-16)

### Features

- gcollapse now allows wild-parsing via option `wild`
  (`wildparse`). E.g. `(sum) x* (mean) mean_x* = x*` would
  be valid syntax.

## gtools-0.11.4 (2018-01-08)

### Features

- `gcollapse, missing` output missing for sums if all inputs are missing.
- `glevelsof` adds option `unsorted` which is faster and does not sort results
- The resulting data from `gcollapse` and `gcontract` are now sorted

### Bug fixes

- `gegen` now imitates `egen` when there are no observations selected
  and leaves the new variable in memory with all missing values.
- `gegen, missing` for total and sum now correctly output missing if all
  inputs are missing
- Fixes https://github.com/mcaceresb/stata-gtools/issues/32 (gcontract
  should allow abbreviations)

## gtools-0.11.3 (2017-11-21)

### Bug fixes

- `gtop` now reads a global from `gtoplevelsof` so if there was an error
  but the exit code should be 0 (in this case, no observations throws an
  error internally but exits with 0 status externally) `gtop` no longer
  throws a syntax error.
- Fixed bug in gdistinct where all missing observations without the
  missing option caused an error.

## gtools-0.11.2 (2017-11-20)

### Bug fixes

- Fixed bug in glevelsof where if string data is already sorted the
  system does not compute the length of the de-duplicated string
  data. This caused malloc to attempt to assign the length of an
  unassigned but declared variable. This treated an object's address as a
  length and caused malloc to request dozens or hundreds of GiB and crash
  on some systems.

## gtools-0.11.1 (2017-11-20)

### Bug fixes

- Fixed bug with naming conflicts between internals and gcollapse.

## gtools-0.11.0 (2017-11-17)

### Backwards Incompatible

- `gquantiles` no longer supports option "binpct[()]"
- `glevelsof` no longer supports option "silent"

### Features

- `gquantiles` supports `by()`

- Option `hash()` gives user control over hashing mechanics.  0 is
  default (internals choose what it thinks will be fastest), 1 tries to
  biject, 2 always calls spookyhash.

### Enhancements

- Improved internal handling of numerical precision issues when
  computing quantiles (gquantiles, gcollapse, and gegen).

- When the data is sorted, groups are indexed directly instead of using
  the hash. This was 2x faster in testing for an overall ~33% or so speedup.

- `gtop` is a shortcut for `gtoplevelsof`

- `bench(2)` and `bench(3)` offer the user more control over what
  benchmark steps are displayed.

### Bug fixes

- Fixes possible bug with levelsof where the plugin tries to read
  more characters than contained in the numfmt local macro. This
  could cause problems on some systems.

------------------------------------------------------------------------

## gtools-0.10.3 (2017-11-12)

### Bug fixes

- Fixes https://github.com/mcaceresb/stata-gtools/issues/29; if the
  sources appear out of order relative to the targets (e.g. sources are
  'a b' but targets use sources 'a a b b' instead of 'a b a b') then
  gcollapse produced the wrong results with option `forceio` or with the
  swtich code. The code now reorders the sources, targets, and statistics
  so the source variables always appear first and the extra targets last.
- Fixes bug in `gtop` where requesting negative levels caused an overflow.
- Fixes bug in `hashsort` where there may be a loss in numerical
  precision when clearing the sort var. It now adds one observation to the
  source data, changes the dummy observation, and drops it. This should
  have the same effect without modifying the original dataset.

### Enhancements

- Counting sort now uses pointers, which is hopefully faster.
- Added special cases for `gquantiles` to hopefully accelerate reading
  source variables from Stata.
- Added special case for `gegen, group` so the code orders the groups
  back to Stata order before copying. It runs faster because it writes
  to Stata sequentially, but it uses more memory.
- The code no longer keeps a copy of the by variables when it is not
  needed (gcollapse, glevelsof, gtop, and gcontract need a copy;
  gegen, hashsort, gquantiles, etc. do not).

## gtools-0.10.2 (2017-11-08)

### Bug fixes

- Fixes bug where integer overflows if `gtoplevelsof, ntop(-#)`
  is requested.

## gtools-0.10.1 (2017-11-08)

### Features

- `gquantiles` is a fast alternative to `_pctile`, `pctile`, and
  `xtile` with several additional features.
- `fasterxtile` is an alias for `gquantiles, xtile`

### Bug fixes

- Fixes https://github.com/mcaceresb/stata-gtools/issues/27
- Fixed numerical precision issue with quantiles.
  Now ((ST_double) N / 100) is computed first.

------------------------------------------------------------------------

## gtools-0.9.4 (2017-11-03)

### Bug fixes

- Fixes https://github.com/mcaceresb/stata-gtools/issues/26
  so that hashsort sets `sortedby` to the outermost levels that
  are positive (e.g. `foreign -rep78` sets `sortedby` to `foreign`).

### Backwards Incompatible

- Apparentlly `gen` in `gsort` was intuitively specified, so I have
  now made `gen` mirror that in `hashsort`. Previous versions offered
  this functionality via `group`.

## gtools-0.9.3 (2017-11-02)

### Bug fixes

- Major bug in gisid: False positive when there was a duplicate row with
  multiple variable levels but data was otherwise sorted.

## gtools-0.9.2 (2017-11-02)

### Bug fixes

- Previous commit had a corrupted binary for the OSX plugin.

### Backwards Incompatible

- Option `benchmark` is no longer abbreviated with `b` and MUST be abbreviated
  as `bench`.  This is to ensure consistency with options `bench(0)`,
  `bench(1)` and `bench(2)`. `bench(1)` is the same as `benchmark`. `bench(2)`
  additionally shows benchmarks for various internal plugin steps. `bench(0)`
  is the same as not including `benchmark`.

### Enhancements

- Improved online docs: Now each example is available as raw code
  (each example page points to it) and inline monospaced code should
  be slightly bigger, improving readability.

## gtools-0.9.1 (2017-11-01)

- Stability improvements.
- Commands should be faster when data is sorted (skips
  hash sorting if data is already sorted).
- Misc bug fixes.

## gtools-0.9.0 (2017-11-01)

### Features

- The plugin now works on OSX
- `gcontract` is a fast alternative to `contrast`
- `gtoplevelsof` is a new command that allows the user to glean the most
  common levels of a set of variables.  Similar to `gcontract` with a `gsort`
  of frequency by descending order thereafter, but `gtoplevelsof` does not
  modify the source data and saves a matrix with the results after its run.
- `gdistinct` now saves its results to a matrix when there are multiple
  variables.
- Improved and normalized documentation

### Bug fixes

- OSX version; fixes https://github.com/mcaceresb/stata-gtools/issues/11
- `gisid` now sient w/o benchmark or verbose; fixes https://github.com/mcaceresb/stata-gtools/issues/20
- Added quotes to `cd cwd` in `gtools`; fixes https://github.com/mcaceresb/stata-gtools/issues/22
- `gcontract` available; fixes https://github.com/mcaceresb/stata-gtools/issues/23

------------------------------------------------------------------------

## gtools-0.8.5 (2017-10-30)

### Features

- `gcollapse, freq()` stores frequency count in variable.
  Same as `gen long freq = 1` and then `(sum) freq`.
- Hashsort is marginally faster (sortindex inversion now
  done internally in C).

## gtools-0.8.4 (2017-10-29)

### Enhancements

- Normalized types in the C base to ensure I have 64-bit integers
  (signed and unsigned) as well as `ST_doubles` all around.
- Bijection limit is now limit on signed 64-bit integers.

## gtools-0.8.3 (2017-10-28)

### Features

- `gisid` includes an internal check to see if the data is sorted.
  This means that if there are two duplicate rows in unsorted data
  or if the data is already sorted, `gisid` will give a result much
  faster. However, if the data is not sorted it will be marginally
  slower as it will execute the rest of the code normally.

### Backwards Incompatible

- `gisid` and `hashsort` are no longer rclass. Both will exit early if
  the data is already sorted, and `gisid` will also exit if it finds a
  duplicate row during the sorted check. Hence they will not always
  store results, making them inconsistent. It would be bad practice
  to continue to have them store `rclass` results.

### Enhancements

- Cleaned up the C base somewhat. Improved modularity.
- All `int` were changed to `size_t` or `int64_t` as applicable, since
  `int` is not necessarily aliased to a 64-bit integer on all platforms.

## gtools-0.8.2 (2017-10-26)

### Features

* `gdistinct` is a replacement for `distinct`.  It is functionally identical
  to `gunique` except it mimics the output format of `distinct`.

## gtools-0.8.1 (2017-10-26)

### Backwards Incompatible

* `merge` now merges labels and formats by default

### Bug fixes

* Fxied examples in README; fixed minor typos in README
* `gegen` now handles the type of `pctile` correctly.

## gtools-0.8.0 (2017-10-19)

### Features

* Refactored code base for somewhat faster runtime, but mainly for bug fixes
  and ease of maintenance.
* gcollapse, gegen, glevelsof, gisid, gunique, hashsort all call the function
  `_gtools_internal`, which then calls the plugin.
* gcollapse includes a target labeling engine.
* gcollapse now supports semean, sebinomial, and sepoisson.
* gegen will take (almost) any egen function. If it is not implemented
  by gtools it will simply use hashsort and then call egen.
* Windows and Linux versions passing tests, OSX version should be feasible.

### Bug fixes

* Fixes https://github.com/mcaceresb/stata-gtools/issues/19
* Fixes https://github.com/mcaceresb/stata-gtools/issues/18
* Fixes https://github.com/mcaceresb/stata-gtools/issues/11

------------------------------------------------------------------------

## gtools-0.7.5 (2017-10-08)

### Features

* Updated benchmarks for new commands, all in Stata/MP.
* Added `counts` option (with `fill`) to `gegen group`.
  `fill` can be a number, `group` to fill in the counts
  normally, or `data` to fill in the first J_th_ observations.
* The number of groups is now stored in `r(J)`, along with
  other useful meta stats.

## gtools-0.7.4 (2017-09-29)

### Features

* `hashsort` is added as a working replacement for `sort` and `gsort`.
  The sort is always stable. `mfirst` is not allowed.

## gtools-0.7.2 (2017-09-28)

### Features

* `gisid` is added as a working replacement for `isid` and `isid, missok`.
  `gisid` taks `if` and `in` statements; however, it does not implement
  `isid, sort` or `isid using`.
* `glevelsof` is added as a working replacement for `levelsof`.
  All `levelsof` features are available.

### Enhancements

* Fixes https://github.com/mcaceresb/stata-gtools/issues/13 so
  `gcollapse` maintains source formats on targets.
* Improved internal handling of if conditions for `egen`.

### Bug fixes

* Prior versions de-facto used a 64-bit hash instead of a 128-bit hash.
  The new version should use the 128-bit hash correctly.
* Prior versions would fail if there was only 1 observation.

## gtools-0.7.1 (2017-09-27)

### Enhancements

* `egen` now only processes observations in range for `id, group`
* `egen, group` now marginally faster when all vars are integers

## gtools-0.7.0 (2017-09-26)

### Enhancements

* Temporary variable no longer created for `egen, tag` or `egen, group`
* Fixes https://github.com/mcaceresb/stata-gtools/issues/6
    * Variables are sorted internally for `egen, group`, which matches `egen`.
    * Variables are sorted internally for `gcollapse`, which is faster.
* Various internal enhancements:
    * The hash is validated faster
    * Hash validation is also used to read in group variables
    * Integer bijection now sorts by the integers correctly,
      obviating the need for a second sort.
    * No need to validate the hash with integer bijection.
    * The memory usage is marginally leaner.
    * Reorganized all the files, making the code-base easier to maintain.
* Various commented internal code deleted.

### Backwards-incompatible

* `gcollapse, unsorted` no longer supported (due to internal sorting)

------------------------------------------------------------------------

## gtools-0.6.17 (2017-09-17); fixes #15

### Bug fixes

* Fixes https://github.com/mcaceresb/stata-gtools/issues/15
  which was introduced trying to fix
  https://github.com/mcaceresb/stata-gtools/issues/15

## gtools-0.6.16 (2017-09-13); addresses #7

### Bug fixes

* Improves the issues raised by https://github.com/mcaceresb/stata-gtools/issues/7
  Now the commands only fail if Stata hits the `matsize` limit (internally, the plugin
  no longer uses the `subinstr` hack to go from locals to mata string matrices and uses
  `tokens` instead, which is more appropriate; further, the plugin tries to set matsize
  to at least the number of variables and gives a verbose error if it fails.)

## gtools-0.6.15 (2017-09-12); fixes #14, #9

### Bug fixes

* Fixes https://github.com/mcaceresb/stata-gtools/issues/14
* Fixes https://github.com/mcaceresb/stata-gtools/issues/9
  (prior fixes were erratic; this should work)

## gtools-0.6.14 (2017-09-12)

### Bug fixes

* No longer crashes on Linux systems with older glibc versions.

## gtools-0.6.13 (2017-09-12)

### Bug fixes

* Should fix https://github.com/mcaceresb/stata-gtools/issues/9
  Added legacy plugin using older `libgomp.so` library.

## gtools-0.6.12 (2017-09-11)

### Bug fixes

* Should fix https://github.com/mcaceresb/stata-gtools/issues/12

## gtools-0.6.11 (2017-08-17)

### Bug fixes

* Fixed https://github.com/mcaceresb/stata-gtools/issues/8 so
  `gegen` is callable via `by:`; it also gives the stat for the
  overall group if called without a `by`.

## gtools-0.6.10 (2017-06-27)

### Bug fixes

* When fixing issue https://github.com/mcaceresb/stata-gtools/issues/5
  I introduced a bug. This is fixed.

## gtools-0.6.9 (2017-06-27)

### Enhancements

* Addressed the possible issue noted in issue
  https://github.com/mcaceresb/stata-gtools/issues/3 and the functions now
  use mata and extended macro functions as applicable.
* `gegen varname = group(varlist)` no longer has holes, as noted in issue
  https://github.com/mcaceresb/stata-gtools/issues/4
* `gegen` and `gcollapse` fall back on `collapse` and `egen` in case there
  is a collision. Future releases will implement an internal way to resolve
  collisions. This is not a huge concern, as SpookyHash has no known
  vulnerabilities (I believe the concern raied in issue https://github.com/mcaceresb/stata-gtools/issues/2
  was base on a typo; see [here](https://github.com/rurban/smhasher/issues/34))
  and the probability of a collision is very low.
* `gegen varname = group(varlist)` now has a consistency test (though
  the group IDs are not the same as `egen`'s, they should map to the `egen`
  group IDs 1 to 1, which is what the tests now check for).

### Bug fixes

* `gegen` no longer ignores unavailable options, as noted in issue
  https://github.com/mcaceresb/stata-gtools/issues/4, and now it throws an error.
* `gegen varname = tag(varlist)` no longer tags missing values, as noted
  in issue https://github.com/mcaceresb/stata-gtools/issues/5
* Additional fixes for issue https://github.com/mcaceresb/stata-gtools/issues/1
* Apparentlly the argument Stata passes to plugins have a maximum length. The
  code now makes sure chuncks are passed when the PATH length will exceed the
  maximum. The plugin later concatenates the chuncks to set the PATH correctly.

## gtools-0.6.8 (2017-06-25)

### Bug fixes

* Fixed issue https://github.com/mcaceresb/stata-gtools/issues/1
* The problem was that the wrapper I wrote to print to the Stata
  console has a maximum buffer size; when it tries to print the
  new PATH it encounters an error when the string is longer than
  the allocated size. Since printing this is unnecessary and
  will only ever be used for debugging, I no longer print the PATH.

## gtools-0.6.7 (2017-06-18)

### Debugging

* Debugging issue https://github.com/mcaceresb/stata-gtools/issues/1
  on github (in particular, `env_set` on Windows).

## gtools-0.6.6 (2017-06-18)

### Bug fixes

* Removed old debugging code that had been left uncommented
* Improved out-of-memory message (now links to relevant help section).

## gtools-0.6.5 (2017-06-18)

### Features

* The function now checks numerical variabes to see if they are integers.
  Working with integers is faster than hashing.
* The function is now smarter about generating targets. In prior versions,
  when the target statistic was a sum the function would force the target
  type to be `double`. Now if the source already exists and is a float, the
  function now checks if the resultimg sum would overflow. It will only
  recast the source as double for collapsing if the sum might overflow, that
  is, if `_N * min < -10^38` or `10^38 < _N * max` (note +/- 10^38 are the
  largest/smallest floats stata can represent; see `help data_types`).

### Bug fixes

* Fixed bug where Stata crashes when it can no longer allocate memory. It now
  exists with error.
* In Windows, `gcollapse` and `gegen` now check whether `spookyhash.dll` can be
  found before trying to modify the `PATH` environment variable.

## gtools-0.6.4 (2017-06-18)

### Bug fixes

- On windows, when all variables are numeric and the second variable
  is constant, there used to be a division by 0 crash. This was fixed.

## gtools-0.6.3 (2017-06-18)

### Bug fixes

- Forgot to provide OS-specific versions of `env_set` as well...
- Linux and Windows versions passing from github.

## gtools-0.6.2 (2017-06-18)

### Bug fixes

- Forgot to commit new files in `./build`

## gtools-0.6.1 (2017-06-17)

### Bug fixes

- Program now installs correctly from build folder on Windows. I had
  to add the assumed `spookyhash.dll` path to the system `PATH` at
  eacah run. I also provide a `gtools.ado` file that allows the user to
  troubleshoot some possible issues that may arise.

### Misc

- Tried and failed to compile on OSX using Travis, but cleaned it up
  enough that it will be easier to compile once I get access to OSX, if
  ever. Windows version still OK.

## gtools-0.6.0 (2017-06-16)

### Features

- Windows version passing all tests; benchmarked on virtualbox.

### Bug fixes

- `gegen tag()` now gives the correct result when there are no obs that
  match the tag (e.g. all missing). Previous versions returned all
  missing values. The new version returns all 0s, matching `egen`.

### Known problems

* The multi-threaded version does not load on Windows. Getting this to
  work on Windows was painful enough that I have 0 plans to debug it at
  this time. The single-threaded version works fine, however, and is
  already plenty fast.

* The marginal time to add a variable to memory is non-linear. If there
  are 100 variables in memory, adding the 101th varaible will take
  longer than if there are 0 variables in memory and we are adding the
  first one. This is problematic because we try to estimate the time
  by benchmarking adding two variables. The non-linear relation is not
  obvious as it would depend on the user's system's RAM and CPU. Hence
  we simply scale the benchmark by K / 2.

* Stata's timer feature is only accurate up to miliseconds. Since adding
  the two variables for benchmarking is faster than adding marginal
  variables thereafter, occasionally Stata incorrectly estimates the
  time to add a variable to be 0 seconds. Empirically it does not bear
  out that adding variables after the benchmark variables takes more
  than 0 seconds. Hence we assume that Stata would actually take 0.001
  seconds to add 2 variables to memory.

### Planned

* Sort variables in C, not in Stata (high priority; performance)
* Allow merge with an if statement (low priority; feature).
* If you sort the data in C, then assert the sort is unique and
  print "(hashed correctly grouped observations: resulting sort is unique)"
* Allow `greedy` option to skip drops and recasting? (Depending on
  the implementation this may be slower because adding variables takes
  longer with more variables in memory.)

------------------------------------------------------------------------

## gtools-0.5.2 (2017-06-15)

### Misc

* Added Travis CI integration.
* Improved README (logo, cleaner flow, better compilation instructions).

### Bug fixes

- Properly added spookyhash as submodule.
- Compiles with Travis CI

## gtools-0.5.1 (2017-06-14)

### Bug fixes

* In prior versions, if `gcollapse` was called with no observations or
  when the result of `if in` gave no observations, the function throwed
  an error. Now the program exits and prints "no observations" to the
  console (`gcollapse` returns an empty data set; `gegen` returns a
  variable with all missing values).
* In prior versions, in some cases, when multiple statistics are
  generated from a single source and the summary statistics are
  collapsed to disk, the first statistic may be swapped for another.
  This happened because the function tries to use source variables and
  targets, and it also tries to be smart about which target statistic
  to use the source variable for. So if you request mean and min for
  a byte, the function will want outputs to be at least float and
  byte, and will use the source variable for the min and generate
  an additional variable for the mean. This had been implemented
  incorrectly when collapsing to disk.
* Added additional unit tests for collapsing to disk.

## gtools-0.5.0 (2017-06-14)

### Features

* The function tries to be smarter Be smart about memory management.
  When `merge` is not specified, N is larger than 1M, and there are
  more than 3 additional targets to create in memory, the function
  tries to figure out whether collapsing the data and writing the
  collapsed data for the extra targets to disk is faster than creating
  the targets in memory and collapsing to memory.

  While collapsing to memory is faster than collapsing to disk,
  generating variables in memory with N observations, before collapsing,
  is slower than for J observations, after collapsing. For J small enough
  (e.g. 10 groups vs 1M observations) it is more efficient to collapse
  to disk and read the data back in after. The rules
    * `merge` merges back to the original data, so we do not collapse to disk.
    * `forceio` forces the function to collapse to disk.
    * `forcemem` forces the function to collapse to memory.
    * Otherwise, if N > 1M and the number of variables to add is K > 3,
      the function benchmarks how long it would take C to write and
      read `K * 8 * J / 1024^2` MiB and then creating K variables
      in memory for J observations vs creating K variables for N
      observations.
    * Creating observations is memory is given the "benefit of the doubt"
      by a factor of 10. So we have to estimate writing to disk
      and creating variables after the collapse will be at least
      10 times faster for the swtich to take place.
* String by variables are now read into C and written back into Stata
  directly. Prior versions generated temporary variables to read string
  by variables into the temporary variables and then copied them back.
  This was inefficient.
* Moved away from using regexes in C code. Now the function just passes
  the quantile as a string directly from Stata and uses `atof`. This is
  marginally faster and eliminates a dependency.

### Backwards-incompatible

* All undocumented `mf_` options have been changed to `debug_`

### Bug fixes

* Option `debug_checkhash` seems to work properly now. This was fixed by
  zero-ing the string that was used as the temporary buffer for reading
  in Stata variables.

### Known problems

* The marginal time to add a variable to memory is non-linear. If there
  are 100 variables in memory, adding the 101th varaible will take
  longer than if there are 0 variables in memory and we are adding the
  first one.
* This is problematic because we try to estimate the time by
  benchmarking adding two variables. The non-linear relation is not
  obvious as it would depend on the user's system's RAM and CPU.
  Hence we simply scale the benchmark by K / 2.
* Stata's timer feature is only accurate up to miliseconds. Since adding
  the two variables for benchmarking is faster than adding marginal
  variables thereafter, occasionally Stata incorrectly estimates the
  time to add a variable to be 0 seconds. Empirically it does not bear
  out that adding variables after the benchmark variables takes more
  than 0 seconds. Hence we assume that Stata would actually take 0.001
  seconds to add 2 variables to memory.

### Planned

* Allow `greedy` option to skip drops and recasting? (Depending on
  the implementation this may be slower because adding variables takes
  longer with more variables in memory.)
* Sort variables in C, not in Stata (high priority; performance)
* Allow merge with an if statement (low priority; feature).
* If you sort the data in C, then assert the sort is unique and
  print "(hashed correctly grouped observations: resulting sort is unique)"

------------------------------------------------------------------------

## gtools-0.4.1 (2017-05-29)

### Bug fixes

* `gegen` now generates the expression passed to its functions if the
  argument is not a varlist.

## gtools-0.4.0 (2017-05-23)

### Features

* Somewhat faster: The function tries to choose the multi-threaded
  version of the plugin when available, and non-multi otherwise. It is
  also smarter about recasting variables.
* Various undocumented options to test and benchmark different
  algorithms for reading and collapsing the data.

### Bug fixes

* `gegen` now computes the first and last non-missing observations
  correctly when there is an if statement involved in the call.
* `gcollapse|gegen` now correctly finds first and last when there are
  missing values.
* `gcollapse|gegen` now correctly populates firstnm and lastnm with
   missing values when all observations are missing.
* `gcollapse|gegen` now output the sum as 0 of all missing values
  to mimic `collapse`.
* `benchmark` now correctly times parallel execution.

### Known problems

* `gcollapse` does not work correctly with `merge` when the user asks
  for an if statement, so when it is tried the function exits with
  error.
* Checkhash may give false positives when checking strings. It will
  occasionally read more data than necessary to make the comparison,
  resulting in a false positive. Working on a fix, but for now I moved
  the function to be undocumented, `mf_checkhash`

### Misc

* Updated benchmarks
* Cleaned up method for reading data in from Stata. Kept the sequential
  and the out-of-order methods (parallel out of order for the
  multi-threaded version).
* Normalized method of reading data from Stata in gegen.
* Added some undocumented options to control what is executed in
  parallel and what isn't when running with `multi`.

### Planned

* Allow merge with an if statement (low priority; feature).
* Sort variables in C, not in Stata (high priority; performance)
* If you sort the data in C, then assert the sort is unique and
  print "(hashed correctly grouped observations: resulting sort is unique)"
* Be smart about memory management when J is small relative to N.

------------------------------------------------------------------------

## gtools-0.3.3 (2017-05-21)

### Features

* Significantly sped up integer-only hash by computing the ranges in C,
  not in Stata.
* Data is now read sequentially from Stata. More extensive testing
  revealed this is usually faster.
* In the multi-threaded version, data is read in parallel. More extensive
  testing revealed this is usually faster.

### Planned

* Decide on a method to read in variables from stata; implement across
  all single and multi-threaded versions.

## gtools-0.3.2 (2017-05-20)

### Features

* Added `checkhash` option to check for hash collisions.

### Bug fixes

* Finished the benchmarks, both locally (IC) and on the server (MP)
* Normalized the execution of `gegen` and `gegen, multi`

### Misc

* Tested reading data from Stata sequentially (it's not faster)
* Improved README
* Various typo fixes in comments

## gtools-0.3.1 (2017-05-19)

### Bug fixes

* Speed up quantiles! (Also, all other functions). Encoded stat string
  to numeric: It's WAY faster to select a function based on a number
  than a string. String comparisons are expensive. I was also parsing
  the quantile string at EVERY function call, which was the bottleneck
  in computing quantiles, not quicksort.
* first and last observations now computed correctly when quantiles are
  also requested.
* Fixed `gtools.pkg`
* Added documentation for `gegen`
* Function names and calls closer to `egen`

## gtools-0.3.0 (2017-05-19)

### Features

* `gegen` implementation with all of `gcollapse`'s functions, as well as
  `total` as an alias for `sum`, `group` to generate an id for a given
  group, and `tag` to tag the first observation in a given group.
* `if` and `in` are correctly supported in `gegen`.
* Both `gegen` and `gcollapse` have multi-threaded versions.
* Note `gegen newvar = group(varlist)` tags the first appearance of a
  group as 1, and so on. `egen` sorts the data, so the group taged as 1
  is the 'first' group, and so on. If you want that behavior you should
  use `egen`; `gegen` is only suitable when you want to index the data
  and you don't mind that it won't be sorted right away.

### Bug fixes

* Fixed bug in hashing strings so it now passes the correct number of
  bytes to the spooky hash (in prior versions, string hashes were not
  consistent).
* Fixed bug in integer bijection so it can now handle missing values.
* Fixed bug in quantile function where the first comparison element was
  selected to be the first entry read from Stata, not the first group entry.

### Problems

* Only available when `c(os)` is Unix.
* Memory management is terrible. See previous notes or `README.md`
  section on memory management.
* Several by-able egen functions are missing
* Not provided
    * semean
    * sebinomial
    * sepoisson
    * rawsum

### Planned

* `checkmem` feature to call `fcollapse` or `collapse` when the plugin
  suspects there will not be enough memory.
* The rest of `gegen`'s by-able functions
* Improve memory management.

------------------------------------------------------------------------

## gtools-0.2.0 (2017-05-19)

### Features

* `gcollapse` `smart` option indexes the data based on the sorted groups
  so no hashing and sorting is necessary. It no longer calls `collapse`,
  which is slower.
* `gcollapse` provides `merge` to merge the collapsed data back with the
  original data. This is much fastere than collapsing and then merging.
* `gcollapse` provides `multi` to invoke the multi-threaded version of
  the collapse routine.
* Computing percentiles is less inefficient, but it is still inefficient.
  When this is particularly slow, the `multi` option can compensate.
* The code base is cleaner and the C code is (mostly) commented.

### Problems

* Only available when `c(os)` is Unix.
* Memory management is terrible. See previous notes or `README.md`
  section on memory management.
* Not provided
    * semean
    * sebinomial
    * sepoisson
    * rawsum

### Planned

* `checkmem` feature to call `fcollapse` or `collapse` when the plugin
  suspects there will not be enough memory.
* `gegen` as a drop-in replacement for `egen`'s by-able functions that
  should be faster than `fegen`.
* Improve memory management.

------------------------------------------------------------------------

## gtools-0.1.0 (2017-05-16)

### Features

* `gcollapse` provides a C-based near-drop-in replacement for `collapse`
  that is (almost always) several times faster than `fcollapse`.
* Though computing percentiles is currently very inefficient (see below),
  `gcollapse`, unlike `collapse` (or `fcollapse` as best I can tell) can
  compute quantiles, not just percentiles (e.g. p2.5, p15.25)

### Problems

* Only available `c(os)` is Unix.
* Quantile implementation is massively inefficient (via C's `qsort`). In
  fact, it it possible that `fcollapse` will be ~50% faster if there are
  many levels/sgroups (hundreds of thousands or millions) and more than
  a few percentiles (3 or more).
* Memory management is terrible. `fcollapse` does not consume too much
  more memory than collapse because mata can add and drop variables at
  requisite times. C, on the other hand, cannot add or drop variables
  in Stata, and target variables must already exit. This means that for
  every additional summary variable (i.e. 2 or more) based on a single
  source variable, Stata must create them before collapsing the data.
  Thus it is possible that systems that run `collapse` or `fcollapse`
  with ease are not able to run `gcollapse` due to the additional
  memory overhead. See the memory management section in `README.md` to
  estimate whether your system will be able to take advantage of the
  speed improvements of `gcollapse` if you are concerned about memory
  management.
* Not provided
    * semean
    * sebinomial
    * sepoisson
    * rawsum

### Planned

* `gegen` as a drop-in replacement for `egen`'s by-able functions that
  `should be faster than fegen`.
* Improve quantile implementation (perhaps using a better implementation
  of `quickselect`; my attempt at implementing quickselect was slowe
  than `qsort`).
* Improve memory management.

repository: mcaceresb/stata-gtools

gha_language: Stata

repo_name: mcaceresb/stata-gtools


text:
# Essen

[![GoDoc](https://godoc.org/github.com/akshitgrover/essen?status.svg)](https://godoc.org/github.com/akshitgrover/essen)
[![Go Report Card](https://goreportcard.com/badge/github.com/akshitgrover/essen)](https://goreportcard.com/report/github.com/akshitgrover/essen)

Essen is a golang based micro web framework, inspired by express.js framework for node.js.
It is in a  very early stage, with some request and response mapping, With time it will ripe.

## Why the name?

The time I started developing it I had that sensation of eating something, in german **essen** is **To eat** so the name :P :'|)

## 10 seconds to code:

```go
package main

import(
	"github.com/akshitgrover/essen"
)

func main(){
	e := essen.App()

	e.Use("/getpath",func(res essen.Response,req essen.Request){
		res.Send(200, "Hello World")
	}) // Use Method is for any request method

	// Similar Methods For Post And Get Request

	e.Listen(<port>) // Specify The PORT
}
```

## P.S.

This project is in very early stage, will be developing it further.

Will love any suggestions and ideas for the same.

## Copyright & License

[MIT License](./LICENSE)

Copyright (c) 2018 Akshit Grover

license: MIT

repository: akshitgrover/essen

branch_name: refs/heads/master

extension: md

license_type: permissive

repo_name: akshitgrover/essen


text:
---
title: Monitoring
weight: 16
aliases: ['/docs/launching/server-configuration/', '/docs/user-guides/server-configuration/', '/docs/user-guides/configuring-components/']
---

This section describes how to monitor Vitess components. Additionally, we recommend that you also add the necessary monitoring and alerting for the TopoServers as well as the MySQL instances running with each vttablet.

## Tools

Vitess provides integrations with a variety of popular monitoring tools: Prometheus, InfluxDB and Datadog. The core infrastructure uses go's `expvar` package to export real-time variables visible as a JSON object served by the `/debug/vars` URL. The exported variables are CamelCase names. These names are algorithmically converted to the appropriate naming standards for each monitoring tool. For example, Prometheus uses a [snake case conversion algorithm](https://github.com/vitessio/vitess/blob/e259a08f017d9f1b5984fcaac5c54e26d1c7c31d/go/stats/prometheusbackend/prometheusbackend.go#L95-L116). In this case, the Prometheus exporter would convert the `Queries.Histograms.Select.500000` variable to `vttablet_queries_bucket{plan_type="Select",le="0.0005"}`.

In the sections below, we will be describing the variables as seen in the `/debug/vars` page.

The two critical Vitess processes to monitor are vttablet and vtgate. Additionally, we recommend that you setup monitoring for the underlying MySQL instances as commonly recommended in the MySQL community.

Beyond what the tools export, it is important to also monitor system resource usage: CPU, memory, network and disk usage.

There is a [popular mixin](https://github.com/vitessio/vitess/tree/main/vitess-mixin) contributed by [Slack](https://slack.com) that is based on how they monitor Vitess internally. It can be used as a starting point if you intend to use Prometheus and Grafana.

Beyond the monitoring variables, the Vitess processes export additional information about their status on other URL paths. Some of those pages are for human consumption, and others are machine-readable meant for building automation.

## VTTablet

### /debug/status

This page has a variety of human-readable information about the current vttablet and contains links to all the other URLs. You can look at this page to get a general overview of what is going on.

### /debug/vars

The following sections describe the various sub-objects of the JSON object exported by `/debug/vars`. These variables can be found at the top level of the JSON object:

#### Queries

Vitess has a structured way of exporting certain performance stats. The most common one is the Histogram structure, which is used by Queries:

``` json
  "Queries": {
    "Histograms": {
      "Select": {
        "1000000": 1138196,
        "10000000": 1138313,
        "100000000": 1138342,
        "1000000000": 1138342,
        "10000000000": 1138342,
        "500000": 1133195,
        "5000000": 1138277,
        "50000000": 1138342,
        "500000000": 1138342,
        "5000000000": 1138342,
        "Count": 1138342,
        "Time":<PHONE_NUMBER>87,
        "inf": 1138342
      }
    },
    "TotalCount": 1138342,
    "TotalTime":<PHONE_NUMBER>87
  },
```

The histograms are broken out into query categories. In the above case, `Select` is the only category, which measures all SELECT statements. An entry like `"500000": 1133195` means that `1133195` queries took under `500000` nanoseconds to execute.

Here is the full list of categories [expanded from the source](https://github.com/vitessio/vitess/blob/e259a08f017d9f1b5984fcaac5c54e26d1c7c31d/go/vt/vttablet/tabletserver/planbuilder/plan.go#L79-L102):

```
Select
SelectLock
Nextval
SelectImpossible
Insert
InsertMessage
Update
UpdateLimit
Delete
DeleteLimit
DDL
Set
OtherRead
OtherAdmin
SelectStream
MessageStream
Savepoint
Release
RollbackSavepoint
ShowTables
Load
Flush
LockTables
UnlockTables
```

The numbers are cumulative. For example, if you wish to count how many queries took between `500000ns` and `1000000ns`, the answer would be `1138196-1133195`, which is `1`.

Later below, we will be covering variables that break out these queries into further sub-categories like per-table, etc. However, we do not generate histograms for them because the number of values generated would be too big.

The thresholds are hard-coded. However, if you are integrating with an extermal tool like Prometheus, it will have its own thresholds.

The counters increment for the lifetime of the vttablet, and all values are updated in real-time.

`Queries.Histograms.Select.Count` is the total count in the `Select` category.

`Queries.Histograms.Select.Time` is the total time in the `Select` category.

`Queries.TotalCount` is the total count across all categories.

`Queries.TotalTime` is the total time across all categories.

There are other Histogram variables described below, and they will always have the same structure.

Use this variable to track:

* QPS
* Latency
* Per-category QPS. For replicas, the only category will be `Select`, but there will be more for primary tablets.
* Per-category latency
* Per-category tail latency
* Per-category cost: This value is calculated as QPS\*Latency. If the latency of a high QPS query goes up, it is likely causing more harm than the latency increase of an occasional query.

#### Results

```json
  "Results": {
    "0": 0,
    "1": 0,
    "10": 1138326,
    "100": 1138326,
    "1000": 1138342,
    "10000": 1138342,
    "5": 1138326,
    "50": 1138326,
    "500": 1138342,
    "5000": 1138342,
    "Count": 1138342,
    "Total": 1140438,
    "inf": 1138342
  }
```

Results is a simple histogram with no timing info. It gives you a histogram view of the number of rows returned per query. This does not include rows affected from write queries.

`Count` is expected to be the same as the `TotalCount` of the `Queries` histogram.

`Total` is the total number of rows returned so far.

#### Mysql

Mysql is a histogram variable like Queries, except that it reports MySQL execution times. The categories are "Exec" and “ExecStream”.

The vttablet queries exec time is roughly equal to the sum of the MySQL exec time, `ConnPoolWaitTime` and `Waits.Consolidations`.

#### Transactions

`Transactions` is a histogram variable that tracks transactions. The categories are "Completed" and “Aborted”. Since these are histograms, they include count as well as timings.

#### Waits

Waits is a histogram variable that tracks various waits in the system. Right now, the only category is "Consolidations". A consolidation happens when one query waits for the results of an identical query already executing, thereby saving the database from performing duplicate work.

This variable used to report connection pool waits, but a refactor moved those variables out into the pool related vars.

#### Errors

``` json
  "Errors": {
    "Deadlock": 0,
    "Fail": 1,
    "NotInTx": 0,
    "TxPoolFull": 0
  },
```

Errors are reported under different categories. It’s beneficial to track each category separately as it will be more helpful for troubleshooting. Right now, there are four categories. The category list may vary as Vitess evolves.

Errors/Query is a useful stat to track.

VTTablet also exports an InfoErrors variable that tracks inconsequential errors that don’t signify any kind of problem with the system. For example, a dup key on insert is considered normal because apps tend to use that error to instead update an existing row. So, no monitoring is needed for that variable.

#### InternalErrors

``` json
  "InternalErrors": {
    "HungQuery": 0,
    "Invalidation": 0,
    "MemcacheStats": 0,
    "Mismatch": 0,
    "Panic": 0,
    "Schema": 0,
    "StrayTransactions": 0,
    "Task": 0
  },
```

An internal error is an unexpected situation in code that may possibly point to a bug. Such errors may not cause outages, but even a single error needs to be escalated for root cause analysis.

#### Kills

``` json
  "Kills": {
    "Queries": 2,
    "Transactions": 0,
    "ReservedConnection": 0
  },
```

Kills reports the queries, transactions and reserved connections killed by vttablet due to timeout. It is a very important variable to look at during incidents.

#### TransactionPool\* and FoundRowsPool\*

There are a few variables with the above prefixes that report the status of the two transaction pools:

``` json
  "TransactionPoolAvailable": 300,
  "TransactionPoolCapacity": 300,
  "TransactionPoolIdleTimeout":<PHONE_NUMBER>00,
  "TransactionPoolMaxCap": 300,
  "TransactionPoolTimeout":<PHONE_NUMBER>0,
  "TransactionPoolWaitCount": 0,
  "TransactionPoolWaitTime": 0,
  "FoundRowsPoolAvailable": 300,
  "FoundRowsPoolCapacity": 300,
  "FoundRowsPoolIdleTimeout":<PHONE_NUMBER>00,
  "FoundRowsPoolMaxCap": 300,
  "FoundRowsPoolTimeout":<PHONE_NUMBER>0,
  "FoundRowsPoolWaitCount": 0,
  "FoundRowsPoolWaitTime": 0,
```

The choice of which pool gets used depends on whether the application connected with the `CLIENT_FOUND_ROWS` flag or not.

* `WaitCount` will give you how often the transaction pool gets full, which causes new transactions to wait.
* `WaitTime`/`WaitCount` will tell you the average wait time.
* `Available` is a gauge that tells you the number of available connections in the pool in real-time. `Capacity-Available` is the number of connections in use. Note that this number could be misleading if the traffic is spiky.

#### Other Pool variables

Just like `TransactionPool`, there are variables for other pools:

* `ConnPool`: This is the pool used for read traffic.
* `StreamConnPool`: This is the pool used for streaming queries.

There are other internal pools used by vttablet that are not very consequential.

#### `TableACLAllowed`, `TableACLDenied`, `TableACLPseudoDenied`

The above three variables are table acl stats. They are broken out by table, plan and user.

#### `QueryPlanCacheSize`

If the application does not make good use of bind variables, this value would reach the QueryCacheCapacity. If so, inspecting the current query cache will give you a clue about where the misuse is happening.

#### `QueryCounts`, `QueryErrorCounts`, `QueryRowCounts`, `QueryTimesNs`

These variables are another multi-dimensional view of Queries. They have a lot more data than Queries because they’re broken out into tables as well as plan. This is a priceless source of information when it comes to troubleshooting. If an outage is related to rogue queries, the graphs plotted from these vars will immediately show the table on which such queries are run. After that, a quick look at the detailed query stats will most likely identify the culprit.

#### `UserTableQueryCount`, `UserTableQueryTimesNs`, `UserTransactionCount`, `UserTransactionTimesNs`

These variables are yet another view of Queries, but broken out by user, table and plan. If you have well-compartmentalized app users, this is another priceless way of identifying a rogue "user app" that could be misbehaving.

#### /debug/health

This URL prints out a simple "ok" or “not ok” string that can be used to check if the server is healthy. The health check makes sure mysqld connections work, and replication is configured (though not necessarily running) if not on primary.

#### /debug/status\_details

This URL prints out a JSON object that lists the state of all the variables that contribute to the current healthy or unhealthy state of a vttablet. If healthy, you should see something like this:

```json
[
  {
    "Key": "Current State",
    "Class": "healthy",
    "Value": "PRIMARY: Serving, Jan 13, 2021 at 20:52:13 (PST)"
  }
]
```

#### /queryz, /debug/query\_stats, /debug/tablet\_plans, /livequeryz

* /queryz is a human-readable version of /debug/query\_stats. If a graph shows a table as a possible source of problems, this is the next place to look to see if a specific query is the root cause. This is list is sorted in descending order of query latency. If the value is greater than 100 milliseconds, it's color-coded red. If it is greater than 10 milliseconds, it is color coded yellow. Otherwise, it is color coded gray.
* /debug/query\_stats is a JSON view of the per-query stats. This information is pulled in real-time from the query cache. The per-table stats in /debug/vars are a roll-up of this information.
* /debug/table\_plans is a more static view of the query cache. It just shows how vttablet will process or rewrite the input query.
* /livequeryz lists the currently running queries. You have the option to kill any of them from this page.

#### /querylogz, /debug/querylog, /txlogz, /debug/txlog

* /debug/querylog is a continuous stream of verbose execution info as each query is executed. This URL can generate a lot of data because it streams every query processed by vttablet. The details are as per this function: https://github.com/vitessio/vitess/blob/main/go/vt/vttablet/tabletserver/tabletenv/logstats.go#L202
* /querylogz is a limited human readable version of /debug/querylog. It prints the next 300 queries by default. The limit can be specified with a limit=N parameter on the URL.
* /txlogz is like /querylogz, but for transactions.
* /debug/txlog is the JSON counterpart to /txlogz.

#### /debug/consolidations

This URL has an MRU list of consolidations. This is a way of identifying if multiple clients are spamming the same query to a server.

#### /schemaz, /debug/schema

* /schemaz shows the schema info loaded by vttablet.
* /debug/schema is the JSON version of /schemaz.

#### /debug/query\_rules

This URL displays the currently active query block rules.

### Alerting

Alerting is built on top of the variables you monitor. Before setting up alerts, you should get some baseline stats and variance, and then you can build meaningful alerting rules. You can use the following list as a guideline to build your own:

* Query latency among all vttablets
* Per keyspace latency
* Errors/query
* Memory usage
* Unhealthy for too long
* Too many vttablets down
* Health has been flapping
* Transaction pool full error rate
* Any internal error
* Traffic out of balance among replicas
* Qps/core too high
* High replication lag
* Errant transactions
* Primary is in read-only mode

## VTGate

### /debug/status

This is the landing page for a vtgate, which gives you the status of how a particular server is doing. Of particular interest there is the list of tablets this vtgate process is connected to, as this is the list of tablets that can potentially serve queries.

### /debug/vars

#### VTGateApi

This is the main histogram variable to track for vtgates. It gives you a break up of all queries by command, keyspace, and type.

#### HealthcheckConnections

It shows the number of tablet connections for query/healthcheck per keyspace, shard, and tablet type.

### TopologyWatcherErrors and TopologyWatcherOperations

These two variables track events related to how vtgate watches the topology. It is particularly important to monitor the error count. This can act as an early warning sign if a vtgate is not able to refresh the list of tablets from the topo.

### /debug/health

This URL prints out a simple "ok" or “not ok” string that can be used to check if the server is healthy.

### /debug/querylogz, /debug/querylog, /debug/queryz, /debug/query\_plans

These URLs are similar to the ones exported by vttablet, but apply to the current vtgate instance.

### /debug/vschema

This URL shows the vschema as loaded by vtgate.

### Alerting

For vtgate, here’s a list of possible variables to alert on:

* Error rate
* Error/query rate
* Error/query/tablet-type rate
* vtgate serving graph is stale by x minutes (topology service is down)
* Qps/core
* Latency

repository: vitessio/website

branch_name: refs/heads/prod

gha_language: Java

repo_name: vitessio/website


text:
dirty-chai
==========
Function form for terminating assertion properties.

## Install
```
npm install dirty-chai --save-dev
```
## Usage

`dirty-chai` is a chai [plugin](http://chaijs.com/plugins).

```js
var chai = require('chai');
var dirtyChai = require('dirty-chai');
var expect = chai.expect

chai.use(dirtyChai);
// ...
expect(true).to.be.true();
```

## Custom Error Messages

With this function form for terminating properties you can also provide custom error messages to show when the assertion fails. This works whether the assertion is somewhere mid-chain or at the end.

```js
expect(true).to.be.true.and.not.false('Reason: Paradox');
expect(true).to.be.true('The fabric of logic has torn').and.not.false();
```

## Affected Assertions

The following built-in assertions are modified by this plugin to now use the function-call form:

* ok
* true
* false
* null
* undefined
* exist
* empty
* arguments
* Arguments

## Caveats

**Always terminate with a function**

These forms can also be mixed, but the chain must always be terminated in the function form or assertions up to that point in the chain will not execute.

```js
expect(true).to.be.true.and.not.false();
expect(true).to.be.true().and.not.false();
```

**Chaining length/arguments**

This breaks both the `length` and `arguments` asserts when they are in the chain following any other assertion. To work around this limitation, do the `length` or `arguments` asserts first in the chain or just do multiple assertion statements.

```js
myArray.should.exist.and.should.have.length(3); // Error: length is not a function

// Do two assert statements instead
myArray.should.exist();
myArray.should.have.length(3);
```

** use with chai-as-promised **

If you're using chai-as-promised, you should `.use` chai-as-promised before dirty-chai:

```js
var chai = require("chai");
var chaiAsPromised = require("chai-as-promised");
var dirtyChai = require("dirty-chai");

chai.use(chaiAsPromised);
chai.use(dirtyChai);
```

## Plugin Assertions

This plugin will also hook and convert any property assertions added by other Chai plugins. The only thing you need to do is make sure to load dirty-chai before any other plugins so that it can get its hooks in place before the other plugins are loaded.

For example, if you load [sinon-chai](https://github.com/domenic/sinon-chai) after dirty-chai, all of its property assertions will now be method assertions.

```js
spy.should.have.been.called();
spy.should.have.been.calledOnce();
spy.should.have.been.calledTwice();
```

## Why?

[Chai](https://github.com/chaijs/chai) is probably one of the most popular assertion libraries in the node. It has over 400 dependents and is downloaded almost 500,000/month.

For stylistic reasons, Chai was designed so that any assertions that did not require parameters would simply assert on property access. This allowed those assertions to elide the empty parens that would be required if those assertions were methods.

This design decision has a pretty big impact on how much trust you can place in your tests, especially if you don't adhere strictly to TDD's red-green-refactor flow. For a detailed descent into why, read [Beware of libraries that assert on property access](https://github.com/moll/js-must#asserting-on-property-access).

There is also the problem of getting errors from linters like JSHint. If you are linting your test code, your linter will complain with an error something like "Expected an assignment or function call and instead saw an expression." Since the linter doesn't know about the property getter it assumes this line has no side-effects, and throws a warning in case you made a mistake.

Squelching these errors is not a good solution as test code is getting to be just as important as, if not more than, production code. Catching syntactical errors in tests using static analysis is a great tool to help make sure that your tests are well-defined and free of typos.

This plugin was written so that we can still take advantage of the large ecosystem of projects and plugins written with/for Chai, while still being able to trust your tests. It converts the built-in property assertions to method assertions, including any property assertions added by plugins.

The list of affected assertions, and many assertions added by plugins, are property getters that assert immediately on access. Because of that, they were written to be used by terminating the assertion chain with a property access.

```js
expect(true).to.be.true;
foo.should.be.ok;
````

A better option was to provide a function-call form for these assertions so that the code's intent is more clear and the linters stop complaining about something looking off. This form is added in addition to the existing property access form and does not impact existing test code.

```js
expect(true).to.be.true();
foo.should.be.ok();
```

gha_language: JavaScript

gha_license_id: MIT


text:
This is a repository of different source files relevent to my neophyte study of DSP.

### Sine

`time.generate()`

Generate sample indecies at given Fs for period T

`sine.generate()`

Generate a sine wave of frequency F0

`sine.freqAngular()`

Calculate angular frequency for F0

`sine.phaseInst()`

Calculate instantaneous phase at sample x(t) at F0

`sine.chirp()`

Generate a (fake?) chirp between F0 and F1 over a given sample range x

`sine.delayChirp()`

Generate a wave of F0, Start generating a chirp between time t1 and t2 up to F1, continue wave F1

### Sampling

`bandpass.fs()`

Calculate the minimum and maximum sample frequency for bandpass sampling, with a center frequency fc in a Bandwidth B with m spectral replications

`bandpass.nyquistCriterion()`

Calculate the Nyquist criterion for bandpass sampling centred on frequency fc with Bandwidth B

`bandpass.replicationSNR()`

Calculate the signal-to-noise ration when using sampling translation with m spectrum replicas

repository: carrotsrc/DSP-Knowledge-blocks

repo_name: carrotsrc/DSP-Knowledge-blocks


text:
# Direct Debit SEPA

* [Authorize](#authorize)
* [Capture](#capture)
* [Capture Authorized Payment](#capture-authorized-payment)
* [Refund](#refund)
* [Managing Mandates](#managing-mandates)

## Authorize

`Authorize` is done via the `preauthorize` API call.

```php
use Payum\Core\Request\Authorize;
use Valiton\Payum\Payone\Api;

$payment = [];
$payment[Api::FIELD_PAYMENT_METHOD] = Api::PAYMENT_METHOD_DIRECT_DEBIT_SEPA;

$payum
    ->getGateway('payone')
    ->execute(new Authorize($payment));
```

![Sequence Diagram Authorize Payment](direct-debit-sepa-authorize.png "Sequence Diagram Authorize Payment")

## Capture

`Capture` is done via the `authorize` API call. Don't get confused.

```php
use Payum\Core\Request\Capture;
use Valiton\Payum\Payone\Api;

$payment = [];
$payment[Api::FIELD_PAYMENT_METHOD] = Api::PAYMENT_METHOD_DIRECT_DEBIT_SEPA;

$payum
    ->getGateway('payone')
    ->execute(new Capture($payment));
```

![Sequence Diagram Capture Payment](direct-debit-sepa-capture.png "Sequence Diagram Capture Payment")

## Capture Authorized Payment

See [Paydirekt Capture Authorized Payment](paydirekt.md#capture-authorized-payment)

## Refund

See [Paydirekt Refund](paydirekt.md#refund)

## Managing Mandates

To perform a direct debit payment with SEPA you need a SEPA mandate. This mandate has to be retrieved *prior* to the
Authorization Request. Usually this is handled implicitly.

```php
use Payum\Core\Security\SensitiveValue;
use Valiton\Payum\Payone\Api;
use Valiton\Payum\Payone\Request\Api\GetSepaMandate;

$payum
    ->getGateway('payone')
    ->execute($mandate = new GetSepaMandate([
        Api::FIELD_CITY => 'Berlin',
        Api::FIELD_COUNTRY => 'DE',
        Api::FIELD_CURRENCY => 'EUR,
        Api::FIELD_IBAN => SensitiveValue::ensureSensitive('DE19123412341234123412'),
        Api::FIELD_LAST_NAME => $form->get('lastname')->getData(),
    ]));

if ($mandate->isApproved()) {
    // do something with the mandate
    $mandate->getMandateIdentification();

    // text for the customer
    $mandate->getMandateText();
}
```

The mandate has to be confirmed with an `Api\PreAuthorize` or a `Api\Authorize` within two hours.

You can also download the Mandate PDF file

```php
use Valiton\Payum\Payone\Api;
use Valiton\Payum\Payone\Request\Api\GetSepaMandatePdf;

$payum
    ->getGateway('payone')
    ->execute($mandatePdf = new GetSepaMandatePdf([
        Api::FIELD_FILE_REFERENCE => 'XX-T0000000',
    ]));

echo $mandatePdf->getFileContents();
```

repository: valiton/payum-payone

gha_language: PHP

repo_name: valiton/payum-payone


text:
# Linked List

Write a Singly-Linked List implementation that uses the Proxy pattern

Deques have four fundamental operations (using Array terminology):

* push (insert value at back)
* unshift (insert value at front)
* pop (remove value at back)
* shift (remove value at front)

The difference between deque and Array performance is that a deque
implements all of these operations in constant time. Even clever
implemetations of Array will often have to copy the entire Array in
order to unshift.

Under the hood we'll use an Element class with two fields, `next` and
`prev`, that are both writable.

To make the API usable, you'll use a Deque class as a proxy for this
list. There two good ways to implement Deque: maintain separate
references to the first and last Element, or maintain a reference to one
of them and ensure that the list is circular.

To keep your implementation simple, the tests will not cover error
conditions. Specifically: pop or shift will never be called on an empty
Deque.

In languages that do not have good support for mutability (such as
Elixir or Erlang), you may choose to implement a functional Deque where
shift and pop have amortized O(1) time. The simplest data structure for
this will have a pair of linked lists for the front and back, where
iteration order is front ++ reverse back.

You will need the Minitest gem. To install it, open a
terminal window and run:

    gem install minitest

## Source

Inspired by 'Data Structures and Algorithms with Object-Oriented Design Patterns in Ruby', singly linked-lists. [view source](http://www.brpreiss.com/books/opus8/html/page96.html#SECTION004300000000000000000)

repository: rootulp/exercism

branch_name: refs/heads/main

gha_language: Go

repo_name: rootulp/exercism


text:
chirps <IP_ADDRESS> (2022-01-15)
=========================

### ENHACEMENTS

* Methods for objects of class "SpatExtent" in `get_chirps()` and `get_chirts()` to return a raster within a given area

chirps 0.1.4 (2022-01-13)
=========================

### ENHANCEMENTS

* Add new function `get_chirts()` to fetch temperature data from CHC server (https://data.chc.ucsb.edu/products/CHIRTSdaily/v1.0/global_cogs_p05/)
* Implement data fetching from CHC server in `get_chirps()` which offers a better alternative for requests with multiple data points using GoC files from CHC server (https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/cogs/) and the `terra` package
* New S3 methods in `get_chirps()` for objects of class 'SpatVector' and 'SpatRaster' from the `terra` package
* Data can be returned as an object of class 'matrix' when using the argument `as.matrix = TRUE` in the S3 methods for objects of class 'default', 'SpatVector' and 'SpatRaster'
* Updates the URL to request data from ClimateSERV

### CHANGES IN BEHAVIOUR

* New argument `server = ` is added to indicate from which server the function should send the request, either 'CHC' or 'ClimateSERV'. Please use the argument `server = "ClimateSERV"` for backward compatibility with previous versions of the package.
* API requests to ClimateSERV use package httr instead of curl
* Argument `operation = ` in `get_chirps()` is only required when `server = "ClimateSERV"`
* Updates function `as.geojson()` to matches with the new requirements for ClimateSERV

chirps 0.1.3 (2021-07-10)
=========================

* GitHub version with ongoing updates and changes in behaviour.

chirps 0.1.2 (2020-07-12)
=========================

* Add citation info for JOSS paper
* Fix vignette build
* A S3 method `as.geojson()` is added to replace the functions `data.frame_to_geojson()` and `sf_to_geojson()`

chirps 0.1.0 (2020-07-01)
=========================

* rOpenSci release version

### ENHANCEMENTS

* Add `get_imerg()` to fetch IMERG data https://disasters.nasa.gov/instruments/imerg

chirps 0.0.8 (2020-05-22)
=========================

### CHANGES IN BEHAVIOUR

* Remove Imports of pkg 'tibble' which was basically to provide a "cool" print method.
* A new `print()` method is added for objects that inherits the class 'chirps_df'
* Pkg 'methods' was moved from Imports to Depends
* Comments/suggestions given by Jake Zwart in rOpenSci pkg review are added

chirps 0.0.6 (2020-01-29)
=========================

### ENHANCEMENTS

* Comments/suggestions given by Claudia Vitolo in rOpenSci pkg review are added
* `dataframe_to_geojson()`, `sf_to_geojson()` are added as exported functions avoiding `chirps:::`
* documentation for `tapajos` is given avoiding `chirps:::`

chirps 0.0.5 (2020-01-09)
=========================

### ENHANCEMENTS

* Fix comments given by good practice `goodpractice::gp()`. Avoid long code lines, it is bad for readability. Avoid 1:length(...), 1:nrow(...), 1:ncol(...), 1:NROW(...) and 1:NCOL(...) expressions. Not import packages as a whole, as this can cause name clashes between the imported packages.

chirps 0.0.4 (2020-01-03)
=========================

### NEW FEATURES

* S3 methods for objects of class "geojson" in `get_chirps()` and `get_esi()`

* Package vignette

* Prepare for submission to ropensci

### ENHANCEMENTS

* Validations in internal functions to transform 'sf' into geojson

* Add properties features to geojson output in `get_chirps()` and `get_esi()` via `.add_geojson_properties()`

chirps 0.0.3 (2019-12-31)
=========================

### NEW FEATURES

* `get_esi` is added to retrieve Evaporative Stress Index with S3 methods for "data.frame" and "sf"

* S3 methods for objects of class "data.frame" and "sf" in `get_chirps`

### CHANGES IN BEHAVIOUR

* `.get_request_progress` and a `while` condition are added to check the progress of large requests and prevent the function to fail.

* `.GET` is added as a general function to retrieve other datasets from ClimateSERV

* improvements in internal functions documentation

chirps 0.0.2 (2019-12-05)
=========================

### NEW FEATURES

* Calculate precipitation indices with `precip_indices` over a time span

chirps 0.0.1 (2019-12-03)
=========================

* GitHub-only release of prototype package.

repository: ropensci/chirps

gha_language: R

gha_license_id: NOASSERTION

repo_name: ropensci/chirps


text:
# SMURFS
## Superpixels from Multi-scale Refinement of Super-regions

Python plug&play library version of SMURFS. Easy to try different functions & parameters of the algorithm with the `pysmurfs` library.

## Installation

Obtain the code:

```bash
$> git clone https://github.com/imaluengo/SMURFS-Superpixels.git
$> cd SMURFS-Superpixels/
```

Install `python` dependencies:

- If using [Anaconda distribution](https://www.continuum.io/downloads) (recommended if no python distribution is installed on your system):

	```bash
	$> conda update numpy scikit-image scikit-learn cython matplotlib
	```

- Other `python` distributions:

	```bash
	$> pip install --upgrade numpy scikit-image scikit-learn cython matplotlib
	```

Compile library:
```bash
$> make
```

## Run SMURFS

```bash
$> python run_smurfs.py --help
usage: run_smurfs.py [-h] [--quick] [--regular] [--out OUT] [--plot]
					 input_file num_superpixels

SMURFS: Superpixels from Multiscale Refinement of Super-regions

positional arguments:
  input_file       Input RGB image file path
  num_superpixels  Desired number of superpixels.

optional arguments:
  -h, --help       show this help message and exit
  --quick          Run only one iteration of the algorithm.
  --regular        Obtain regular (more square) superpixels.
  --out OUT        Output folder. If not given, result will be saved in the
				   current folder.
  --plot           Show plot with results after finishing.
```

Example runs:

- **SMURFS:** 200 superpixels

	```bash
	$> python run_smurfs.py /data/BSR/BSDS500/data/images/test/100007.jpg 200 --plot
	```

- **Quick SMURFS (qSMURFS):** 200 superpixels running a single iteration of SMURFS

	```bash
	$> python run_smurfs.py /data/BSR/BSDS500/data/images/test/100007.jpg 200 --quick --plot
	```

- **Regular SMURFS (rSMURFS):** 200 superpixels with more regular shape

	```bash
	$> python run_smurfs.py /data/BSR/BSDS500/data/images/test/100007.jpg 200 --regular --plot
	```

- **Quick Regular SMURFS (qrSMURFS):** 200 superpixels running a single iteration of rSMURFS

	```bash
	$> python run_smurfs.py /data/BSR/BSDS500/data/images/test/100007.jpg 200 --quick --regular --plot
	```

## Sample Results

- **Left:** Original image.
- **Middle:** Superpixel boundaries marked over the original image.
- **Right:** Original image reconstructed by assigning to every pixel the mean color of its superpixel. Measures how well the superpixels represent the image and how much detail can be recovered from them.

![Result 1](img/fig1.png?raw=true "Result1")
![Result 2](img/fig2.png?raw=true "Result2")

## Reference

Please, consider citing the following publication if you use SMURFS within your research:

	@inproceedings{luengo2016smurfs,
	  title={SMURFS: superpixels from multi-scale refinement of super-regions},
	  author={Luengo, Imanol and Basham, Mark and French, Andrew P},
	  booktitle={British Machine Vision Conference (BMVC)},
	  year={2016}
	}

Publication available at:

[http://www.bmva.org/bmvc/2016/papers/paper004/index.html](http://www.bmva.org/bmvc/2016/papers/paper004/index.html)

**NOTE:** This is the Python version of the above publication, resulting in the same accuracy but sightly worse speed. Fully functional and stable C++ version, as presented in the above paper, is coming soon.

## Acknowledgements

Work in collaboration between:

- [University of Nottingham -- Computer Vision Laboratory](http://cvl.cs.nott.ac.uk/)

- [Diamond Light Source](http://www.diamond.ac.uk/Home.html)

repository: imaluengo/SMURFS-Superpixels

repo_name: imaluengo/SMURFS-Superpixels


text:
## Run Benchmark
To train/test saliency/correspondence estimation with specific network, you can type e.g.

- ``
python train.py task=saliency network=pointnet
``

- ``
python test.py task=correspondence network=rscnn
``

Note that for PointNet++/PointNet2 and RSCNN networks, custom ops must be built by running ``python setup.py install`` under `[pointnet2|rscnn]_utils/custom_ops` folder.

You can also run `setup_env.sh` to setup the pytorch environment with no pain.


text:
## configuration

```hcl
storage "consul" {
	address = "<IP_ADDRESS>:8500"
	path = "vault"
}

listener "tcp" {
  address     = "<IP_ADDRESS>:8200"
  tls_disable = 1
}

ui = true
```

## Start vaule

```bash
./vault server -config=config/configuration.hcl
```

repository: khannedy/notes

repo_name: khannedy/notes