text:
# Change Log

## 0.30.7

- Fix cast target

## 0.30.6

- Make casting more robust

## 0.30.5

- Fix .overlap(SET/BOOLEAN)

## 0.29.3

- Fix Plywood Set constructor function does not handle null/"null" and falsy values properly

## 0.29.2

- Add support for converting inExpression to SQL

## 0.29.1

- Bump node to 14 and druid-query-toolkit dep to 0.17

## 0.28.2

- Regex and contains expressions now cast their operand to VAR_CHAR

## 0.27.4

- Fix bug in `rebaseOnStart` when the range starts or ends on 0

## 0.27.3

- Fix rollup script

## 0.27.2

- Revert immutable-class and chronoshift dependency bumps

## 0.27.1

- Improve hybrid time filters for time compares

## 0.27.0

- Bump dependencies, reformat code

## 0.26.9

- Simplify an external with a FALSE filter to an empty dataset without querying it
- Revert to use `APPROX_QUANTILE` for `approximateHistogram` native type

## 0.26.8

- quantileExpression uses filter where instead of case when

## 0.26.7

- Version bump to pick up version `0.26.5` changes

## 0.26.6

- `Updated druid-query-toolkit` from `0.11.0` to `0.13.2`

## 0.26.5

- `sampleValue` in returns `null` for `sqlRefExpression`

## 0.26.4

- Recycle timeRef in decomposition

## 0.26.3

- Treat SQL `t.__time` as a time attribute

## 0.26.2

- Move join decomposition into base external to make it accessible for the DruidSqlExternal

## 0.26.1

- Fix `getJoinDecompositionShortcut` to use leftJoin if `timeShift` is present

## 0.26.0

- Remove support for issuing JS expressions to Druid. Deprecated since Druid 0.10.0.

## 0.25.4

- Handle dates in Druid expressions

## 0.25.3

- Move some stuff to static

## 0.25.2

- Add `SqlAggregateExpression.registerKnownAggregation`, cleanup

## 0.25.1

- Use `GROUP BY ()` instead of `GROUP BY ''`

## 0.25.0

- Bump dependency

## 0.24.1

- Fix `.cast` when applied to `s$` expressions

## 0.24.0

- Remove deprecated `introspectionStrategy` (always use segment metadata)
- Do not allow `.` to be unescaped in SQL data source name
- Limit min supported Druid version to 0.14.0
- Use `DOUBLE` instead of `FLOAT` when possible
- DruidDialect now takes the native attribute into account when generating SQL for `countDistinct` and `quantile`

## 0.23.4

- Use `CAST(... AS TIMESTAMP)` instead of `TIME_PARSE(...)` to convert things to time

## 0.23.3

- Use different filter aggregation

## 0.23.2

- Fix how filter is applied on a `COUNT(*) FILTER (WHERE ...)` expression

## 0.23.1

- Quick fix for having filter decomposition for topN compares

## 0.23.0

- Bump minor that should have been done in 0.22.10

## 0.22.13

- Overlap and In expressions should wrap their `OR` chains in parens

## 0.22.12

- Union of sets should work if the types are different when one of them is `NULL`

## 0.22.11

- Use `CONTAINS_STRING` and `ICONTAINS_STRING` for DruidSQL dialect

## 0.22.10

- Update druid-query-toolkit dep

## 0.22.9

- Improved `withQuery` introspection better detects TIMESTAMP and BOOLEAN columns

## 0.22.8

- Improved `withQuery` introspection, now works with `*`

## 0.22.7

- Added `withQuery` introspection

## 0.22.6

- DruidSQLExternal should forward the context

## 0.22.5

- Support multi key join

## 0.22.4

- Fix small typo in with query

## 0.22.3

- Fix QuantileExpression in SQL

## 0.22.2

- Fix filtering for filtered averages in resplit queries for complex inner aggregations

## 0.22.1

- Updated DruidQueryToolkit

## 0.22.0

- Remove PlyQL
- Switch code to prettier
- Add `withQuery` to SQLExternal
- add `s$` (SQL ref function)
- add `sqlAgregate`
- Remove plywood-light build

## 0.21.8

- Fix measure re-splitter with filter

## 0.21.7

- Updated Druid SQL driver to be up to date with latest Druid

## 0.21.6

- Better decomposition for time compare

## 0.21.5

- Fix bug when having filter was being swallowed for the timeseries

## 0.21.4

- Fix filtering for filtered averages in resplit queries

## 0.21.3

- Better handle `undefined` in druid groupBys

## 0.21.2

- Add `rebaseOnStart` to `Range`

## 0.21.1

- Make time joiner more robust

## 0.21.0

- Bumping dependencies

## 0.20.13

- Adjusted more types, to make plywood friendlier with strict nulls

## 0.20.12

- Adjusted types

## 0.20.11

- Move force finalize into expression options

## 0.20.10

- Add get and set options

## 0.20.9

- Allow options on expressions
- Fix version checks on `null` in Druid expressions

## 0.20.8

- Use expressions over JS aggs when possible
- Support more datasketches

## 0.20.7

- Handle large overrides better, fix `O(n^2)` issue

## 0.20.6

- More robust `toDate` does not crash if bogus date is introspected in DruidExternal

## 0.20.5

- Added `forceFinalize` to fix DAU calculation

## 0.20.4

- Fix clipping of badly rounded doubles

## 0.20.3

- Fix bug when planning complex fallback
- Allow multiple aggregations in custom agg

## 0.20.2

- Better sql escaping

## 0.20.1

- Fix Druid filtering on alternative time columns represented as LONG

## 0.20.0

- Updated dependencies

## 0.19.15

- Fix escaping in Druid `like`

## 0.19.14

- Send less javascript to Druid

## 0.19.13

- Fix typo in `Dataset#fullJoin`

## 0.19.12

(publish error)

## 0.19.11

- Allow re-splitting complex expressions

## 0.19.10

- Fix totals calculation for resplit measures

## 0.19.9

- Support Druid timeseries sorted descending
- Fixed Druid planning bug where contains case insensitivity preference would be swallowed up when LHS is a concatenation over a single dimension
- Fixed Druid planning bug with concatenating a dimension to itself

## 0.19.8

- Fix `in` filtering on fancy lookup expressions

## 0.19.7

- Fix previous NPE for real

## 0.19.6

- Fix NPE

## 0.19.5

- Support contains on Druid expressions

## 0.19.4

- Changed how concat is calculated in Druid to me more efficient

## 0.19.3

- Shortcut to decompose certain compare type queries

## 0.19.2

- Handle concat with different expressions

## 0.19.1

- Handle empty string sources

## 0.19.0

- Update dependencies

## 0.18.11

- Use expressions in fallback when extraction functions fail

## 0.18.10

- Correct time bucket inflation in case of string dates (as returned by topN queries)

## 0.18.9

- Move some Dataset methods into statics

## 0.18.8

- Nothing. Npm is not replicating so it is a crap shoot to install 0.18.7 hence bumping version and publishing again to hopefully kickstart it into action.

## 0.18.7

- avoid topN for filtered on time aggregate sort

## 0.18.6

- allow multiple resplit measures

## 0.18.5

- basic resplit ability

## 0.18.4

- use `usePrefixForLastTerm`

## 0.18.3

- Remove aggressive fallback simplification

## 0.18.2

- Added `termsDelegate` and fullText search prototype
- fix error where virtual columns can refer to themselves
- added number bucket expression support

## 0.18.1

- Control query concurrency

## 0.18.0

- Removing support for Druid < 0.10.0, cleaning up code

## 0.17.44

- Support for MatchExpression in Druid expression builder

## 0.17.43

- Better error messages for unsupported Druid expressions in older (< 0.11.0) Druid
- Fixed legacy cross functional test

## 0.17.42

- Use expressions to handle cases where an extraction function would have been used instead

## 0.17.41

- Changed Druid expression division to work how the post aggregators used to work (return 0 when dividing by 0)

## 0.17.40

- Add `.ln()` to expression parser
- Fix Druid concat to null out on empty strings

## 0.17.39

- Use expressions in post aggs
- Added `LogExpression` (`.log()`)

## 0.17.38

- Fix bug where CSV/TSV headers were not being escaped
- Added `attributeTitle` to TabulatorOptions
- Added `attributeFilter` to TabulatorOptions

## 0.17.37

- Support complex boolean math expressions for Druid

## 0.17.36

- Correctly gate `scan` query on 0.11.0

## 0.17.35

- More forgiving deserialization of sets

## 0.17.34

- Use `scan` query instead of `select`

## 0.17.33

- Adding more tests
- Fix null handling in IN filters for SQL drivers

## 0.17.32

- Correctly handle IN filters on virtualColumns

## 0.17.31

- Also working in filter

## 0.17.30

- Started virtualColumn implementation

## 0.17.29

- `Expression#defineEnvironment` works recursively

## 0.17.28

- Better stringification of strings in expressions

## 0.17.27

- Change the External#introspect method to take a `depth: "deep" | "default" | "shallow"` parameter instead of `deep: boolean` (`deep: true` => `depth: deep`)
- Druid introspect route will query time range on `depth: "default"` and not query it on `depth: "shallow"`

## 0.17.26

- Fix `Set#isAtomicType` to work correctly for `NULL`

## 0.17.25

- Respect filtered aggregations for quantile and custom aggregates

## 0.17.24

- Allow `maxRows` and support dataset trimming

## 0.17.23

- Now setting `outputType` in Druid dimension specs

## 0.17.22

- Updated tests for Druid 0.11.0
- Added and enabled previously skipped tests
- Fixed bug where `Set(null)` and `Set("null")` would be subsets of each other

## 0.17.21

- Added `round: true` to cardinality

## 0.17.20

- Fixed issue with milti-value list filter

## 0.17.19

- Fixed planning for nested JS postagg
- Fixed tests

## 0.17.18

- Updated dependencies
- `TimeRangeExpression.getQualifiedDurationDescription` now accepts a capitalization flag
- Testing with newer Druid
- Support Druid `DOUBLE` types

## 0.17.17

- Allow deep introspection
- Move away from any-promise towards native promises

## 0.17.16

- Fix bug in several PlyQL functions where string inputs were interpreted as plywood expressions leading to errors on values with spaces

## 0.17.15

- Fix bug in `LOCATE`

## 0.17.14

- Fix bug in `.cast("NUMBER")`
- Change tests to work with node8
- Updated examples

## 0.17.13

- Add engine property to `rawQueries`

## 0.17.12

- Allow `rawQueries` collection for externals

## 0.17.11

- Fix bug in formula parsing related to tuning
- Added tuning support to PlyQL

## 0.17.10

- Add support for a `tuning` config for Druid approximate histograms
- Add support for a finalizing postAgg for a custom aggregation
- Replaced TS `Lookup<T>` type with the TS2.2 native `Record<string, T>`

## 0.17.9

- Fix typos
- Add `AttributeInfo#changeType`
- Fix to DSV in dataset for NULL type columns

## 0.17.8

- Changed the meaning of IN to be a subset only operation with back compat provisions.
- Better build process for people with older npm
- `keys` not emitted if empty for `Dataset#toJS`
- `Set#simplify` => `Set#simplifyCover`
- `Set#generalUnion` => `Set#unionCover`
- `Set#generalIntersect` => `Set#intersectCover`
- Fixed SQL externals for filtered `Min` and `Max` aggregations
- Fixed an instance of unhanded error in pipe
- Simplification for case independent contains
- Allow `table_alias.*` in PlyQL
- Support new features added on Druid 0.10.0
- Allow approxHistogram calculation on numeric columns
- verboseRequesterFactory parameter `preQuery` renamed to `onQuery`
- verboseRequesterFactory callback parameters now get a single argument with lots of info
- verboseRequesterFactory will now name queries

## 0.16.5

- PlyQL: support for `IF()` `CASE WHEN THEN END`, and `NULLIF()`

## 0.16.4

- `DruidExternal` fixed some problems with JS code generation
- `DruidExternal` support for planning `.then().fallback()`

## 0.16.3

- `DruidExternal` GroupBys will not correctly use 'numeric' order
- `DruidExternal` fixed GroupBy output name

## 0.16.2

- Stable release

## 0.16.1

**Major breaking changes in this release**

- Druid: Allow count distinct on cross product (e.g. `$data.countDistinct($a ++ $b)`)
- Transition to TS 2.1
- Better division by 0 handling
- Extracted Requester type definition into `plywood-base-api`
- Switched from `Q` to `any-promise`
- Use has-own-prop library
- Changed `Dataset#toJS` to return an object rather than just the `data` array
- Datasets now actively maintain their `keys` array
- `Dataset#flatten` now returns a new (flat) `Dataset` and thus the `parentName` option is no longer supported
- In `Dataset#flatten`, `Dataset#ToCSV`, e.t.c `orderedColumns` option is no longer supported use `.select()` instead
- Added `columnOrdering` option to `Dataset#flatten` with values `'as-seen'` and `'keys-first'`
- `Dataset#getColumns` is now just `return this.flatten(options).attributes`
- `Dataset#getNestedColumns` was removed
- Added `ThenExpression`
- Expressions have all been standardized to apply to Sets as well as atomics
- Fix `NULL` types handling everywhere
- Remove `AttributeInfo#serialize`
- Removed `AttributeInfo#special`, `UniqueAttributeInfo`, `ThetaAttributeInfo`, and `HistogramAttributeInfo`
- Added `AttributeInfo.nativeType` that stores the original database type of the attribute
- `DruidExernal` will now be able to plan using `longSum`
- `DruidExernal` added ability to split on constant
- Removed deprecation warnings and deleted crutches from the 0.15 release
- `MinExpression` and `MaxExpression` will now correctly output their type as `TIME` if the argument is `TIME`
- Experimental support for DruidSQL
- `DruidExternal` correctly defining numeric ordering in topN metricsSpecs
- `DruidExternal` will now explicitly set `fromNext: false` when paginating select

## 0.15.13

- Fixed bug where bucketing in a multi-value dimension was off for non UTC timezones

## 0.15.12

- Version check now works with all parts of the version

## 0.15.11

- Fix error in version check that had trouble with versions like `0.10.0`

## 0.15.10

- PlyQL: Support for `SHOW CHARACTER SET`
- PlyQL: Support for `SHOW COLLATION`
- PlyQL: Tolerance for encoding defined strings `N'Hello World'` and `_utf8'Hello World'`

## 0.15.7

- Added timezone based formatting ability

## 0.15.6

- Better publish script

## 0.15.5

- Fixing JS code gen for number casting `+x !== parseFloat(x)` when `x == " "`
- Implement `bucket` extractionFn for DruidExternal

## 0.15.4

- Added `Range#isFinite`
- Fixed logic around heatmap query

## 0.15.3

- Added more CAST options

## 0.15.2

- Fixed special attributes not reporting `unsplitable` correctly
- Fixed numberInflater sometimes returning `NaN`

## 0.15.1

- All Dataset computation and aggregation functions now take expressions instead of functions.
  Each has a corresponding function that takes a function like the old API.
  So `Dataset#apply(name, x)` now expects `x` to be and expression but `Dataset#applyFn(name, x)` has the old API
- `AttributeInfo#type` defaults to `STRING`
- Removed PlyQL parser from plywood-lite

## 0.14.12

- Druid countDistinct planing no longer does `byRow: true` to be more inline with how plywood aggregates work in general
- Fix countDistinct to be able to accept all expression types
- Default step in timeShift and timeRange maker functions

## 0.14.11

- Added a plywood-light version with no externals or requester utils

## 0.14.10

- Run tests against Druid groupBy v2 + fix tests

## 0.14.9

- Fixed time resolution in `derivedAttributes`

## 0.14.8

- Better handling for derivedAttributes by type checking them in the external constructor

## 0.14.7

- Fixed `rawAttributes` when empty
- made `Expression#distribute` work for nested expression

## 0.14.6

- removed deprecated uses of `isInstanceOf`

## 0.14.5

- 'action' expressions no longer return `{ op: "ref", name: "_" }` in their toJS further improving API back compat.

## 0.14.4

- export `CustomDruidAggregations` and `CustomDruidTransforms` types

## 0.14.3

- better API backwards compatibility

## 0.14.2

- Typo

## 0.14.1

- Totally reworked internal representation all BlahAction classes are now BlahExpression
- removed `.custom` use `.customAggregate`
- changed `lookup` to `lookupFn`
- changed `LimitAction#limit` to `LimitExpression#value`
- changed `QuantileAction#quantile` to `QuantileExpression#value`
- Fixed NOT() in Druid having clause
- removed `Expression#actionize`
- removed `Expression#firstAction`
- removed `Expression#lastAction`
- removed `Expression#headActions`
- removed `Expression#popAction` use `.operand` instead

## 0.13.7

- Sets do not auto-unify elements
- Limit action works with Infinity

## 0.13.6

- Update deps

## 0.13.5

- Collect works for all types

## 0.13.4

- Fixed public / private typings

## 0.13.3

- Smart query limiting
- Support for `show status`
- Fixed BETWEEN bounds

## 0.13.2

- Added collect action
- No more context folding

## 0.13.1

- More efficient query parallelisation
- Misc cleanup

## 0.12.12

- Better type injection with TDI

## 0.12.11

- Quarter supported

## 0.12.10

- Allow bug fixes in deps

## 0.12.9

- Transition to better build system

## 0.12.8

- Support Druid `regexFiltered` and `listFiltered` dim specs

## 0.12.7

- Respect PlyQL column ordering by generating extra select statements

## 0.12.6

- SegmentMetadata passes context also
- Upgraded to latest PEG.js
- Fixed bug in OR action
- Support case insensitive RefExpression
- PlyQL `SHOW` query rewriting is case insensitive
- Moved methods out of `plywood` and into `immutable-class`:
  - `plywood.find` to `SimpleArray.find`
  - `plywood.findIndex` to `SimpleArray.findIndex`
  - `plywood.findByName` to `NamedArray.findByName`
  - `plywood.findIndexByName` to `NamedArray.findIndexByName`
  - `plywood.overrideByName` to `NamedArray.overrideByName`
  - `plywood.overridesByName` to `NamedArray.overridesByName`

## 0.12.5

- Fixes in `AttributeInfo#valueOf`

## 0.12.4

- Added `AttributeInfo#change`

## 0.12.3

- Transitioned to newer immutable tester

## 0.12.2

- Renamed `customExtractionFns` to `customTransforms`
- Using lowercase `alphanumeric` in `dimensionOrder`
- Renamed `customAction` to `customAggregationAction`

## 0.12.1

- Changed to build to use external modules
- Moved methods out of `plywood.helper`, specifically:
  - `helper.parseJSON` to `Dataset.parseJSON`
  - `helper.expressionLookupFromJS` to `Expression.expressionLookupFromJS`
  - `helper.expressionLookupToJS` to `Expression.expressionLookupToJS`
  - `helper.find` to `plywood.find`
  - `helper.findIndex` to `plywood.findIndex`
  - `helper.findByName` to `plywood.findByName`
  - `helper.findIndexByName` to `plywood.findIndexByName`
  - `helper.overrideByName` to `plywood.overrideByName`
  - `helper.overridesByName` to `plywood.overridesByName`
  - `helper.shallowCopy` to `plywood.shallowCopy`
  - `helper.deduplicateSort` to `plywood.deduplicateSort`
  - `helper.mapLookup` to `plywood.mapLookup`
  - `helper.emptyLookup` to `plywood.emptyLookup`
  - `helper.nonEmptyLookup` to `plywood.nonEmptyLookup`
  - `helper.verboseRequesterFactory` to `plywood.verboseRequesterFactory`
  - `helper.retryRequesterFactory` to `plywood.retryRequesterFactory`
  - `helper.concurrentLimitRequesterFactory` to `plywood.concurrentLimitRequesterFactory`
  - `helper.promiseWhile` to `plywood.promiseWhile`
- Moved `simpleLocator` out
- Removed `retryRequester` which was deprecated

## 0.11.9

- Expressions can parse pure JSON also

## 0.11.8

- Updated file notice headers
- Updated dependencies

## 0.11.7

- Added `STRING_RANGE` type
- Core: Support for `.indexOf()`, `.transformCase()` action
- PlyQL: Support for `LOCATE`, `UCASE`, `LCASE`

## 0.11.6

- No JS globals in JS functions

## 0.11.5

- Core: Support for `.cast()` action
- PlyQL: Support for `CAST`, `FROM_UNIXTIME`, `UNIX_TIMESTAMP` actions
- DruidExternal: fix for `.cast('number').numberBucket()`

## 0.11.4

- Cleaned up npm package and updated dependencies

## 0.11.3

- Externals take account of `requester` for comparisons
- Misc cleanup

## 0.11.2

- Made `Range` and abstract class
- PlyQL: Fixed `AS` for tables, also `"AS"` keyword is now optional (like in MySQL)
- Added `.cardinality()` action
- Added `CARDINALITY` to PlyQL

## 0.11.1

- Takes advantage of most features in Druid 0.9.1
- Removed untested RangeAttributeInfo with helpful error message to use numbers instead
- DruidExternal: added `-legacy-lookups` version flag
- Trying to secondary filter on `__time` in Druid <= 0.9.1 now throws an error (at planning stage) as it is impossible
- Added `.length()` action
- Added `LENGTH`, `LEN`, and `CHAR_LENGTH` to PlyQL

## 0.10.33

- DruidExternal: Segment metadata works on union dataSource
- Using abstract classes (forces dependency on TypeScript >= 1.6)

## 0.10.32

- Rename `dataSource` and `table` to `source`
- Added `External.getExternalFor`

## 0.10.31

- Fix bug where JS reserved words could not be used in aggregates

## 0.10.30

- Compute expression HEAD also

## 0.10.29

- PlyQL: From(sub query) works
- Native quantiles also work

## 0.10.28

- Added compare to ranges so they sort correctly

## 0.10.27

- DruidExternal: Fix bug where `$blah / 10` did not work in split
- DruidExternal: Fix bug where `1 + $blah` did not cast to number
- DruidExternal: Pure `.fallback()` now works
- Removed some dead code

## 0.10.26

- DruidExternal: Ability to split on a secondary time dimension by `PT2H` e.t.c.
- PlyQL: `SHOW FULL TABLES` rewrites correctly

## 0.10.23

- Druid Select query no longer bound to 10000 max limit and will use iterative strategy
- Numeric Dimensions can be aggregated

## 0.10.22

- Better simplification of `.is(TimeRange)` and `.is(NumberRange)`
- Removed some dead code

## 0.10.21

- Druid: Default to JavaScript filter for negative bounds in bound filter
- Druid Fixed number range behaviour in nested splits
- Add tests around negative numbers

## 0.10.20

- Added PostgresExternal
- Refactored common SQL External functionality into SQLExternal base class
- Fixed timezone bucketing in MySQL External
- Support countDistinct on Theta sketches
- Druid GroupBy will no longer receive arbitrary limit

## 0.10.19

- Made filters on non primary time dimensions work in DruidExternal
- Added basic tests for legacy Druid versions

## 0.10.18

- `timeFloor` and `timeBucket` actions make sure duration is floorable

## 0.10.17

- `helper.overrideByName` maintains original order
- DruidExternal explicitly sends UTC timestamps

## 0.10.16

- Ability to define delegates in Externals
- selectAction now type checks and only allows defined attributes to be selected
- `DruidExternal.getSourceList` now returns the sources in a sorted order
- Adding range extent

## 0.10.15

- New chronoshift fixes bug with flooring `PT12H`

## 0.10.14

- Fixed parsing of `NULL` in set expressions
- Added information function to PlyQL

## 0.10.13

- Allow PlyQL to support `SHOW SESSION VARIABLES WHERE ...` rewriting

## 0.10.12

- Fix distribution of `SUM(1) => COUNT()`
- Added `DruidExternal.getVersion`
- Fixed post computation tasks
- Added support for `USE` on PlyQL
- PlyQL allows `NULL` in set literal e.g. `{'a', 'b', NULL}`

## 0.10.11

- PlyQL describe query now rewrites

## 0.10.10

- Fixed problem with DruidExternal having filter generating an OR instead of an AND for time ranges

## 0.10.9

- Fixed problem with filtering on multi-dimensional dimensions with a lookup
- Added `Dataset#findDatumByAttribute`

## 0.10.8

- Fixed bug where flatten would fail on an empty nested dataset.

## 0.10.7

- Accidentally published again.

## 0.10.6

- Added `finalizer` option to `Dataset#toTabular`
- Fixed quoting when exporting to CSV or TSV

## 0.10.5

- Fixed DruidExternal sometimes generating an invalid Druid query when doing a multi-dimensional split that includes time
- Unsuppress datasource when it is limited or selected
- Better show query support

## 0.10.4

- Made Dataset#apply and Dataset#select truly immutable
- Fix Druid introspection bug on JS ingestion aggregators
- Allow data-less PlyQL queries like `SELECT 1+1`
- Default PlyQL `AS` text now matches SQL implementation.

## 0.10.3

- Removed duplicate entry that killed in strict mode.

## 0.10.2

- Added ability to do ISO8601 in PlyQL time literals
- Ability to sort in select queries
- Fix `timeFloor` now work with limit

## 0.10.1

- Fixed bug where select queries including multi value dimensions would sometimes error out in DruidExternal
- Changed `helper.overrideByName` to `helper.overridesByName`, `helper.overrideByName` now does a single override
- Added `finalLineBreak` option to `Dataset#toTabular`, `Dataset#toCSV`, `Dataset#toTSV` (default to 'include' for tabular and 'suppress' for toCSV and toTSV)
- `Dataset` now always has introspected dimensions
- Support for zero Intervals in PlyQL
- Support for `DATE_FORMAT` in PlyQL as used in MySQL for `TIME_FLOOR`

## 0.9.25

- Dramatically reduced PlyQL parser size
- Added `YEAR` as a possible `timePart` value
- Added `CURDATE`, `PI`, `YEAR`, `MONTH`, `WEEK_OF_YEAR`, `DAY_OF_YEAR`, `DAY_OF_MONTH`,
  `DAY_OF_WEEK`, `HOUR`, `MINUTE`, `SECOND`, `DATE`, `CURRENT_TIMESTAMP`, `LOCALTIME`,
  `LOCALTIMESTAMP`, `UTC_TIMESTAMP`, `SYSDATE`, `CURRENT_DATE`, `UTC_DATE`, `DAY_OF_YEAR`,
  `DOY`, `DOW`, `DAYOFMONTH`, `DAY`, `WEEKOFYEAR`, and `WEEK` to PlyQL
- Fixed bug where `DAY_OF_YEAR`, `DAY_OF_MONTH`, and `DAY_OF_WEEK` was zero indexed
- Added support for `timeFloor` split in Druid external

## 0.9.24

- Added SET expressions to PlyQL `{'A', 'B', 'C'}`
- PlyQL can now parse IN with arbitrary expression on right hand side

## 0.9.23

- Fix .match() not working on SET/STRING

## 0.9.22

- Moved date parsing to Chronoshift and updated to latest Chronoshift
- Parser can now configure a timezone within which it will parse 'local' date strings
- Optimized post processing by not calculating the filter for final splits

## 0.9.21

- Added ability to define the timezone through the environment
- Timezone always defaults to UTC
- Date parsing no longer done by PlyQL parser
- Added tests for legacy Druid versions

## 0.9.20

- Added convenient filtered aggregators to PlyQL like: `SUM(added WHERE cityName = 'San Francisco')`
- Fixed PlyQL incorrectly parsing `SELECT COUNT(page)`

## 0.9.19

- Fixed null handling in extractionFns for Druid 0.8.3

## 0.9.18

- Fixed overlap on `[null]`
- Enabled all Druid 0.9.0 tests (testing using Druid 0.9.0-iap1)

## 0.9.17

- DruidExternal now supports lookups on SET/STRING dimensions

## 0.9.16

- Fixed bug where DruidExternal did not work with filtered cardinality aggregator

## 0.9.15

- New immutable class methods

## 0.9.14

- Allow derivedAttributes to be specified in ExternalJS
- derivedAttributes can now be used in filters and splits
- Fixed bugs in type inference
- Fixed `.substr()` getJS to work correctly with nulls

## 0.9.13

- Added (experimental) rollup mode to all Externals, only implemented in DruidExternal
- Derived attributes now work for DruidExternal

## 0.9.12

- DruidExternal now supports boolean ref filters like: `$wiki.filter($isAnonymous)`

## 0.9.11

- added `getSourceList` to externals
- Fixed bug in expression simplification where `$s.contains('A') and $x.is('B') => false`
- Respect unsplitable measures in DruidExternal

## 0.9.10

- DruidExternal timeAttribute now defaults to `__time`
- Moved find\* methods into helper
- Externals can re-introspect

## 0.9.9

- Fixed Druid 'select' failing on time interval outside of Druid cluster

## 0.9.8

- Using Imply Analytics Platform 1.1.1 docker image for Druid tests
- Derived columns in Druid select will now trigger if version is `/^0.9.0-iap/`

## 0.9.7

- `version` is now a parameter on all External
- In Druid external `druidVersion` was renamed to `version`
- Fixed MySQL translation of .is() action to use null safe equals (<=>)
- Fixed MySQL translation of .contains()
- Added `<=>` to PlyQL
- Druid external now introspects version and does not make broken query

## 0.9.6

- Support for new features in Druid 0.9.0
- Fix for Druid "bound" / "between" filter
- Much faster PlyQL parsing

## 0.9.5

- The FROM clause in PlyQL now accepts relaxed table names allowing for: `SELECT * FROM my-table*is:the/best_table`

## 0.9.4

- Set `skipEmptyBuckets: "true"` in timeseries queries to get the Plywood expected behaviour

## 0.9.3

- Added `.select('attr1', 'attr2', 'attr3')` action allowing `SELECT attr1, attr2, attr3 FROM ...` to be expressed in PlyQL
- Added support for `.quantile(p)` aggregator in DruidExternal (using [approximateHistograms](http://druid.io/docs/latest/development/approximate-histograms.html))
- Better support for filtered aggregates in DruidExternal

## 0.9.2

- Custom type guards for all the modal classes

## 0.9.1

- New 'value' mode in externals. `$wikipedia.sum($added)` is now computable.
- More relaxed date literal parsing in PlyQL

## 0.8.21

- Fix native `overlap` calculation

## 0.8.20

- Support Date and Timestamp literal as described here: https://dev.mysql.com/doc/refman/5.7/en/date-and-time-literals.html

## 0.8.19

- Upgraded to TypeScript 1.8

## 0.8.18

- SELECT queries in Druid are now mindful of the attributes and inflate values correctly.

## 0.8.17

- All cross functional tests now running with rollup.
- Temporary columns are no longer being returned by Druid.

## 0.8.16

- Fixed issue with `.in(...).not()` in the DruidExternal

## 0.8.15

- `overlap` now allows STRING arguments, simplifies to `in`

## 0.8.14

- DruidExternal will use topNs for defined small splits even when there is no limit set (like `BOOLEAN`, `timeBucket(HOUR_OF_DAY)`)
- Fixed bug for MySQLExternal not dealing well with BOOLEAN splits

## 0.8.13

- Added `overlap` action
- Added multi-database query test / example

## 0.8.12

- Fixed sorting direction when sorting on a split in Druid
- Ability to do filtered aggregations on SQL drivers
- `== NULL` works correctly with SQL drivers

## 0.8.11

- MySQL functional tests can now run in docker
- Better checks for expression types in aggregates

## 0.8.10

- Better handling for filtered attributes now allows filtered attributes on `countDistinct` and `custom`
- Fixed sorting for numeric columns in DruidExternal

## 0.8.9

- Add `absolute`, `power`, `sqrt`, `fallback` action
- Corresponding `ABS`, `POW/POWER`, `EXP`, `SQRT`, and `IFNULL/FALLBACK` to PlyQL
- Fixed type checking in `InAction`

## 0.8.8

- Fixed `match` on nulls to return `null`

## 0.8.7

- Support `SELECT DISTINCT` syntax in PlyQL

## 0.8.6

- Add `timeRange` action (`TIME_RANGE` in PlyQL) - not supported by DruidExternal yet

## 0.8.5

- Add `timeFloor` action (`TIME_FLOOR` in PlyQL) - not supported by DruidExternal yet
- Rename unused `offsetTime` action to `timeShift` (`TIME_SHIFT` in PlyQL) and add `step` parameter - not supported by DruidExternal yet
- Fixed how `concat` works on nulls to return null
- Added `.isnt` and `.negate` to expression parsing grammar (they were mistakenly excluded)
- Added `MATCH` function to PlyQL
- Timezone parameter is now optional in all the `time*` functions in Plywood and PlyQL
- Added `NOW()` to PlyQL
- Better DruidExternal column introspection error detection

## 0.8.4

- Allow parsing of `DESCRIBE` verb for PlyQL

## 0.8.3

- Changed `avoidSegmentMetadata` to `introspectionStrategy: 'datasource-get'` defaults to `'segment-metadata-fallback'`

## 0.8.2

- Brand new introspection code for `druidExternal` no more `useSegmentMetadata` flag instead there is a `avoidSegmentMetadata` flag
- Fix `.lookup()` stringification
- PlyQL comments are treated like they are in MySQL: `--` needs to be followed by a space, `#` starts a comment,
  and `/* ... */` inline and multiline comments are supported

## 0.8.1

- Removing d3 dependency and minimizing code size

## 0.7.30

- Fixed interval conversion to add a millisecond and not a second
- Allow for filtering on exact date in DruidExternal
- Fixed missing error message when invalid interval is given

## 0.7.29

- In DruidExternal a time attribute will no longer collide with an existing attribute

## 0.7.28

- Added ability to write `$('time').in('2015-03-03Z', '2015-10-10Z')`
- Added ability to parse `$time.in('2015-03-03Z', '2015-10-10Z')`
- Fixed bug in `$number.in(1, 2)` syntax

## 0.7.27

- Native `timePart` now works

## 0.7.26

- Updated Druid External to use `doubleMin` and `doubleMax`.

## 0.7.25

- Fixed bug in sorting with `null` in the list
- Updated to latest typescript (1.7.5)

## 0.7.24

- Added the `.lookup('my_lookup')` expression that translates to a query time lookup in Druid
- Added the corresponding `LOOKUP` function to PlyQL
- Fixed split simplification

## 0.7.23

- In PlyQL `COUNT(blah)` returns the count where blah is not `null` in accordance with SQL standard.
- Added query IDs to `verboseRequester`
- Flashed out PlyQL docs
- Fixed bug in parameter parsing with leading numbers

## 0.7.22

- Added `.extract()` function
- Added `EXTRACT` function to PlyQL
- Fixed bug in PlyQL where a space before the trailing ) would cause a parse error
- More docs :-)

## 0.7.21

- DruidExternal supports extractionFn filters such as `.filter($x.substr(0, 1) === 'D')`
- added `druidVersion` to DruidExternal to support feature differentiation
- added native Druid substring filter support or druidVersion >= 0.9.0

## 0.7.20

- Added some docs
- Created [plywood-proxy](https://github.com/implydata/plywood-proxy)
- PlyQL `LIKE` now works to SQL spec (added `ESCAPE` clause)
- Added `CONTAINS` clause to PlyQL
- Added `REGEXP` clause to PlyQL

## 0.7.19

- Added `firstAction` and `lastAction` to all expressions
- `popAction` no longer takes an optional action type (warning added)

## 0.7.18

- DruidExternal now works with upper/lower time bound only

## 0.7.17

- Fixed string handling in parser arguments
- Added `contains` to expression parsing grammar

## 0.7.16

- Added `timePart` to MySQL dialect
- Added time part option for `MONTH_OF_YEAR`

## 0.7.15

- Fixed bug to ensure `Dataset#flatten` works with empty datasets

## 0.7.14

- Added `Set#toggle`

## 0.7.13

- Added simplification rule for `.sort(X).filter(Y) => .filter(Y).sort(X)`
- Added simplification rule for `.sort(X).sort(X) => .sort(X)`

## 0.7.12

- Added simplification rule for `.not().not() => nothing`

## 0.7.11

- Added `External#updateAttribute`

## 0.7.10

- Fixed `Dataset#average`
- Latest Chronoshift allows for PT15M bucketing, e.t.c.

## 0.7.9

- Better apply, filter, limit sorting
- Multi-dim group dy uses granularity when it can.

## 0.7.8

- Fixed bug in `Set#add`

## 0.7.7

- added tonic example: https://tonicdev.com/npm/plywood

## 0.7.4

- Refactored DruidExternal
- DruidExternal now supports multi-dim splits
- Timezone (2nd argument) in `timeBucket` is now optional, defaults to UTC
- Fixed native `dataset.max`

## 0.7.3

- Better docs
- Context is now added passed along in a DruidExternal

## 0.7.2

- Added `Expression.concat`
- Added `CONCAT` to the PlyQL parser

## 0.7.1

- Paving the road for multi-dimensional splits `.split({ Page: '$page', User: '$user' })`
- Fixed problems with SELECT queries
- Allow for SQL parsing of `SELECT *`
- Fix support for `sort` and `limit` in SELECT queries
- Better escaping in MySQL driver

## 0.6.3

- Changed JS fallbacks in DruidExternal to use native code generators and removed some null bugs.

## 0.6.2

- Added more rules to the SQL parser: `IS`, `LIKE`, and awareness of `UPDATE`, `SET`, e.t.c

## 0.6.1

- Changed the return type of `Expression.parseSQL` to return and object with the keys `verb`, `expression`, and `table`
  `Expression.parseSQL(blah)` ==> `Expression.parseSQL(blah).expression`
- In PlyQL changed the meaning of `GROUP BY <number>` to be a reference to a column (just like in MySQL and Postgres)

## 0.5.2

- Added support for case sensitive (`normal`) / case insensitive (`ignoreCase`) contains.
- Improved concat action support in Druid

## 0.5

- Chain expression `.toJS()` now returns `action` instead of `actions` array if there is only one action.
  Both are still valid to parse.
- Misc bug fixes
- Added SortAction DESCENDING, ASCENDING, toggleDirection
- DruidExternal guards against duplicate aggs and postAggs

## 0.4

- Changed attribute definitions to be an array instead of an object and added deprecation message.

repository: implydata/plywood

repo_name: implydata/plywood


text:
# thebluealliance-phpwrapper
A PHP wrapper that assists in the use of the API provided by http://www.thebluealliance.com

# Installation

### Composer
Installing the application through Composer is the fastest way to get all setup.
To do this navigate to the root of your project in the terminal and run

`composer require dacotahharvey/thebluealliance-phpwrapper`

Then when you use the project include the two following lines at the top of your file

```
require_once __DIR__ . '/lib/vendor/autoload.php';
use TheBlueAlliance_PHPWrapper\TBARequest;
```

### Manually Cloning

1. In your project make a new folder called tbaAPI
2. Download this repository.
3. Unzip the files that you downloaded and place them in the tbaAPI folder that you previously created.
4. Include the following line at the top of the file that you wish to use the wrapper in

```
include __DIR__ . '/tbaapi/TBARequest.php';
```

# Library Usage

To use this library you need to have an API Key from The Blue Alliance. You can find more information on how to do this
here https://www.thebluealliance.com/apidocs

1. Create a new reference to the library like so
```
$tbaRequest = new tbaAPI\TBARequest(api_key);
```

2. Call the functions provided to retrieve your data like so

```
echo $tbaRequest->getDistricts(['year' => 2016]);
```

All of the functions provided by the wrapper take the same three paramters

1. $request_parameters - Any paramters that that the endpoint needs for the query
2. $headers - Any additional headers that you want to send with the request
3. $full_response - Whether you want the full response or just the JSON returned by The Blue Alliance

Lets say for example we wanted to make a request to get the team 1114.
We can find the documentation for this function [here](https://www.thebluealliance.com/apidocs#team-request)
Using the var `tba` from above we can do the following

```
$result = $tbaRequest->getTeam(['team' => 1114]);
```

# Executing a Premade Script
Executing a php script is very easy once the initial setup has been taken care of.
The scripts provided with this project will work best with PHP 7, however they will
also perform with a lower version. Running the scripts that are contained within this repository is simple. It can be accomplished using the three following steps

1. Clone the repository
2. In each of the sciripts you will find the library initalization. You must change this in each of the scripts that you wish to run as currently they empty. You can find more information on how to do this in the Library Usage section of this Readme.
3. Use the following commands in the terminal. Note that the composer install is only used once to generate the autoload file that must exist to properly use the plugin

```
composer install
cd scripts
cd 2017
```

Then choose the file that you want to run. Each script will contain instructions on how to run it. If you try to run a script with improper commands it will tell you how it should be run. An example of a script being executed once you are in the proper directory is

```
php team-match-turnaround.php 2017dar frc1114
```

The script will then run in your terminal instance.

### PHP Installation in Mac OSX

Mac OSX comes with PHP by default! If you are ambitious you can upgrade to PHP7.
The tutorial steps below will help with that.

1. Homebrew is the best way to upgrade your Mac instance to PHP7. Homebrew is a tool
that will allow you to install packages via the terminal. A fantastic tutorial on what
Homebrew is and how to use it can be found here:
https://www.howtogeek.com/211541/homebrew-for-os-x-easily-installs-desktop-apps-and-terminal-utilities/

2. This tutorial is a fantastic reference point for upgrading to PHP7. You can also
perform the steps in reverse to downgrade back to PHP5!
https://developerjack.com/blog/2015/12/11/Installing-PHP7-with-homebrew/

### PHP Installation on Windows

Installing PHP through the command line is an arduous task on windows. The PHP library
offers a tutorial on this that can be found here http://php.net/manual/pl/install.windows.commandline.php

Alternatively you can use Xxamp. The homepage for Xxamp with a download link can
be found here https://www.apachefriends.org/index.html and a fantastic tutorial on
how to use xxamp can be found here https://blog.udemy.com/xampp-tutorial/

# Contact Me

If you have any questions or concerns feel free to contact me at<EMAIL_ADDRESS>I am always more than willing to help write a script or setup an environment that
will allow you to execute your own scripts!

license: MIT

branch_name: refs/heads/master

extension: md

gha_language: PHP

license_type: permissive


text:
### 关于

受 [android-interview-questions](https://github.com/MindorksOpenSource/android-interview-questions) 项目启发，这里想发挥众多 Android 中国开发者的力量，整理一份高质量、范围全的 Android 面试指南，旨在帮助更多的 Android 开发者提升技术，找到工作。

现在还是项目初期，项目背景见这里：[想跟大家一起做件小事](http://mp.weixin.qq.com/s/t038R0bDDZ6dg4bwDoj2cQ)，欢迎持续关注。

## Contents
 * [数据结构和算法](#数据结构和算法)
 * [Java 核心](#core-java)
 * [Android 核心](#core-android)
 * [架构](#architecture)
 * [设计问题](#design-problem)
 * [工具和技能](#tools-and-technologies)
 * [Android 测试驱动开发](#android-test-driven-development)
 * [其他](#others)

### 数据结构和算法

> 关于数据结构和算法的问题的难度取决于你面试的是哪一家。

* 数组
* 链表
    - 链表，有点像树，但又不同于数组，它是由一组代表序列的节点组成。每个节点包含数据和一个指针。这里的数据可以是任何东西，但是指针必须是指向链表中的下一个节点。链表包含头和尾，头是链表的第一个节点，尾是最后一个节点。链表不是环形的数据结构，所以尾没有指向头，其指针为空。链表的基本操作的时间复杂度如下:

        | 算法    | 平均值  | 最差情况   |
        |:-------:|:-------:|:----------:|
        | 空间    | O(n)    | O(n)       |
        | 搜索    | O(n)    | O(n)       |
        | 插入    | O(1)    | O(1)       |
        | 删除    | O(1)    | O(1)       |

* 双向链表
* 栈
    - A Stack is a basic data structure with a "Last-in-First-out" methodology. Which means that
    the last item that was added to the stack, is the first item that comes out of the stack. A
    Stack is like a stack of books. In order to get to the first book that was added in the stack
    (the bottom book), all of the books that were added after need to be removed first. Adding to a
    Stack is called a Push, removing from a stack is called a Pop, and getting the last item
    inserted into the stack without removing it is called Top. [The most common way to implement a
     stack is by using a LinkedList, but there are also StackArray (implemented with an array)
     which does not replace null entries, and there is also a Vector implementation that does
     replace null entries.](https://en.wikibooks.org/wiki/Data_Structures/Stacks_and_Queues#Performance_Analysis)

                Algorithm
                Average
                Worst Case
                Image representation

                Space
                O(n)
                O(n)

                Search
                O(n)
                O(n)

                Insert (Push)
                O(1)
                O(1)

                Delete (Pop)
                O(1)
                O(1)

              Top
              O(1)
              O(1)

* 队列
* 优先队列
* 动态规划
* 字符串处理
* 二叉树
* 二叉搜索树
* 排序算法
* 哈希表
* 广度优先搜索
* 深度优先搜索
* 贪婪算法

### License
```
   Copyright (C) 2017 stormzhang

   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
```

repository: qvoid/android-interview-questions-cn

repo_name: qvoid/android-interview-questions-cn


text:
---
layout: post
title: "Trip Table Estimation and Prediction for Dynamic Traffic Assignment Applications"
date: 2019-06-11 13:42:12
categories: arXiv_AI
tags: arXiv_AI Prediction
author: Sajjad Shafiei, Adriana-Simona Mihaita, Chen Cai
mathjax: true
---

* content
{:toc}

##### Abstract
The study focuses on estimating and predicting time-varying origin to destination (OD) trip tables for a dynamic traffic assignment (DTA) model. A bi-level optimisation problem is formulated and solved to estimate OD flows from pre-existent demand matrix and historical traffic flow counts. The estimated demand is then considered as an input for a time series OD demand prediction model to support the DTA model for short-term traffic condition forecasting. Results show a high capability of the proposed OD demand estimation method to reduce the DTA model error through an iterative solution algorithm. Moreover, the applicability of the OD demand prediction approach is investigated for an incident analysis application for a major corridor in Sydney, Australia.

##### Abstract (translated by Google)

##### URL
[http://arxiv.org/abs/1906.04739](http://arxiv.org/abs/1906.04739)

##### PDF
[http://arxiv.org/pdf/1906.04739](http://arxiv.org/pdf/1906.04739)


text:
# _txrevise_
_txrevise_ R package provides utilites to pre-process Ensembl transcript annotations to quantify differences in transcript strucuture (alternative promoters, alternative splicing, alternative poly-adenylation) either between experimental conditions or genotypes (e.g. for transcript usage quntitative trait loci (tuQTL) mapping).

If you use txrevise in your research, please cite the following paper: [Alasoo, Kaur, et al. "Genetic effects on promoter usage are highly context-specific and contribute to complex traits." Elife 8 (2019): e41673](https://doi.org/10.7554/eLife.41673)

## Pre-computed event annotations
Running _txrevise_ on the latest version of Ensembl can be quite timeconsuming. Thus, to make it easier to get started, we have pre-computed alternatve transcription events in the GFF3 format for both GRCh37 and GRCh38 reference genomes.

We have constructed two types of events. In the main event files we have masked the alternative internal exons in the promoter and 3' end events (labeled upstream and downstream). This ensures that we can reliably distinguish promoter usage and 3' end usage from alternative splicing, which are likely to be driven by distinct molecular mechanisms. The main annotation files are:

- [Homo_sapiens.GRCh37.87.version_1.tar.gz](https://zenodo.org/record/1302499/files/Homo_sapiens.GRCh37.87.version_1.tar.gz)
- [Homo_sapiens.GRCh38.92.version_1.tar.gz](https://zenodo.org/record/1302499/files/Homo_sapiens.GRCh38.92.version_1.tar.gz)
- [Homo_sapiens.GRCh38.96.version_1.tar.gz](https://zenodo.org/record/3232932/files/Homo_sapiens.GRCh38.96.version_1.tar.gz)

We have also constructed an alternative set of "raw" annotation files where the alternative internal exons in promoter and 3' end events have not masked. This maximises QTL discovery, because we can now also detect splicing event near promoters and 3 ends, but it comes at the expense that we are no longer able to distinguish between different molecular mechanisms. The raw annotation files are:

-   [Homo_sapiens.GRCh37.87.raw_events.version_1.tar.gz](https://zenodo.org/record/1302499/files/Homo_sapiens.GRCh37.87.raw_events.version_1.tar.gz)
-   [Homo_sapiens.GRCh38.92.raw_events.version_1.tar.gz](https://zenodo.org/record/1302499/files/Homo_sapiens.GRCh38.92.raw_events.version_1.tar.gz)

## Constructing transcription events

### Automatically
We have written a [Snakefile](https://github.com/kauralasoo/txrevise/blob/master/scripts/Snakefile) for [snakemake](http://snakemake.readthedocs.io/en/stable/) that
 - automates all of the steps
 - can be easily [executed in parallel](http://snakemake.readthedocs.io/en/latest/executable.html) on multiple cores or on a compute cluster if desired
 - can use a [Singularity](https://sylabs.io/guides/3.6/user-guide/) container, preventing R package version conflicts and ensuring reproducible results
Thus, using the Snakefile is the recommended method. Building the Singlarity container is described [here](https://github.com/kauralasoo/txrevise/blob/master/build_container.sh) and running the snakemake pipeline [here](https://github.com/kauralasoo/txrevise/blob/master/scripts/run_snakemake.sh).

### Manually
This section contains step-by-step instructions for manually constructing transcriptional events based on Ensembl transcript annotations.

#### Dependencies
Make sure that you have R 3.5 installed together with the following packages:

 - [optparse](https://cran.r-project.org/package=optparse)
 - [dplyr](https://cran.r-project.org/package=dplyr)
 - [purrr](https://cran.r-project.org/package=purrr)
 - [data.table](https://cran.r-project.org/package=data.table)
 - [ggplot2](https://cran.r-project.org/package=ggplot2)
 - [devtools](https://cran.r-project.org/package=devtools)
 - [rtracklayer](https://bioconductor.org/packages/rtracklayer/)
 - [GenomicFeatures](https://bioconductor.org/packages/GenomicFeatures/)
 - txrevise

You can install _txrevise_ directly from GitHub using the following command:

	library("devtools")
	install_github("kauralasoo/txrevise")

#### Step 1: Download the GTF file
First, you need to download the GTF file from the Ensembl website. For example, if you want to use Ensembl version 96, you should download the following file:

	wget ftp://ftp.ensembl.org/pub/release-96/gtf/homo_sapiens/Homo_sapiens.GRCh38.96.gtf.gz

#### Step 2: Extract tanscript tags from the GTF file
Ensembl GTF file contains a tags field marking protein coding transcript that are truncated at 5' or 3' ends. Txrevise uses this information to extend truncated transcripts. Unfortunately, the `import` function from rtracklayer does not import the tags field correctly. Thus, we need to first extract the transcript tags manually using a custom Python 3 script that comes with txrevise.

	python scripts/extractTranscriptTags.py --gtf Homo_sapiens.GRCh38.96.gtf.gz > Homo_sapiens.GRCh38.96.transcript_tags.txt

#### Step 3: Prepare transcript annotations for event construction
Next, we need to convert the transcript annotations (in GTF format) to a binary representation that can be efficiently used by txrevise.

	Rscript scripts/prepareAnnotations.R --gtf Homo_sapiens.GRCh38.96.gtf.gz --tags Homo_sapiens.GRCh38.96.transcript_tags.txt --out Homo_sapiens.GRCh38.96.txrevise_annotations.rds

#### Step 4: Construct transcription events
Finally, we can use the `constructEvents.R` script to construct alternative transcription events. Since the implementation of the various algorithms in txrevise have not been optimised for efficiency, processing the full Ensembl GTF file can take several days. However, different genes can trivially processed in parallel. To simplify parallel execution, `constructEvents.R` has the `--batch` option that takes as an input two integers separated by a space. For example, '1 200' would mean that all genes are split into 200 batches and and only genes in the first are be processed. To process all genes, you simply need to iterate from '1 200' to '200 200'.

	Rscript scripts/constructEvents.R --annot Homo_sapiens.GRCh38.96.txrevise_annotations.rds --batch '1 2000' --out txrevise_events  --fill TRUE

#### Step 5: Merge output files
Each run of `constructEvents.R` produces up to six output files: alternative promoter, internal exon and 3' end events (labeled as upstream, contained and downstream) for two possible sets of shared exons (grp_1 and grp_2). See the [vignette](http://htmlpreview.github.io/?https://github.com/kauralasoo/txrevise/blob/master/inst/doc/construct_events.html) for more details on how the events are constructed.

To be able to use these events with transcript quantification software such as [Salmon](http://salmon.readthedocs.io/en/latest/) or [kallisto](https://pachterlab.github.io/kallisto/), we first need to merge all files from different batches. The `grep -v "^#"` command removes the comment lines from the beginning of each individual GFF3 file.

	cat txrevise_events/txrevise.grp_1.upstream.* | grep -v "^#" > txrevise.grp_1.upstream.gff3
	cat txrevise_events/txrevise.grp_1.contained.* | grep -v "^#" > txrevise.grp_1.contained.gff3
	cat txrevise_events/txrevise.grp_1.downstream.* | grep -v "^#" > txrevise.grp_1.downstream.gff3

	cat txrevise_events/txrevise.grp_2.upstream.* | grep -v "^#" > txrevise.grp_2.upstream.gff3
	cat txrevise_events/txrevise.grp_2.contained.* | grep -v "^#" > txrevise.grp_2.contained.gff3
	cat txrevise_events/txrevise.grp_2.downstream.* | grep -v "^#" > txrevise.grp_2.downstream.gff3

#### Step 6: Quantify event expression
The GFF3 files constructed by _txrevise_ can be used with any transcript quantification algorithm (e.g. [Salmon](http://salmon.readthedocs.io/en/latest/) or [kallisto](https://pachterlab.github.io/kallisto/)). **Importantly, since the promoter, internal exon and 3' end events constructed by txrevise share some of the same exons, they should always quantified independently**. For example, if you have six GFF3 files from txrevise (txrevise.grp_1.upstream.gff3, txrevise.grp_1.contained.gff3, txrevise.grp_1.downstream.gff3, txrevise.grp_2.upstream.gff3, txrevise.grp_2.contained.gff3, txrevise.grp_2.downstream.gff3), you should also run the quantification algorithm six times with each of the GFF3 file independently.

## Extracting event sequences for quantification
Many transcript expression quantification tools (e.g. Salmon or kallisto) do not directly work with transcript annotations in GFF3 format and require the transcript sequences in FASTA format instead. The simplest way to convert GFF3 annotations into transcript (or event) sequences is to use the `gffread` tool from the [cufflinks](http://cole-trapnell-lab.github.io/cufflinks/) package:

	gffread -w <output.fa> -g <reference_genome.fa> <input.gff3>

repository: kauralasoo/txrevise

gha_language: R

repo_name: kauralasoo/txrevise


text:
cabal-doctest
=============

[![Hackage](https://img.shields.io/hackage/v/cabal-doctest.svg)](https://hackage.haskell.org/package/cabal-doctest) [![Build Status](https://travis-ci.org/phadej/cabal-doctest.svg?branch=master)](https://travis-ci.org/phadej/cabal-doctest)

A `Setup.hs` helper for running `doctests`.

Simple example
--------------

For most use cases—a `.cabal` file with a single library containing
doctests—adapting the simple example located
[here](https://github.com/phadej/cabal-doctest/tree/master/simple-example)
will be sufficient. (Note that this example requires `Cabal-1.24` or later, but
you can relax this bound safely, although running doctests won't be supported
on versions of `Cabal` older than 1.24.)

To use this library in your `Setup.hs`, you should specify a `custom-setup`
section in your `.cabal` file. For example:

```
custom-setup
 setup-depends:
   base >= 4 && <5,
   Cabal,
   cabal-doctest >= 1 && <1.1
```

/Note:/ `Cabal` dependency is needed because of
[Cabal/GH-4288](https://github.com/haskell/cabal/issues/4288) bug.

You'll also need to specify `build-type: Custom` at the top of the `.cabal`
file. Now put this into your `Setup.hs` file:

```haskell
module Main where

import Distribution.Extra.Doctest (defaultMainWithDoctests)

main :: IO ()
main = defaultMainWithDoctests "doctests"
```

When you build your project, this `Setup` will generate a `Build_doctests`
module. To use it in a testsuite, simply do this:

```haskell
module Main where

import Build_doctests (flags, pkgs, module_sources)
import Data.Foldable (traverse_)
import Test.DocTest (doctest)

main :: IO ()
main = do
    traverse_ putStrLn args -- optionally print arguments
    doctest args
  where
    args = flags ++ pkgs ++ module_sources
```

Example with multiple .cabal components
---------------------------------------

`cabal-doctest` also supports more exotic use cases where a `.cabal` file
contains more components with doctests than just the main library, including:

* Doctests in executables
* Doctests in Internal libraries (if using `Cabal-2.0` or later)

Unlike the simple example shown above, these examples involve _named_
components. You don't need to change the `Setup.hs` script to support
this use case. However, in this scenario `Build_doctests` will generate extra
copies of the `flags`, `pkgs`, and `module_sources` values for each additional
named component.

Simplest approach is to use `x-doctest-components` field, for example
```
x-doctest-components: lib lib:internal exe:example
```

In that case, the testdrive is general:

```haskell
module Main where

import Build_doctests (Component (..), components)
import Data.Foldable (for_)
import Test.DocTest (doctest)

main :: IO ()
main = for_ components $ \(Component name flags pkgs sources) -> do
    print name
    putStrLn "----------------------------------------"
    let args = flags ++ pkgs ++ sources
    for_ args putStrLn
    doctest args
```

There's also a more explicit approach: if you have an executable named `foo`,
then separate values named `flags_exe_foo`, `pkgs_exe_foo`, and `module_sources_exe_foo` will
be generated in `Build_doctests`. If the name has hyphens in it
(e.g., `my-exe`), then `cabal-doctest` will convert those hyphens to
underscores (e.g., you'd get `flags_my_exe`, `pkgs_my_exe`, and
`module_sources_my_exe`).
Internal library `bar` values will have a `_lib_bar` suffix.

An example testsuite driver for this use case might look like this:

```haskell
module Main where

import Build_doctests
       (flags,            pkgs,            module_sources,
        flags_exe_my_exe, pkgs_exe_my_exe, module_sources_exe_my_exe)
import Data.Foldable (traverse_)
import Test.DocTest

main :: IO ()
main = do
    -- doctests for library
    traverse_ putStrLn libArgs
    doctest libArgs

    -- doctests for executable
    traverse_ putStrLn exeArgs
    doctest exeArgs
  where
    libArgs = flags            ++ pkgs            ++ module_sources
    exeArgs = flags_exe_my_exe ++ pkgs_exe_my_exe ++ module_sources_exe_my_exe
```

See
[this example](https://github.com/phadej/cabal-doctest/tree/master/multiple-components-example)
for more details.

Additional configuration
------------------------

The `cabal-doctest` based `Setup.hs` supports few extensions fields
in `pkg.cabal` files to customise the `doctest` runner behaviour, without
customising the default `doctest.hs`.

```
test-suite doctests:
  if impl(ghc >= 8.0)
    x-doctest-options: -fdiagnostics-color=never
  x-doctest-source-dirs: test
  x-doctest-modules: Servant.Utils.LinksSpec

  ...
```

* `x-doctest-options` Additional arguments passed into `doctest` command.
* `x-doctest-modules` Additional modules to `doctest`. May be useful if you
  have `doctest` in test or executables (i.e not default library complonent).
* `x-doctest-src-dirs` Additional source directories to look for the modules.

Notes
-----

* Recent versions of `Cabal` (for instance, 2.0) can choose to build a
  package's `doctest` test suite _before_ the library. However, in order for
  `cabal-doctest` to work correctly, the library _must_ be built first, as
  `doctest` relies on the presence of generated files that are only created
  when the library is built. See
  [#19](https://github.com/phadej/cabal-doctest/issues/19).

  A hacky workaround for this problem is to depend on the library itself in a
  `doctests` test suite. See
  [the simple example's .cabal file](https://github.com/phadej/cabal-doctest/blob/master/simple-example/simple-example.cabal)
  for a demonstration. (This assumes that the test suite has the ability to
  read build artifacts from the library, a separate build component. In
  practice, this assumption holds, which is why this library works at all.)

* `custom-setup` section is supported starting from `cabal-install-1.24`.
  For older `cabal-install's` you have to install custom setup dependencies
  manually.

* `stack` respects `custom-setup` starting from version 1.3.3. Before that
  you have to use `explicit-setup-deps` setting in your `stack.yaml`.
  ([stack/GH-2094](https://github.com/commercialhaskell/stack/issues/2094))

* There is [an issue in the Cabal issue tracker](https://github.com/haskell/cabal/issues/2327)
  about adding `cabal doctest` command. After that command is implemented,
  this library will be deprecated.

* You can use `x-doctest-options` field in `test-suite doctests` to
  pass additional flags to the `doctest`.

* For `build-type: Configure` packages, you can use
  `defaultMainAutoconfWithDoctests` function to make custom `Setup.hs` script.

* If you use the default `.` in `hs-source-dirs`, then running `doctests`
  might fail with weird errors (ambigious module errors). Workaround is
  to move sources under `src/` or some non-top-level directory.

* `extensions:` field isn't supported. Upgrade your `.cabal` file to use at least
  `cabal-version: >= 1.10` and use `default-extensions` or `other-extensions`.

* If you use QuickCheck properties (`prop>`) in your doctests,
  the `test-suite doctest` should depend on `QuickCheck` and `template-haskell`.
  This is a little HACK: These dependencies aren't needed to build the
  `doctests` test-suite executable.  However, as we let `Cabal` resolve
  dependencies, we can pass the resolved (and installed!) package identifiers to
  to the `doctest` command.  This way, `QuickCheck` and `template-haskell` are
  available to `doctest`, otherwise you'll get errors like:

```
    Variable not in scope:
      mkName
        :: [Char]
           -> template-haskell-<IP_ADDRESS>:Language.Haskell.TH.Syntax.Name
```

or

```
    Variable not in scope:
      polyQuickCheck
        :: Language.Haskell.TH.Syntax.Name -> Language.Haskell.TH.Lib.ExpQ
```

Copyright
---------

Copyright 2017 Oleg Grenrus.

Available under the BSD 3-clause license.


text:
---
layout: default
---

# 1996-02-20 - JavaScript to grab email

## Header Data

From: Jyri Kaljundi<EMAIL_ADDRESS>To<EMAIL_ADDRESS>Message Hash:<PHONE_NUMBER>a6d7ff81f35b63d5fac2e29bb96f01f5aacb6c522f0efe9d546406

Message ID<EMAIL_ADDRESS>Reply To: _N/A_

UTC Datetime: 1996-02-20 16:11:47 UTC

Raw Date: Wed, 21 Feb 1996 00:11:47 +0800

## Raw message

```
{% raw  %}From: Jyri Kaljundi<EMAIL_ADDRESS>Date: Wed, 21 Feb 1996 00:11:47 +0800
To<EMAIL_ADDRESS>Subject: JavaScript to grab email
Message-ID<EMAIL_ADDRESS>MIME-Version: 1.0
Content-Type: text/plain

Another annoying feature in JavaScript and Netscape. Have a look at
<http://www.popco.com/grabtest.html>

The page uses JavaScript to steal your e-mail address and sends a test
e-mail message to the address it grabbed. It works with Netscape, which
is probably only browser supporting JavaScript.

Probably there will soon be thousands of pages which include this code,
and people using Netscape 2.0 will be spammed with commercial messages.
So just put some false e-mail address in your Netscape browser to disable
this feature.

The script was developed and announced by Glenn Fleishman, moderator of
the Internet Marketing mailing list. Thanks for letting people know.

Jüri Kaljundi
<EMAIL_ADDRESS>Digiturg http://www.digit.ee/

{% endraw %}
```

## Thread

+ Return to [February 1996](/archive/1996/02)

+ Return to "[Ian Goldberg <iang@cs.berkeley.edu>](/authors/ian_goldberg_iang_at_cs_berkeley_edu_)"
+ Return to "[Jeff Barber <jeffb@sware.com>](/authors/jeff_barber_jeffb_at_sware_com_)"
+ Return to "[Jyri Kaljundi <jk@digit.ee>](/authors/jyri_kaljundi_jk_at_digit_ee_)"
+ Return to "["Mark M." <markm@voicenet.com>](/authors/mark_m__markm_at_voicenet_com_)"
+ Return to "[Mike Rose <mrose@stsci.edu>](/authors/mike_rose_mrose_at_stsci_edu_)"

+ 1996-02-20 (Wed, 21 Feb 1996 00:11:47 +0800) - JavaScript to grab email - _Jyri Kaljundi<EMAIL_ADDRESS>  + 1996-02-20 (Wed, 21 Feb 1996 02:47:17 +0800) - [Re: JavaScript to grab email](/archive/1996/02/0b1b582353daa5815af976d47631dd64221353acb0d5b4d0f7bfcbc02b26321d) - _Mike Rose<EMAIL_ADDRESS>    + 1996-02-20 (Wed, 21 Feb 1996 06:35:35 +0800) - [Re: JavaScript to grab email](/archive/1996/02/4b1c80383223be07913a60b95d260e1d8dc48d7dcaeb5b1787bbc7fb00c6528c) - _"Mark M."<EMAIL_ADDRESS>      + 1996-02-21 (Wed, 21 Feb 1996 12:46:27 +0800) - [Re: JavaScript to grab email](/archive/1996/02/0f9e034c7516cfbce030c40aba87ccefccfd417895dfa092988b10ec79d65fee) - _Jeff Barber<EMAIL_ADDRESS>    + 1996-02-20 (Wed, 21 Feb 1996 07:50:55 +0800) - [Re: JavaScript to grab email](/archive/1996/02/8ec3d819028e52155fcbedf1d417f0d22298d4e1163cddea2d5378f76b3eb7aa) - _Mike Rose<EMAIL_ADDRESS>    + 1996-02-21 (Wed, 21 Feb 1996 08:49:08 +0800) - [Re: JavaScript to grab email](/archive/1996/02/795d78534a4e8fd22bc48ebe78ebd363e27abf50458345ab838b2d54d8daf84b) - _Mike Rose<EMAIL_ADDRESS>    + 1996-02-21 (Thu, 22 Feb 1996 03:50:20 +0800) - [Re: JavaScript to grab email](/archive/1996/02/8e7584abe4a687c4a3d3d77a20e3beae16b978df8c0c00a51dd07cf83e65e71f) - _Ian Goldberg<EMAIL_ADDRESS>

gha_license_id: MIT


text:
## A simple demo pages

just use normal html pages with introduce some features for administrators and managers

### Nginx location proxy for prefixed url

use specified url for location module then confirm the end of location '/'

`/` == `host:post/` pattern is same to
`/plain/` == `host:post/` pattern

```
100 server {
101     server_name demo.blititor.com;
102
103     client_body_in_file_only clean;
104     client_body_buffer_size 32K;
105
106     client_max_body_size 20M;
107
108     sendfile on;
109     send_timeout 300s;
110
111     location /plain/ {
112         proxy_set_header X-Real-IP $remote_addr;
113         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
114         proxy_set_header Host $http_host;
115         proxy_set_header X-NginX-Proxy true;
116         proxy_pass http://<IP_ADDRESS>:3011/;
117         proxy_redirect off;
118     }
119 }
```

### History

- 1.0.0: initial release

### Credit

- soomtong (github.com/soomtong)

repository: soomtong/blititor

branch_name: refs/heads/develop

gha_language: JavaScript

repo_name: soomtong/blititor


text:
# Telangiopsis D.A. Eggert & T.N. Taylor, 1971 GENUS

#### Status
ACCEPTED

#### According to
Interim Register of Marine and Nonmarine Genera

#### Published in
Botanical Gazette 132: -. [30-37]

#### Original name
null

### Remarks
null

repository: mdoering/backbone

repo_name: mdoering/backbone


text:
+++
title = "HTTP"
description = "Understanding the format of the web"
tags = ["web", "http"]
slug = "http"
previous = "exceptions.html"
next = "web/http-post.html"
+++
HTTP stands for Hypertext Transfer Protocol. It is a standard for how requests and responses should be formatted for
a server and a web browser. When you open a link in your web browser, you are sending a HTTP request to a server and it
is responding with a HTTP response. The browser then takes the response, parses it and displays it to the user.

Let us look at a HTTP request for PHP Apprentice. This is the request sent to the server from Firefox:
```http
GET /basics.html HTTP/2.0
Host: phpapprentice.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:65.0) Gecko/20100101 Firefox/65.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Pragma: no-cache
Cache-Control: no-cache
```
The first line of the request tells us three things. First, the method of the request. A `GET` request tells the server
that the browser just wants to download whatever is stored at the link. The second value `/basics.html` determines
what should be loaded from `phpapprentice.com`. Finally, the third value tells the server what type of HTTP request it is.

All of the lines below the top line are key/value pairs called headers. They give the server different pieces of information
about the client and the request to the server. For example, `Host` tells the server what website the browser is
trying to access.

In response to the client's HTTP request, the server will respond with a HTTP response. Here is the response for the above request:
```http
HTTP/2.0 200 OK
server: GitHub.com
content-type: text/html; charset=utf-8
last-modified: Sun, 13 Jan 2019 22:57:50 GMT
etag: W/"5c3bc26e-15a0"
access-control-allow-origin: *
expires: Mon, 18 Feb 2019 02:38:31 GMT
cache-control: max-age=600
content-encoding: gzip
x-github-request-id: 8596:461C:46C99E:5BB366:5C6A184F
accept-ranges: bytes
date: Mon, 18 Feb 2019 03:06:02 GMT
via: 1.1 varnish
age: 52
x-served-by: cache-fty21342-FTY
x-cache: HIT
x-cache-hits: 2
x-timer: S1550459163.720652,VS0,VE0
vary: Accept-Encoding
x-fastly-request-id: 00f0eec6f3037e428b8fc86742fe0f9965501a51
content-length: 2084

<!DOCTYPE html>
<html lang="en">
<head>
...
```

The top line of the response is in two parts, similar to a request.
The first part `HTTP/2.0` indicates the connection type, like the client.
The second part `200 OK` is the response code. `200` indicates a successful response.
There are many different response codes, the most famous being `404 Not Found`.

A response will have headers as well, but they are to give the client information about the response. As you can see,
the `Content-Type` is set to `text/html`, which tells the client how to render the content of the response.

Notice that below the list of response headers, there is also the content of the response. In this case, it is just HTML
which the browser can you use to display the page.

HTTP requests and responses are the core of web communication. Next, we will look at a request type other than `GET`.

repository: restoreddev/phpapprentice

gha_language: CSS

gha_license_id: NOASSERTION

repo_name: restoreddev/phpapprentice


text:
# Meal Tracker
#### This is a repository for Angular 2 Template

## Description
This App will help you track the type of Meal eaten and Details and calories in the food.Basically you can Edit in case you have a typing error or you remember something which will be more helpful.The App comes with extra features which will give you the power to track your health

## Technologies Used
* HTML
* TypeScript
* Node
* Bower
* Angular 2

#### Set-up Information and technology overview

This project was setup using the Angular2  to create its structure. Kindly ensure you have angular 2, nodeJs, and typescript environments set up. Below are instructions to help:

#### Development server

Run gulp build gulp serve for a dev server. It will automatically fire start your browser. The app will automatically reload if you change any of the source files.

#### Further help

To get more help on the Angular use contact<EMAIL_ADDRESS>or go check out the Angular.

repository: toelapiut/meal-tracker

repo_name: toelapiut/meal-tracker


text:
Capistrano Deployment Tags
==========================
This plugin to Capistrano will add a timestamped Git tag
at each deployment, automatically.  It is intended to be used with
the multistage recipe and will tag each release by environment.
You can, however, use it without multistage simply by setting :branch
and :stage in your recipe.

What It Does
------------
Simply: it makes it so you can track your deployments from Git.
If I were to issue the command:

`cap production deploy`

This would result in one new git tag with the environment and
timestamp:

`production-2012.04.02-203155-utc`

These tags can be used for any number of useful things including
generating statistics about deployments per day/week/year, tracking
code size over a period of time, detecting Rails migrations, and
probably a thousand other things I haven't thought of.

Usage
-----
capistrano-deploytags is available on
[rubygems.org](https://rubygems.org/gems/capistrano-deploytags).
You can install it from there with:

`gem install capistrano-deploytags`

If you use Bundler, be sure to add the gem to your Gemfile.
In your Capistrano `config/deploy.rb` you should add:

`require 'capistrano-deploytags'`

This will create two tasks, one that runs before deployment and one
that runs after.

*NOTE:* You will be creating and pushing tags from the version of the code in the
current checkout. This plugin needs to be run from a clean checkout of your
codebase. You should be deploying from a clean checkout anyway, so in most
cases this is not a restriction on how you already do things. The plugin will
check if your code is clean and complain if it is not.

*ALSO:* The plugin will do a pull to make sure you have the code on your local
system that will actually be deployed before checking the tree for changes.
Know this ahead of time as this may affect how you deal with your deployment
branches.

Setting the Remote
------------------
By default, Capistrano Deploytags will use the first remote in the list returned
by `git remote`. If you prefer to use a different remote, then you may change the
`:git_remote` setting from your `deploy.rb`, the stage, or on the command line with
`-S git_remote=your-remote`.

Working on Your Deployment Scripts
----------------------------------
Because you must have a clean tree to deploy, working on your deployment
scripts themselves can be a bit frustrating unless you know how to make it
work. The easiest way around this problem is to simply commit your changes
before you deploy. You do not have to push them. The plugin will then
happily carry on deploying without complaint.

Alternatively, you could disable the plugin temporarily with one of the
methods described below.

Disabling Tagging for a Stage
-----------------------------
Sometimes you do not want to enable deployment tagging for a particular
stage. In that event, you can simply disable tagging by setting `no_deploytags`
like so:

```ruby
set :no_deploytags, true
```

You can also set this from the command line at any time with `-S no_deploytags=true`.

*NOTE:* this will disable the use of the plugin's functionality entirely for
that stage. The tasks will run, but will do nothing. This means that tasks that
are hooked to the Capistrano Deploytags tasks will also still run, but they may
find their expectations are not met with regards to the cleanliness of the git
tree.

Customizing the Tag Format
--------------------------
You may override the time format in `config/deploy.rb`:

```ruby
set :deploytag_time_format, "%Y.%m.%d-%H%M%S-utc"
```

Viewing Deployment History
--------------------------
It's trivial to view the deployment history for a repo. From a checkout
of the repo, type `git tag -l -n1`. The output looks something like:

```
dev-2013.07.22-105130 baz deployed a4d522d9d to dev
dev-2013.07.22-113207 karl deployed 4c43f8464 to dev
dev-2013.07.22-114437 gavin deployed 776e15414 to dev
dev-2013.07.22-115103 karl deployed 619ff5724 to dev
dev-2013.07.22-144121 joshmyers deployed cf1ed1a02 to dev
```

A little use of `grep` and you can easily get the history for a
particular (e.g. `git tag -l -n1 | grep dev`).

It should be noted that the names used when tags are created are the
local user name on the box where the deployment is done.

Helpful Git Config
------------------
You might find it useful to add this to your ~/.gitconfig in order
to get a nice history view of the commits and tags.

```ini
[alias]
   lol = log --pretty=oneline --abbrev-commit --graph --decorate
```

You can then view the list by typing `git lol` from the checked out
code path.

Deploying a Previous Commit
---------------------------
Because you have to actually be on the head of the branch you are
deploying in order for tagging to work properly, deploying a previous
commit doesn't work as you might expect. The simple solution is to
create a new branch from the previous commit you wish to deploy and
supplying `-S branch=<new branch>` as arguments to Capistrano.

Running from Jenkins
--------------------
Because Jenkins will check out the code with the current revision
number you will be in a detached state. This causes the plugin to be
unhappy about the git tree. The solution is to add `-S branch=$GIT_COMMIT`
to the cap deploy line called from your Jenkins build. This will cause
the diffs and comparisons done by the deploytags gem to be correct.

Credits
-------
This software was written by [Karl Matthias](https://github.com/relistan)
with help from [Gavin Heavyside](https://github.com/gavinheavyside) and the
support of [MyDrive Solutions Limited](http://mydrivesolutions.com).

License
-------
This plugin is released under the BSD two clause license which is
available in both the Ruby Gem and the source repository.

repository: bramswenson/capistrano-deploytags

repo_name: bramswenson/capistrano-deploytags


text:
---
layout: engineering-education
status: publish
published: true
url: /engineering-education/edge-tf-lite/
title: Machine Learning on Edge Devices Using TensorFlow Lite
description: This article details machine learning on edge computing devices which use TensorFlow Lite and RaspberryPi. Talking about the advantages of on-device machine learning inference such as Latency, Bandwidth, privacy, and security.
author: rohan-reddy
date: 2020-07-30T00:00:00-10:00
topics:
- Edge Computing
excerpt_separator: <!--more-->
images:

  - url: /engineering-education/edge-tf-lite/hero.jpg
    alt: edge device example image
---
[Edge Devices](https://en.wikipedia.org/wiki/Edge_device) are computing devices that live at the "edge" of the network and perform work in the exact location that is needed. Edge devices include [IoT devices](https://en.wikipedia.org/wiki/Internet_of_things), [smart home equipment](https://www.pcmag.com/news/the-best-smart-home-devices-for-2020) and computers embedded as household or industrial items. IoT devices are growing rapidly and continue to do so for the next couple of years.
<!--more-->

![img](/engineering-education/edge-tf-lite/logo.png)

![img](/engineering-education/edge-tf-lite/traditional.png)

Image Source: [Udacity](https://www.udacity.com/)

The above diagram shows the traditional flow of data in a machine learning system. A device collects information from the environment, which is sent via a network connection to a back-end server that performs inference. The server sends some data back to the device if necessary.

![img](/engineering-education/edge-tf-lite/edge.png)

Image Source: [Udacity](https://www.udacity.com/)

But we can perform inference on the device itself, we get to skip a bunch of steps. **There are huge advantages to this**.

* **Bandwidth**. Instead of sending a lot of data to the server and back, we can use on-device inference to send tiny amounts of data when needed. This is useful
for remote places with no proper internet.
* **Latency**. Sending data to a server involves a round-trip delay which gets in the way when working with real-time data. This is no longer an issue when our model
is at the edge. When inference is super fast we can solve high-performance actions like real-time object tracking for a robot.
* **Privacy and Security**. When data stays on device users benefit from increased privacy and security since personal information never leaves their devices. This benefits privacy-sensitive applications like security webcams and health-care data.

### RaspberryPi + TensorFlow Lite

The [Raspberry Pi](https://www.raspberrypi.org/) (R Pi) is a low cost, very small computer that runs a Linux-based operating system called [Raspbian or Raspberry Pi OS](https://www.raspberrypi.org/downloads/). It's often used for building prototype devices since it has fairly typical hardware specifications and it's easy to connect to sensors and peripherals like cameras.

[TensorFlow](https://www.tensorflow.org/) is a popular open-source machine learning framework, which is used for a variety of tasks. [TensorFlow Lite](https://www.tensorflow.org/lite/) is a lightweight library
for deploying models on mobile and embedded devices. It is a lighter, less-featured deep learning framework for on-device inference. TensorFlow lite provides APIs in Python (which we will be using on Raspberry Pi), Java (for Android) and Swift (for iOS). We can see Tensorflow lite in action in our day-to-day applications, for example - Gmail uses TensorFlow Lite for Smart Reply and Auto-Complete. Google Assistant uses it for Natural Language Processing and Understanding. TensorFlow Lite can be used where developing, inferring from and deploying TensorFlow models is not efficient in terms of memory or CPU capacity.

### How to use TensorFlow lite

TensorFlow lite has two main components:

* **TensorFlow Lite Converter**. Training a neural network is a time consuming process, especially if it is on a large dataset. So, we [*save*](https://www.tensorflow.org/tutorials/keras/save_and_load) a model in formats like [`.h5`](https://en.wikipedia.org/wiki/Hierarchical_Data_Format) or `SavedModel`. Saving a model makes it easier to share and deploy the model. TensorFlow lite converter is used to convert these models into an efficient form for use by the interpreter.

A `SavedModel` model for a simple image classification model trained on [MNIST](http://yann.lecun.com/exdb/mnist/) data has a size of 1.5 MB, the same model converted to `.tflite` is about 300 KB.

* **TensorFlow Lite Interpreter**. TensorFlow Lite Interpreter runs specially optimized models on many different hardware types, including mobile phones, embedded Linux devices, and microcontrollers.

The development workflow for using TensorFlow lite involves the following steps:

1. Choose a model. A model is a data structure that contains the logic and knowledge of a machine learning network trained to solve a particular problem. We can train our own model for custom business problems or obtain a pre-trained model from [TensorFlow Hub](https://www.tensorflow.org/hub).
2. Convert the model. TensorFlow lite is designed to execute models efficiently on mobile and other embedded devices with limited compute and memory resources. Some of this efficiency comes from the use of a special format for storing models. TensorFlow models must be converted into this format before they can be used by TensorFlow Lite. The [TensorFlow Lite converter](https://www.tensorflow.org/lite/convert) is a tool available as a Python API that converts trained TensorFlow models into the TensorFlow Lite format.
3. Run inference with the model. Inference is the process of running data through a model to obtain predictions. It requires a model, an interpreter, and input data.
4. Optimize the model. TensorFlow Lite provides tools to optimize the size and performance of your models, often with minimal impact on accuracy. [Model Optimization Toolkit](https://www.tensorflow.org/lite/guide/get_started#model_optimization_toolkit).

### Getting Started

The best way to run TensorFlow lite interpreter is running python scripts. You can build Tensorflow from source or install Tensorflow lite interpreter package using `pip`. For the former method visit the
documentation [guide](https://www.tensorflow.org/install/source_rpi). Building from source is useful when you want to [convert a model](https://www.tensorflow.org/lite/devguide#2_convert_the_model_format) or train a model.

The **awesome thing about TensorFlow lite** is that you don't need to be a machine learning expert to start doing cool experiments. The only pre-requisite is a basic knowledge of [Python](https://www.learnpython.org/). The most common applications of deep learning (like Object Detection, Pose Estimation, Smart Reply, ...) have been [implemented by the community](https://www.tensorflow.org/lite/models) and they are available for developers to use off-the-shelf.
In this article we will be performing [Image Classification](https://developers.google.com/machine-learning/practica/image-classification) one of the most common applications of deep learning.

#### Image Classification

An image classification model takes an image file and predicts what the image is or represents. An image classification model is trained to recognize various classes of images. For this tutorial we will use the `mnist` dataset and train a neural network to identify hand written digits (The most commonly used dataset). I use TensorFlow 2 (not TensorFlow lite) to build a model and train it, and then convert the model into TF lite model.

To install Tensorflow: `pip install tensorflow`.

```python
import tensorflow as tf
# prepping the data
fashion_mnist = tf.keras.datasets.fashion_mnist
(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()
train_images = train_images / 255.0
test_images = test_images / 255.0

#building the model
model = tf.keras.Sequential([
    tf.keras.layers.Flatten(input_shape=(28, 28)),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dense(10)
])

#train the model
model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])
model.fit(train_images, train_labels, epochs=10)

#converting to TF Lite
model.save("my_model)
converter = tf.lite.TFLiteConverter.from_saved_model("my_model")
tflite_model = converter.convert()
interpreter = tf.lite.Interpreter(model_content=tflite_model)
interpreter.allocate_tensors()

input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
```

#### Image Classification with Pretrained Model

Run the following commands in the command line to set download the files and software.

`pip3 install https://dl.google.com/coral/python/tflite_runtime-2.1.0.post1-cp37-cp37m-linux_armv7l.whl`

`$ wget https://storage.googleapis.com/download.tensorflow.org/models/tflite/mobilenet_v1_1.0_224_quant_and_labels.zip`

We use the [MobileNet](https://arxiv.org/abs/1704.04861) image classification model. MobileNet is a [pretrained model](https://towardsdatascience.com/transfer-learning-from-pre-trained-models-f2393f124751), which means it has been trained on a very large dataset to classify over 1000 classes of images including people, objects, animals, etc.

`image-classification.py`

![img](/engineering-education/edge-tf-lite/steps.png)

Image Source: [TensorFlow.org](https://www.tensorflow.org)

```python
from tflite_runtime.interpreter import Interpreter
from PIL import Image
import numpy as np
#load the model
interpreter = Interpreter(model_path="mobilenet_v1_1.0.224.tflite")
#allocate memory for the model
interpreter.allocate_tensors()
#get model input tensors details.
input_details = interpreter.get_input_details()
#gets model output details. Returns a list of output details.
output_details = interpreter.get_ouptut_details()
#load image from memory
img = Image.open(filename).convert("RGB")
#preprocess the image
img = img.resize((224, 224))
#converting image to an numpy array
input_data = np.array(img)
#np.expand_dims expands the shape of an array. This is done to take into account the *batch* dimension.
input_data = np.expand_dims(input_data, axis=0)
#Point data to be used for testing the interpreter and run it
interpreter.set_tensor(input_details[0]["index"], input_data)
#invoke the interpreter for inference
interpreter.invoke()
#Obtain Results. The function get_tensor() returns a copy of the tensor data.
predictions = interpreter.get_tensor(output_details[0]["index"])
#The model outputs a probability for every class it has been trained on, we take the top 10 most probable classes.
top_indices = np.argsort(predictions)[::-1][:10]

print(labels[top_indices[0]], predictions[top_indices[0]])
```

We can expect sample input and output to be.

![img](/engineering-education/edge-tf-lite/doginput.png)

#### TensorFlow Lite and TensorFlow compatibility

TensorFlow lite supports a number of TensorFlow operations used for getting predictions. As TensorFlow operations are optimized for TensorFlow lite, they may be omitted or combined. Since the set of TensorFlow Lite operations is smaller than TensorFlow's, not every model is convertible. Even for supported operations, very specific usage patterns are sometimes expected, for performance reasons. Please refer to [this guide](https://www.tensorflow.org/lite/guide/ops_compatibility) for complete list of supported and unsupported operations.

### Conclusion

Machine Learning is a powerful tool which can help automate many tasks which are beyond the scope of classical programming, it is assumed that you need a powerful computing machine with a GPU to train models. But with the evolution of hardware and optimization of software for low-end computing devices, we can perform complex machine learning tasks on devices such as micro-controllers, mobile phones, smart home devices, etc.

### Resources & References
1. [Free Udacity Course](https://www.udacity.com/course/intro-to-tensorflow-lite--ud190)
2. [Full Code](https://github.com/tensorflow/examples/tree/master/lite/examples/image_classification/raspberry_pi)
3. [Image Classification](https://www.tensorflow.org/lite/models/image_classification/overview)
4. [TensorFlow Lite Guide](https://www.tensorflow.org/lite/guide)
5. [McKinsey Co](https://www.mckinsey.com/industries/private-equity-and-principal-investors/our-insights/growing-opportunities-in-the-internet-of-things)

repository: mwanikialex/engineering-education

repo_name: mwanikialex/engineering-education


text:
# Description of the results
This folder contains the results of different simulations in this work. The results are saved on ``pickle-file``.

## Folder structure
Here, you find five different ``pickle-files``:
 - ``sim_results_fixed_target_15.pkl``: First results achieved in this work (outdated)
 - ``sim_results_fixed_target_15_Validation.pkl``: Results during testing phase, for the case ``offers_input.csv`` with Energy_target = 15 kWh for all episodes
 - ``sim_results_fixed_target_15_batch_improve.pkl``: Latest training results for the case ``offers_input.csv`` with Energy_target = 15 kWh for all episodes
 - ``training_results_30_partners.pkl``: Training results for the case ``offers_input_30_partners.csv``
 - ``training_results_30_partners_batch_improve.pkl``: Improved training results for the same case ``offers_input_30_partners.csv``, with the latest improvements in the training loop

## Results file description:
The results are organized in a python dictionary called ``data``:
 - ``data['agents']``: it has the number and label of the RL policies
 - ``data['simulation']``: it has the Energy_target, environment, number of episodes, number of steps per episode
 - ``data['outcome']``: it has the total reward, total regret, last step, Energy_target per episode
 - ``data['policy_sol']``: it has the actions, state, reward, regret Q-action function per number of steps
 - ``data['policy_dist']``: it has the final Q-action function per episode

branch_name: refs/heads/main


text:
# Land of Literra
A text-based online adventure role-playing game. Written in Python and C#.
Originally an idea I had a long time ago, turned into a school project.

##### Technical
The whole idea of how I plan on making this, is that everything will be runned and controlled from the server, meaning that with the clients being so versatile and controllable from the server, all updates will only have to go through the server without clients having to update their software.
There will be multiple servers clients can connect to, and serverlists are downloaded from one of Literra's admin's servers everytime the client loads. The remote location for the serverlist is currently hardcoded into the client, but will likely later be configurable by any admin of a server.

##### Game
The game itself will feature a large map with many quests, items, buildings, arenas, and other entities. Players will be interactable with eachother, not just by chat but also "physically". If any player afflict the world in any way, the changes will be for all other players too. Think of an Elder Scrolls game - textbased and online.

###### Duels and PvP
There will be two ways to fight in this game. The reason for this is because the first way can easily become unfair due to multiple factors. The first way being that, say in order to attack an opponent, one would write "use sword on player". But seeing this will be bad for people with high ping and slow type-speed, a second option was needed without removing the first one being intense and making the game more exiting.
The second way of fighting will be simply turnbased. This way, one can have a fair fight(..if we disregard any random-factors in e.g. damagemultipliers).

###### Map
Maps will be single files on the servers with the ability to load custom maps into the server. Being the maps are solely on the server, they can be updated and expanded at anytime, and perhaps one day, even with the players logged on and the server running.
###### Misc
"literra" means "letters" in latin, - "letters" as in characters, not messages.
"LoT" is short for "Land of Literra", because I refuse to name it "LoL". Deal with it.

## License

The MIT License (MIT)

Copyright (c) 2015 John Hiness

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

 IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

repository: JohnHiness/LandOfLiterra

repo_name: JohnHiness/LandOfLiterra


text:
# crontab-core-engine

## Install

```bash
// clone repository and run
npm install
```

## Dependency
Expects Redis to be running on localhost. Config can be changed in `default.json`.

## Run

You can either manually start the required processes or use `pm2`

### Manual

Start 2 separate processes
1. ./lib/time/program.js
1. ./lib/scheduling/program.js

> cwd need to be the root of this repository

### PM2 (Node Process Manager 2)

```bash
npm install -g pm2
```

```bash
pm2 start ecosystem.config.js
```

> cwd need to be the root of this repository

## Usage

### Create a Crontab Entry

HTTP POST http://localhost:8670/crontab
```json
{
	"id" : "foo",
	"text" : "* * * * * PAYLOAD_A \n 0 * * * * PAYLOAD_B"
}
```

`text` is in crontab format; each line represents a crontab job.

https://en.wikipedia.org/wiki/Cron

In this example, `PAYLOAD_A` will be triggered every minutes and `PAYLOAD_B` will be triggered once an hour.

### Delete
HTTP DELETE http://localhost:8670/crontab/:id

ex: http://localhost:8670/crontab/foo

## Handle Job Execution

In this fork of [taskmill-core-cron](https://github.com/a7medkamel/taskmill-core-cron) you need to change the `handler` method in `./lib/time/program.js` to execute your desired behavior.

The current handler will console.log() the event

> This will be refactored in a future version of this fork.


text:
# Configure Monaco

> Monaco support is experimental. There is a known issue that Monaco's elements will be misplaced when the slide scales. We are working on it and trying to find a solution. Before that, the workaround there is to set the scale to `1:1` and use the browser's zoom to manually scale to the size of your screen.

Create `./setup/monaco.ts` with the following content:

```ts
import { defineMonacoSetup } from '@slidev/types'

export default defineMonacoSetup(async (monaco) => {
  // use `monaco` to configure
})
```

Learn more about [configuring Monaco](https://github.com/Microsoft/monaco-editor).

## Usage

To use Monaco in your slides, simply append `{monaco}` to your code snippets:

~~~js
//```js
const count = ref(1)
const plusOne = computed(() => count.value + 1)

console.log(plusOne.value) // 2

plusOne.value++ // error
//```
~~~

To

~~~js
//```js {monaco}
const count = ref(1)
const plusOne = computed(() => count.value + 1)

console.log(plusOne.value) // 2

plusOne.value++ // error
//```
~~~

## Exporting

By default, Monaco will ONLY work on `dev` mode. If you would also like to have it available in the exported SPA, you can configure it in your frontmatter:

```yaml
---
monaco: true # default "dev"
---
```

## Types Auto Installing

When you use TypeScript with Monaco, types for dependencies will be installed to the client-side automatically.

~~~ts
//```ts {monaco}
import { ref } from 'vue'
import { useMouse } from '@vueuse/core'

const counter = ref(0)
//```
~~~

In the example above, just make sure `vue` and `@vueuse/core` are installed locally as dependencies / devDependencies, Slidev will handle the rest and your editor will just work!

gha_language: TypeScript


text:
# A sample project to practice swift.

## Helpful Resources:

### Protocol Oriented Swift:

- https://developer.apple.com/videos/play/wwdc2015/408/
- http://masilotti.com/better-swift-unit-testing/
- https://www.raywenderlich.com/148448/introducing-protocol-oriented-programming