text:
# KEP-1027: Union types

## Table of Contents

<!-- toc -->
- [Release Signoff Checklist](#release-signoff-checklist)
- [Summary](#summary)
- [Motivation](#motivation)
  - [Goals](#goals)
  - [Non-Goals](#non-goals)
- [Proposal](#proposal)
  - [User Stories (Optional)](#user-stories-optional)
    - [Story 1](#story-1)
    - [Story 2](#story-2)
  - [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional)
  - [Risks and Mitigations](#risks-and-mitigations)
- [Design Details](#design-details)
  - [Discriminator Field](#discriminator-field)
  - [Go Markers](#go-markers)
    - [Discriminator Values](#discriminator-values)
      - [Empty Union Members](#empty-union-members)
    - [Examples](#examples)
  - [OpenAPI](#openapi)
  - [Normalization and Validation](#normalization-and-validation)
    - [Normalization](#normalization)
    - [Validation](#validation)
    - [Ratcheting Validation](#ratcheting-validation)
    - [Migrating existing unions](#migrating-existing-unions)
  - [Test Plan](#test-plan)
      - [Prerequisite testing updates](#prerequisite-testing-updates)
      - [Unit tests](#unit-tests)
      - [Integration tests](#integration-tests)
      - [e2e tests](#e2e-tests)
  - [Graduation Criteria](#graduation-criteria)
    - [Alpha Graduation](#alpha-graduation)
  - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy)
  - [Version Skew Strategy](#version-skew-strategy)
- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire)
  - [Feature Enablement and Rollback](#feature-enablement-and-rollback)
  - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning)
  - [Monitoring Requirements](#monitoring-requirements)
  - [Dependencies](#dependencies)
  - [Scalability](#scalability)
  - [Troubleshooting](#troubleshooting)
- [Implementation History](#implementation-history)
- [Drawbacks](#drawbacks)
- [Alternatives](#alternatives)
<!-- /toc -->

## Release Signoff Checklist

Items marked with (R) are required *prior to targeting to a milestone / release*.

- [ ] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR)
- [ ] (R) KEP approvers have approved the KEP status as `implementable`
- [ ] (R) Design details are appropriately documented
- [ ] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
  - [ ] e2e Tests for all Beta API Operations (endpoints)
  - [ ] (R) Ensure GA e2e tests for meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md)
  - [ ] (R) Minimum Two Week Window for GA e2e tests to prove flake free
- [ ] (R) Graduation criteria is in place
  - [ ] (R) [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) must be hit by [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md)
- [ ] (R) Production readiness review completed
- [ ] (R) Production readiness review approved
- [ ] "Implementation History" section is up-to-date for milestone
- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
- [ ] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

<!--
**Note:** This checklist is iterative and should be reviewed and updated every time this enhancement is being considered for a milestone.
-->

[kubernetes.io]: https://kubernetes.io/
[kubernetes/enhancements]: https://git.k8s.io/enhancements
[kubernetes/kubernetes]: https://git.k8s.io/kubernetes
[kubernetes/website]: https://git.k8s.io/website

## Summary

Modern data model definitions like OpenAPI v3 and protobuf (versions 2 and 3)
have a keyword to implement “oneof” or "union". They allow APIs to have a better
semantics, typically as a way to say “only one of the given fields can be
set”. We currently have multiple occurrences of this semantics in kubernetes core
types, at least:
- VolumeSource is a structure that holds the definition of all the possible
  volume types, only one of them must be set, it doesn't have a discriminator.
- DeploymentStrategy is a structure that has a discrminator
  "DeploymentStrategyType" which decides if "RollingUpate" should be set

The problem with the lack of solution is that:
- The API is implicit, and people don't know how to use it
- Clients can't know how to deal with that, especially if they can't parse the
  OpenAPI
- Server can't understand the user intent and normalize the object properly

## Motivation

Currently, changing a value in an oneof type is difficult because the semantics
is implicit, which means that nothing can be built to automatically fix unions,
leading to many bugs and issues:
- https://github.com/kubernetes/kubernetes/issues/35345
- https://github.com/kubernetes/kubernetes/issues/24238
- https://github.com/kubernetes/kubernetes/issues/34292
- https://github.com/kubernetes/kubernetes/issues/6979
- https://github.com/kubernetes/kubernetes/issues/33766
- https://github.com/kubernetes/kubernetes/issues/24198
- https://github.com/kubernetes/kubernetes/issues/60340

And then, for other people:
- https://github.com/rancher/rancher/issues/13584
- https://github.com/helm/charts/pull/12319
- https://github.com/EnMasseProject/enmasse/pull/1974
- https://github.com/helm/charts/pull/11546
- https://github.com/kubernetes/kubernetes/pull/35343

This is replacing a lot of previous work and long-standing effort:
- Initially: https://github.com/kubernetes/community/issues/229, then
- https://github.com/kubernetes/community/pull/278
- https://github.com/kubernetes/community/pull/620
- https://github.com/kubernetes/kubernetes/pull/44597
- https://github.com/kubernetes/kubernetes/pull/50296
- https://github.com/kubernetes/kubernetes/pull/70436

Server-side [apply](http://features.k8s.io/555) is what enables this proposal to
become possible.

### Goals

Provide a simple mechanism for API authors to label fields of a resource as
members of a oneOf, in order to receive standardized validation and
normalization, rather than having to author it themeselves per
resource as currently done as a workaround in various validation
functions (e.g. `pkg/apis/<group>/validation/validation.go`).

* Validation - ensuring only one member field is set (or at most one if
  desired).
* Normalization - ensuring the API server can understand the intent of clients
  that are unable to update/modify fields the clients are unaware of due to
  version skew.

### Non-Goals

Migrating all existing unions away from their bespoke validation
logic (e.g validation functions), is an explicit non-goal and will be pursued in
a separate KEP or later release.

## Proposal

### User Stories (Optional)

<!--
Detail the things that people will be able to do if this KEP is implemented.
Include as much detail as possible so that people can understand the "how" of
the system. The goal here is to make this feel real for users without getting
bogged down.

-->

#### Story 1

As a CRD owner, I can use simple semantics (such as openapi tags/go markers), to express the
desired validation of a oneOf (at most one or exactly one field may be set), and
the API server will perform this validation automatically.

#### Story 2

As a client, I can read, modify, and update the union fields of an object, even
if I am not aware of all of the possible fields, and the server will properly
interpret my intent.

### Notes/Constraints/Caveats (Optional)

<!--
What are the caveats to the proposal?
What are some important details that didn't come across above?
Go in to as much detail as necessary here.
This might be a good place to talk about core concepts and how they relate.
-->

### Risks and Mitigations
- We need to ensure we do not break existing union types. This can be done by
  not forcing existing unions to conform to the newly proposed union semantics.
  Integration testing with older types should give us the confidence to be sure
  we have done so.
- There is a lot of risk for errors when there exists skew between clients and
  server. In the section on normalization, we discuss mitigating these risks.

<!--
What are the risks of this proposal, and how do we mitigate? Think broadly.
For example, consider both security and how this will impact the larger
Kubernetes ecosystem.

How will security be reviewed, and by whom?

How will UX be reviewed, and by whom?

Consider including folks who also work outside the SIG or subproject.
-->

## Design Details

### Discriminator Field

We propose that all new unions maintain a "discriminator". This is a field that
points to which of the other "union member" fields is to be respected as the
truly desired union field in the case that there are any conflicts.

In order to demonstrate the need for the discriminator, we developed an
extensive [test matrix](https://docs.google.com/spreadsheets/d/1dIOOqgrgvMI9b2412eVuRSydEaOxhYOoqk6bUjZOY9c/edit?resourcekey=0-wlOfJTC_EX-qpU680STHMA#gid=3601413) that looks at various configurations of the performing
REST operations on a union where the client or server is unaware of a newly
added field to the union (due to version skew).

We present a [guide doc](https://docs.google.com/document/d/1Wruosjo0ELLl1yxauzpsUjgH2fK9KdgXDmOdJ5sG7Kg/edit?resourcekey=0-8Pwzx6EvsFR7VQoXzCTY4Q) on how to interpret the test matrix, but the major
conclusions are as follows (along with the test case number from the test matrix):

* (Case #22 and #27) If an unstructured client (i.e. a client that represents data as raw json maps with no knowledge of the schema)
  is unaware of field on the union, but wants to clear
  the union entirely (assuming the union is optional), it will have no way of doing
  so without a discriminator. With a discriminator, the client can express its
  intention by setting the discriminator to the empty value and the server can
  respect its intentions and clear any fields the client is unaware of.
* (Case #12 and #16) If a structured client is unaware of a field in the union that is set and it
  just wants to echo back the union it received in a get request (such as when
  updating other parts of the object), a client without a discriminator will
  silently drop the currently set field, while a client with the discriminator
  will not change the discriminator value, indicating to the server that no
  changes are desired in the union.
* (Case #34 and #39) If a client sets a union field that the server is not aware of, the server
  will silently drop it and attempt to clear the object of the union field. With
  a discriminator, the server will see the unrecognized discriminator value and
  can fail loudly.
* (Case #23 and #28) When a client goes to set a field it knows of, but a separate field it doesn't
  know about is currently set, the server can simply know to always respect the
  discriminator. Without a discriminator, the server will have to do convoluted
  logic to detect that the previously set field has not been modified and that
  only one of the other union fields has been.

### Go Markers

We're proposing a new type of tags for go types (in-tree types, and also
kubebuilder types):

- `// +unionDiscriminator` before a field means that this field is the
  discriminator for the union. This field MUST be an enum defined as a string (see section on
  discriminator values). This field MUST be required if there is no default
  option, omitempty if the default option is the empty string, or optional and
  omitempty if a default value is specified with the `// +default` marker.
- `// +unionMember[=<memberName>][,optional] before a field means that this
  field is a member of a union. The `<memberName>` is the name of the field that will be set as the discriminator value.
  It MUST correspond to one of the valid enum values of the discriminator's enum
  type. It defaults to the go (i.e `CamelCase`) representation of the field name if not specified.
  `<memberName>` should only be set if authors want to customize how the fields
  are represented in the discriminator field. `<memberName>` should match the
  serialized JSON name of the field case-insensitively. The comma separated
  optional value determines whether or not the member field must be set when the
  discriminator selects it. Meaning, when `optional` is present on a field, the
  discriminator can select the field even if the field is not set. If optional
  is not present in the `unionMember` tag, then the object will fail validation
  if the discriminator selects the field but it is nil. A field can be marked as
  optional without specifying memberName via `// +unionMember,optional`.

#### Discriminator Values

Here we present a description of how discriminators and their valid values
should be defined.

As described above, the discriminator field must be a string and required.
Because, there are only a few specific values that the discriminator can be, we
propose that all discriminators should be defined as an enum, and should be
tagged so via the enum go marker `// +enum`.

If no option is a valid option for a union, such an option must be defined as a
member of the discriminator values enum. By convention, this "no member"
discriminator should be the empty string, but there is nothing stopping API
authors from defining their own "no option" discriminator value.

##### Empty Union Members

In some cases there are more discriminator values than there are member fields
defined in the struct when that specific member requires no configuration. An
example is the `DeploymentStrategy` where it has one member field `rollingUpdate`,
but two valid discriminator values `RollingUpdate` and `Recreate`. By using an
enum as the discriminator value we are able to define values beyond the member
fields in order to accommodate this pattern.

#### Examples

Below is an example of how to define a union based on the above design

```
// +enum
type UnionType string

const (
  FieldA UnionType = "FieldA"
  FieldB UnionType = "FieldB"
  FieldC UnionType = "FieldC"
  FieldD UnionType = "FieldD"
  FieldNone UnionType = ""
)

type Union struct {
  // +unionDiscriminator
  // +required
  UnionType UnionType

  // +unionMember
  // +optional
  FieldA int
  // +unionMember
  // +optional
  FieldB int
}
```

Note unions can't span across multiple go structures (all the fields that are part of a union has to be together in the
same structure), examples of what is allowed:

```
// This will have one embedded union.
type TopLevelUnion struct {
  Name string `json:"name"`

  Union `json:",inline"`
}

// +enum
type UnionType string

const (
  FieldA UnionType = "FieldA"
  FieldB UnionType = "FieldB"
  FieldC UnionType = "FieldC"
  FieldD UnionType = "FieldD"
  FieldNone UnionType = ""
)

// This will generate one union, with two fields and a discriminator.
type Union struct {
  // +unionDiscriminator
  // +required
  UnionType UnionType `json:"unionType"`

  // +unionMember
  // +optional
  FieldA int `json:"fieldA"`
  // +unionMember,optional
  // +optional
  FieldB int `json:"fieldB"`
}

// +enum
type Union2Type string

const (
  Alpha Union2Type = "ALPHA"
  Beta = "BETA"
)

// This generates a union where the unionMember markers demonstrate how to
customize the names used for each field in the discriminator.
type Union2 struct {
  // +unionDiscriminator
  // +required
  Type2 Union2Type `json:"type"`
  // +unionMember=ALPHA
  // +optional
  Alpha int `json:"alpha"`
  // +unionMember=BETA,optional
  // +optional
  Beta int `json:"beta"`
}

```

### OpenAPI

OpenAPI v3 already allows a "oneOf" form, which is accepted by CRD validation
(and will continue to be accepted in the future). That oneOf form will be used
for validation, but is "on-top" of this proposal.

A new extension is created in the openapi to describe the behavior:
`x-kubernetes-unions`.

This is a list of unions that are part of this structure/object. Each item in
the list represents a discriminator for the union, and for each member field,
the discriminator value of that field and whether or not that field is optional.

Conversion between OpenAPI v2 and OpenAPI v3 will preserve these fields.

The following is an example of what the generated OpenAPI definition will look
like for a given go type.

```
const (
  FieldA Union1Type = "FieldA"
  FieldB Union1Type = "FieldB"
  FieldC Union1Type = "FieldC"
  FieldD Union1Type = "FieldD"
  FieldNone Union1Type = ""
)

// This will generate one union, with two fields and a discriminator.
type Union struct {
  // +unionDiscriminator
  // +required
  Union1 Union1Type `json:"union1"`

  // +unionMember
  // +unionDiscriminatedBy=Union1
  // +optional
  FieldA int `json:"fieldA"`
  // +unionMember,optional
  // +unionDiscriminatedBy=Union1
  // +optional
  FieldB int `json:"fieldB"`

}
```

The OpenAPI x-kubernetes-unions extension will then be attached to the discriminator's property and
deserialized into go structs as follows:
```
// XKubernetesUnions is the top level extension
type XKubernetesUnions struct {
  // FieldMembers are the mapping of all valid discriminator values in a
  // union to the corresponding member field.
  // Discriminator value is the value to which the discriminator is set to
  // in order to indicate that a given member field is the currently set member
  // of the union.
  // MemberField may be nil in the case of empty union members where a valid
  // discriminator value has no corresponding member field.
  FieldMembers map[DiscriminatorValue]*MemberField `json:"fieldMembers"`
}

// DiscriminatorValue is the value that the discriminator is set to
// in order to indicate the selection of a union member.
type DiscriminatorValue string

// MemberField
type MemberField struct {
  // Name is the name of the field corresponding to the member.
  // It will be the json representation of the field marked with the
  // `// +unionMember` marker in the go type.
  Name string `json:"name"`
  // Optional determines whether the discriminator _may_ select this member
  // even when the member field is empty. Optional defaults to false.
  Optional bool `json:"optional"`
}
```

### Normalization and Validation

#### Normalization

Normalization refers to the process by which the API server attempts to understand
and correct clients which may provide the server with conflicting or incomplete
information about a union in update or patch requests.

Issues primarily arise here because of version skew between a client and a server,
such as when a client is unaware of new fields added to a union and thus doesn't
know how to clear these new fields when trying to set a different field.

For unions that follow this design, normalization is simple: the server should always respect the
discriminator.

This means that when the server receives an update request with a discriminator set to a
given field, and multiple member fields are set it should clear all fields
except the one pointed to by the discriminator _if and only if_ the
discriminator has been modified. Having multiple fields set, and a discriminator
not modified is invalid and caught later by the validation step (see below).

For both custom resources and built-in types, we expect union normalization to be
called by the request handlers shortly after mutating admission occurs.

#### Validation

Objects must be validated AFTER the normalization process.

Some validation situations specific to unions are:
1. When multiple union fields are set and the discriminator has not been modified we should
   error loudly that the client must change the discriminator if it changes any
   union member fields.
2. When the server receives a request with a discriminator set to a given
   field, but that given field is empty and not marked as optional, the server should fail with a clear
   error message. Note this does not apply to discriminator values that do not
   correspond to any field (as in the "empty union members case").

For both custom resources and built-in types, validation will occur as part of
the request validation, before validating admission occurs.

For custom resources, union validation will be done at the same point as the
existing structural schema validation that occurs in the custom resource handler.
This ensures that any generic validation changes made to all custom resources (such
as the ratcheting validation discussed below), behaves appropriately with union
validation.

#### Ratcheting Validation

When updating CRDs to support union validation, it is possible that existing CRs
become invalid.

The naive solution is to require existing CRs to be updated to a valid state
before they can be updated again.

This creates many potential landmines, and so ratcheting validation is proposed
as an alternative. Ratcheting validation means that objects will ignore stricter
validation rules if and only if the existing object also fails the stricter
validation for the same reason.

Ratcheting validation for custom resources is a [separate
effort](https://github.com/kubernetes/kubernetes/issues/94060) proposed
outside of this unions effort. For the initial alpha graduation of unions, we
do not propose supporting ratcheting validation. We will require all invalid CRs
to be made valid before they can be updated (the naive solution).

In order to potentially support ratcheting validation in the future, we will
ensure that all callers of union validation retain access to both old and new
objects, so that future ratcheting validation can be implemented within the
union validation library.

#### Migrating existing unions

As mentioned, one of the goals is to migrate at least one existing union to
using the new marker based union validation and normalization. While open
questions remain around the priority and urgency of migrating existing unions,
nonetheless we should be able to come to a consensus on which types to migrate
first.

For discriminated unions, a couple relatively straightforward discriminated types are
`MetricSpec` and `MetricStatus`. These have clearly defined discriminator values
that map one-to-one to a member field, which make them good candidates for
initial migration.

For non-discriminated unions, there are a few relatively straightforward types
that make good candidates for initial migration, such as `ContainerState`

Until migrated, union types without a discriminator (i.e. only existing unions that have not been migrated
to the current desgin), cannot be tagged with the go markers described above and
thus will not be treated as "unions" in the sense of this currently proposed
normalization and validation logic.

These legacy unions must continue to perform normalization and
validation manually, per resource in the validation functions.

### Test Plan

[x] I/we understand the owners of the involved components may require updates to
existing tests to make this code solid enough prior to committing the changes necessary
to implement this enhancement.

##### Prerequisite testing updates

<!--
Based on reviewers feedback describe what additional tests need to be added prior
implementing this enhancement to ensure the enhancements have also solid foundations.
-->

##### Unit tests

<!--
In principle every added code should have complete unit test coverage, so providing
the exact set of tests will not bring additional value.
However, if complete unit test coverage is not possible, explain the reason of it
together with explanation why this is acceptable.
-->

<!--
Additionally, for Alpha try to enumerate the core package you will be touching
to implement this enhancement and provide the current unit coverage for those
in the form of:
- <package>: <date> - <current test coverage>
The data can be easily read from:
https://testgrid.k8s.io/sig-testing-canaries#ci-kubernetes-coverage-unit

This can inform certain test coverage improvements that we want to do before
extending the production code to implement this enhancement.

- `<package>`: `<date>` - `<test coverage>`

-->
Core functionality will be extensively unit tested in the SMD typed package
(union_test.go).

Parts of the kubernetes endpoints handlers package that are modified to call
into the SMD code will also be unit tested as appropriate.

##### Integration tests

We will have extensive integration testing of the union code in the
`test/integration/apiserver` package.

We will be testing along the dimensions of:
* Which fields of the union get modified (none, existing fields, newly updated
  fields)
* Type of union (discriminated vs non-discriminated)
* Whether the client is aware of all the fields
* Whether the server is aware of all fields
* Whether the union is optional or required

A fully documented test matrix exists in a [google
spreadsheet](https://docs.google.com/spreadsheets/d/1dIOOqgrgvMI9b2412eVuRSydEaOxhYOoqk6bUjZOY9c/edit?resourcekey=0-wlOfJTC_EX-qpU680STHMA#gid=3601413) along with a
[guide
doc](https://docs.google.com/document/d/1Wruosjo0ELLl1yxauzpsUjgH2fK9KdgXDmOdJ5sG7Kg/edit?resourcekey=0-8Pwzx6EvsFR7VQoXzCTY4Q) on how to read and understand the test matrix.

As part of implementing the test matrix we will be able to prove the viability
of upgrading existing unions by writing tests to mimic using the standardized
union semantics on existing unions (even if actually upgrading these unions is
outside the scope of alpha graduation)

<!--
This question should be filled when targeting a release.
For Alpha, describe what tests will be added to ensure proper quality of the enhancement.

For Beta and GA, add links to added tests together with links to k8s-triage for those tests:
https://storage.googleapis.com/k8s-triage/index.html

- <test>: <link to test coverage>
-->

##### e2e tests

<!--
This question should be filled when targeting a release.
For Alpha, describe what tests will be added to ensure proper quality of the enhancement.

For Beta and GA, add links to added tests together with links to k8s-triage for those tests:
https://storage.googleapis.com/k8s-triage/index.html

We expect no non-infra related flakes in the last month as a GA graduation criteria.

- <test>: <link to test coverage>
-->
 We are considering adding kubectl e2e tests to mimic kubectl users performing
 various operations on objects with union fields.

### Graduation Criteria

#### Alpha Graduation

- CRDs can be created with union fields and are properly validated when
  created/updated.
- Prove the viability of upgrading existing unions to the new semantics by
  mimicking existing unions in e2e tests.
- Existing unions that don't have discriminators do not break when upgraded.

### Upgrade / Downgrade Strategy

<!--
If applicable, how will the component be upgraded and downgraded? Make sure
this is in the test plan.

Consider the following in developing an upgrade/downgrade strategy for this
enhancement:
- What changes (in invocations, configurations, API use, etc.) is an existing
  cluster required to make on upgrade, in order to maintain previous behavior?
- What changes (in invocations, configurations, API use, etc.) is an existing
  cluster required to make on upgrade, in order to make use of the enhancement?
-->

Turning the flag on for alpha just enables different runtime codepaths (i.e.
performing the unified union validation and normalization)

Any schema markers (added by CRD authors or propagated from tags on built-in
types) will appear in the schema, but not do anything if the flag is off.

### Version Skew Strategy

<!--
If applicable, how will the component handle version skew with other
components? What are the guarantees? Make sure this is in the test plan.

Consider the following in developing a version skew strategy for this
enhancement:
- Does this enhancement involve coordinating behavior in the control plane and
  in the kubelet? How does an n-2 kubelet without this feature available behave
  when this feature is used?
- Will any other components on the node change? For example, changes to CSI,
  CRI or CNI may require updating that component before the kubelet.
-->

See test matrix and commentary about discriminators. It clearly documents how
the server will use the discriminator to understand the client's intention even
if the client is not aware of all union fields because of version skew.

Skew with alpha flag on/off shouldn't make much of a difference.
* Objects created with the union semantics, but applied to a cluster with the
  alpha flag off will simply not perform union validation and normalization.
* Objects created without union semantics will simply not trigger union
  validation and normalization (regardless of whether the server has the alpha
  enabled or disabled).

## Production Readiness Review Questionnaire

<!--

Production readiness reviews are intended to ensure that features merging into
Kubernetes are observable, scalable and supportable; can be safely operated in
production environments, and can be disabled or rolled back in the event they
cause increased failures in production. See more in the PRR KEP at
https://git.k8s.io/enhancements/keps/sig-architecture/1194-prod-readiness.

The production readiness review questionnaire must be completed and approved
for the KEP to move to `implementable` status and be included in the release.

In some cases, the questions below should also have answers in `kep.yaml`. This
is to enable automation to verify the presence of the review, and to reduce review
burden and latency.

The KEP must have a approver from the
[`prod-readiness-approvers`](http://git.k8s.io/enhancements/OWNERS_ALIASES)
team. Please reach out on the
[#prod-readiness](https://kubernetes.slack.com/archives/CPNHUMN74) channel if
you need any help or guidance.
-->

### Feature Enablement and Rollback

<!--
This section must be completed when targeting alpha to a release.
-->

###### How can this feature be enabled / disabled in a live cluster?

<!--
Pick one of these and delete the rest.

Documentation is available on [feature gate lifecycle] and expectations, as
well as the [existing list] of feature gates.

[feature gate lifecycle]: https://git.k8s.io/community/contributors/devel/sig-architecture/feature-gates.md
[existing list]: https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/
-->

- [x] Feature gate (also fill in values in `kep.yaml`)
  - Feature gate name: APIUnions
  - Components depending on the feature gate: kube-apiserver

Request handlers in the api server will call into union validation and
normalization function from the structured-merge-diff repo when feature is
enabled.

###### Does enabling the feature change any default behavior?

Enabling the feature could cause existing CRs to fail validation if the
correspond CRD has union fields and the existing CRs have invalid unions that
were unvalidated when initially created in a cluster that had the unions feature
disabled.

These CRs will need to be corrected in order to pass validation (or the feature
disabled).

<!--
Any change of default behavior may be surprising to users or break existing
automations, so be extremely careful here.
-->

###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?

<!--
Describe the consequences on existing workloads (e.g., if this is a runtime
feature, can it break the existing applications?).

Feature gates are typically disabled by setting the flag to `false` and
restarting the component. No other changes should be necessary to disable the
feature.

NOTE: Also set `disable-supported` to `true` or `false` in `kep.yaml`.
-->

Yes, requests will simply skip union validation and normalization.

###### What happens if we reenable the feature if it was previously rolled back?

Custom resources that were skipping union validation when when the feature was
rolled back may have allowed invalid data to persist.

For alpha, we require that all modifying requests (update/patch) fail unless the
data passes union validation. Retrieving newly invalid CRs should still always
succeed.

In the future, we may require looser "ratcheting validation" which would allow
modifications to ignore union validation if the existing object fails the union
validation for the same reason as the new object (see section on "Ratcheting
Validation" above). This is not a priority for alpha.

###### Are there any tests for feature enablement/disablement?

<!--
The e2e framework does not currently support enabling or disabling feature
gates. However, unit tests in each component dealing with managing data, created
with and without the feature, are necessary. At the very least, think about
conversion tests if API types are being modified.

Additionally, for features that are introducing a new API field, unit tests that
are exercising the `switch` of feature gate itself (what happens if I disable a
feature gate after having objects written with the new field) are also critical.
You can take a look at one potential example of such test in:
https://github.com/kubernetes/kubernetes/pull/97058/files#diff-7826f7adbc1996a05ab52e3f5f02429e94b68ce6bce0dc534d1be636154fded3R246-R282
-->

We will have integration tests demonstrating how CRs with persisted invalid data
will need to be corrected when the feature is re-enabled (and requires more
strict union validation).

### Rollout, Upgrade and Rollback Planning

<!--
This section must be completed when targeting beta to a release.
-->

###### How can a rollout or rollback fail? Can it impact already running workloads?

N/A

<!--
Try to be as paranoid as possible - e.g., what if some components will restart
mid-rollout?

Be sure to consider highly-available clusters, where, for example,
feature flags will be enabled on some API servers and not others during the
rollout. Similarly, consider large clusters and how enablement/disablement
will rollout across nodes.
-->

###### What specific metrics should inform a rollback?

* `apiserver_request_total` could be watched to see if the number of create and
  update requests that are failing increase substantially.

<!--
What signals should users be paying attention to when the feature is young
that might indicate a serious problem?
-->

###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?

N/A
<!--
Describe manual testing that was done and the outcomes.
Longer term, we may want to require automated upgrade/rollback tests, but we
are missing a bunch of machinery and tooling and can't do that now.
-->

###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?

N/A
<!--
Even if applying deprecation policies, they may still surprise some users.
-->

### Monitoring Requirements

<!--
This section must be completed when targeting beta to a release.

For GA, this section is required: approvers should be able to confirm the
previous answers based on experience in the field.
-->

###### How can an operator determine if the feature is in use by workloads?

For builtins in alpha, it won't be possible to break clients since turning on vs
off will validate the same thing via different code paths.

For CRDs, you can see if they have the new union markers. If the CRD has no
other validation mechanism, turning off the flag may result in CRs accepting
invalid input.

<!--
Ideally, this should be a metric. Operations against the Kubernetes API (e.g.,
checking if there are objects with field X set) may be a last resort. Avoid
logs or events for this purpose.
-->

###### How can someone using this feature know that it is working for their instance?

1. Create a new CRD with a union field (and no other validation mechanism)
2. Apply the CRD
3. Create a CR with an invalid union (multiple fields set, no discriminator
   set), see if the CR is rejected via union validation

When we write the e2e test, a standard union CRD and test CR will be obtainable
for users to test on their instance.
<!--
For instance, if this is a pod-related feature, it should be possible to determine if the feature is functioning properly
for each individual pod.
Pick one more of these and delete the rest.
Please describe all items visible to end users below with sufficient detail so that they can verify correct enablement
and operation of this feature.
Recall that end users cannot usually observe component logs or access metrics.

- [ ] Events
  - Event Reason:
- [ ] API .status
  - Condition name:
  - Other field:
- [ ] Other (treat as last resort)
  - Details:
-->

###### What are the reasonable SLOs (Service Level Objectives) for the enhancement?

N/A

<!--
This is your opportunity to define what "normal" quality of service looks like
for a feature.

It's impossible to provide comprehensive guidance, but at the very
high level (needs more precise definitions) those may be things like:
  - per-day percentage of API calls finishing with 5XX errors <= 1%
  - 99% percentile over day of absolute value from (job creation time minus expected
    job creation time) for cron job <= 10%
  - 99.9% of /health requests per day finish with 200 code

These goals will help you determine what you need to measure (SLIs) in the next
question.
-->

###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?

N/A
<!--
Pick one more of these and delete the rest.

- [ ] Metrics
  - Metric name:
  - [Optional] Aggregation method:
  - Components exposing the metric:
- [ ] Other (treat as last resort)
  - Details:
-->

###### Are there any missing metrics that would be useful to have to improve observability of this feature?

N/A
<!--
Describe the metrics themselves and the reasons why they weren't added (e.g., cost,
implementation difficulties, etc.).
-->

### Dependencies

<!--
This section must be completed when targeting beta to a release.
-->

###### Does this feature depend on any specific services running in the cluster?

N/A
<!--
Think about both cluster-level services (e.g. metrics-server) as well
as node-level agents (e.g. specific version of CRI). Focus on external or
optional services that are needed. For example, if this feature depends on
a cloud provider API, or upon an external software-defined storage or network
control plane.

For each of these, fill in the following—thinking about running existing user workloads
and creating new ones, as well as about cluster-level services (e.g. DNS):
  - [Dependency name]
    - Usage description:
      - Impact of its outage on the feature:
      - Impact of its degraded performance or high-error rates on the feature:
-->

### Scalability

N/A
<!--
For alpha, this section is encouraged: reviewers should consider these questions
and attempt to answer them.

For beta, this section is required: reviewers must answer these questions.

For GA, this section is required: approvers should be able to confirm the
previous answers based on experience in the field.
-->

###### Will enabling / using this feature result in any new API calls?

No

<!--
Describe them, providing:
  - API call type (e.g. PATCH pods)
  - estimated throughput
  - originating component(s) (e.g. Kubelet, Feature-X-controller)
Focusing mostly on:
  - components listing and/or watching resources they didn't before
  - API calls that may be triggered by changes of some Kubernetes resources
    (e.g. update of object X triggers new updates of object Y)
  - periodic API calls to reconcile state (e.g. periodic fetching state,
    heartbeats, leader election, etc.)
-->

###### Will enabling / using this feature result in introducing new API types?

No

<!--
Describe them, providing:
  - API type
  - Supported number of objects per cluster
  - Supported number of objects per namespace (for namespace-scoped objects)
-->

###### Will enabling / using this feature result in any new calls to the cloud provider?

No
<!--
Describe them, providing:
  - Which API(s):
  - Estimated increase:
-->

###### Will enabling / using this feature result in increasing size or count of the existing API objects?

No
<!--
Describe them, providing:
  - API type(s):
  - Estimated increase in size: (e.g., new annotation of size 32B)
  - Estimated amount of new objects: (e.g., new Object X for every existing Pod)
-->

###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?

In GA (maybe beta), we might expect resource reduction/reliability improvement,
since this removes a need for webhooks.

<!--
Look at the [existing SLIs/SLOs].

Think about adding additional work or introducing new steps in between
(e.g. need to do X to start a container), etc. Please describe the details.

[existing SLIs/SLOs]: https://git.k8s.io/community/sig-scalability/slos/slos.md#kubernetes-slisslos
-->

###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components?

New validation and normalization logic should be negligible given that the
functions will be in the same SMD path currently used by SSA code.

We will have benchmarking to validate this assumption.

<!--
Things to keep in mind include: additional in-memory state, additional
non-trivial computations, excessive access to disks (including increased log
volume), significant amount of data sent and/or received over network, etc.
This through this both in small and large cases, again with respect to the
[supported limits].

[supported limits]: https://git.k8s.io/community//sig-scalability/configs-and-limits/thresholds.md
-->

### Troubleshooting

<!--
This section must be completed when targeting beta to a release.

For GA, this section is required: approvers should be able to confirm the
previous answers based on experience in the field.

The Troubleshooting section currently serves the `Playbook` role. We may consider
splitting it into a dedicated `Playbook` document (potentially with some monitoring
details). For now, we leave it here.
-->

###### How does this feature react if the API server and/or etcd is unavailable?

N/A objects are not reachable

###### What are other known failure modes?

<!--
For each of them, fill in the following information by copying the below template:
  - [Failure mode brief description]
    - Detection: How can it be detected via metrics? Stated another way:
      how can an operator troubleshoot without logging into a master or worker node?
    - Mitigations: What can be done to stop the bleeding, especially for already
      running user workloads?
    - Diagnostics: What are the useful log messages and their required logging
      levels that could help debug the issue?
      Not required until feature graduated to beta.
    - Testing: Are there any tests for failure mode? If not, describe why.
-->

###### What steps should be taken if SLOs are not being met to determine the problem?

## Implementation History
- Unions implemented, but disabled in SMD.

## Drawbacks

<!--
Why should this KEP _not_ be implemented?
-->
An issue that one might have with requiring a discriminator is that it might
seem redundant to have to set a field _and_ set another field indicating to the
server to use the set field. The reasons for doing so are discussed above in the
normalization section.

One other drawback is that our approach does not standardize all existing unions
into a single format. We don't see a way to do so without drastically changing
existing APIs and breaking backwards compatibility

## Alternatives

###### Non-Discrminated

The primary alternative discussed is to not have a discriminator for new union
types. As discussed in the normalization section, requiring a discriminator
allows the server to better understand the intentions of clients that do not
have knowledge of all the fields in a union if newer versions of the server add
new fields to the union.

###### "None" Discriminator Values

A number of strategies were discussed around how to represent the "none" value
of the discriminator (see "Discriminator Values" section above).

* One alternative was to mandate the "none" value always be the empty string.
  The advantage to this is its simplicity and not creating a situation where
  different API authors define there "none" value differently, so that anyone
  could immediately know that a discriminator set to "" (empty string), is not
  selecting any of the member fields. Also, it would allow us to not have to
  define the set of enum values for each discriminator (as we could just use the
  name of the member field). The disadvantage is that by not defining the set of
  enum values, we make it impossible to support the "empty union members" case.
* Another alternative was to make the discriminator a pointer to a string and
  its value nil. The disadvantage here is that this requires more complicated
  union validation logic (first do a nil check, then check the value) and makes
  it harder to determine client intent on patches where the discriminator is not
  set.
* A third alternative is to require all unions be defined in their own separate
  struct. This was rejected because there are many existing unions that define
  random fields that are not members in the union within the same struct as
  fields that do make up the union and we hope to be able to migrate at least
  some of the existing unions to the new semantics.

<!--
What other approaches did you consider, and why did you rule them out? These do
not need to be as detailed as the proposal, but should include enough
information to express the idea and why it was not acceptable.
-->

repository: kubernetes/enhancements

gha_language: Go

repo_name: kubernetes/enhancements


text:
# Generate Sitemap

This script will scan through all pages, chapters books and shelves via the API to generate a sitemap XML file.

**This is a very simplistic single-script-file example of using the endpoints API together**
, it is not a fully-featured & validated script.

Keep in mind, The sitemap generated will reflect content visible to the API user used when running the script.

## Requirements

You will need php (~7.2+) installed on the machine you want to run this script on.
You will also need BookStack API credentials (TOKEN_ID & TOKEN_SECRET) at the ready.

## Running

```bash
# Downloading the script
# ALTERNATIVELY: Clone the project from GitHub and run locally.
curl https://raw.githubusercontent.com/BookStackApp/api-scripts/main/php-generate-sitemap/generate-sitemap.php > generate-sitemap.php

# Setup
# ALTERNATIVELY: Open the script and edit the variables at the top.
export BS_URL=https://bookstack.example.com # Set to be your BookStack base URL
export BS_TOKEN_ID=abc123 # Set to be your API token_id
export BS_TOKEN_SECRET=123abc # Set to be your API token_secret

# Running the script
php generate-sitemap.php <output_file_name>
```

## Examples

```bash
# Create a sitemap called "sitemap.xml within the current directory
php generate-sitemap.php ./sitemap.xml
```

license: MIT

repository: wushian/api-scripts

branch_name: refs/heads/main

extension: md

license_type: permissive

repo_name: wushian/api-scripts


text:
---
title: Integrating with Git
description: Integrating with Git - Cloud Services
exl-id: 57e71b8a-4546-4d7f-825c-a1637d08e608
---
# Integrating Git with Adobe Cloud Manager {#git-integration}

Adobe Cloud Manager comes provisioned with a single git repository that is used to deploy code using Cloud Manager's CI/CD pipelines. Customers can use the Cloud Manager's git repository out of the box. Customers also have the option of integrating an on-premise or **customer-managed** git repository with Cloud Manager.

## Git Integration Overview {#git-integration-overview}

>[!VIDEO](https://video.tv.adobe.com/v/28710/)

This video series explores several use cases when it comes to integrating a customer-managed git repository with Cloud Manager, including:

* [Initial Sync](#initial-sync)
* [Basic Branching Strategy](#branching-strategy)
* [Feature Branch Development](#feature-development)
* [Production Deployment](#production-deployment)
* [Synchronizing Release Tags](#sync-tags)

The video series assumes a basic knowledge of git and source control management. See the [additional resources below](#additional-resources) for more details on git.

>[!NOTE]
>
>The steps and naming conventions outlined in this video series represent some best practices for working with a customer-managed git repository and Cloud Manager. It is expected that the conventions and workflows depicted would be adapted for individual development teams.

## Initial Sync {#initial-sync}

First steps for synchronizing a customer-managed Git repository with Cloud Manager's Git repository.

>[!VIDEO](https://video.tv.adobe.com/v/28711/?quality=12)

## Basic Branching Strategy {#branching-strategy}

Follow the video below to learn the basic branching strategies.

>[!VIDEO](https://video.tv.adobe.com/v/28712/?quality=12)

## Feature Branch Development {#feature-development}

Use a feature branch to isolate code changes in a customer-managed git repository and synchronize with Cloud Manager's git repository in order to use a non-production pipeline for code quality and validation testing.

>[!VIDEO](https://video.tv.adobe.com/v/28723/?quality=12)

## Production Deployment {#production-deployment}

Prepare code for a production release in a customer-managed git repository and synchronize with Cloud Manager's git repository in order to deploy to stage and production environments.

>[!VIDEO](https://video.tv.adobe.com/v/28724/?quality=12)

## Synchronizing Release Tags {#sync-tags}

Synchronize release tags from a Cloud Manager git repository into a customer-managed git repository in order to provide visibility as to what code has been deployed to stage and production environments.

>[!VIDEO](https://video.tv.adobe.com/v/28725/?quality=12)

## Additional Resources {#additional-resources}

* [GitHub Resources](https://try.github.io)
* [Atlassian Git Tutorials](https://www.atlassian.com/git/tutorials/what-is-version-control)
* [Git Cheat Sheet](https://education.github.com/git-cheat-sheet-education.pdf)

branch_name: refs/heads/master


text:
# File mangement system

[CC TV] --> Stream Data --> [Storage]

[Storage] - Data structures to store the stream data

       - It will be blocks of data (not complete data)

At one particular time, CC TV will stream data on modification

At initial setup,  CCTV will stream the data about the empty room. Blocks are created and stored in the in-memory.

Since room is static, there will be no streams after the initial streaming

When a person moves inside the room, pixels changes. Data will be streamed to in-memory now.

Example:

Assumptions:
  1.Block of data of 8 bytes
  2.Multiple files will be saved
  3.In-memory management (no file operations)
  4.Print data structure contents for every input

SaveData(fileName, "content", timestamp)

SaveData("FileName1", "ABCDEFGHIJKLMONPQRSTUVWX", 5)

file name: FileName1

Content: ABCDEFGHIJKLMONPQRSTUVWX

Timestamp: 5

Data stored:

{0, ABCDEFGH, 5}, {1, IJKLMNOP, 5}, {2, QRSTUVWX, 5},

SaveData("FileName1", "ABCDEFGHIJKLMONPQRSTUVWXYZ", 10)

file name: FileName1

Content: ABCDEFGHIJKLMONPQRSTUVWXYZ

Timestamp: 10

Data stored:

{0, ABCDEFGH, 5}, {1, IJKLMNOP, 5}, {2, QRSTUVWX, 5}, {3, YZ, 10}

SaveData("FileName1", "ABCDEFGHIJKLMONPQRSTUVWXXXXXYZ", 15)

file name: FileName1

Content: ABCDEFGHIJKLMONPQRSTUVWXXXXXYZ

Timestamp: 15

Data stored:

{0, ABCDEFGH, 5}, {1, IJKLMNOP, 5}, {2, QRSTUVWX, 5}, {3, YZ, 10}, {4, XXXXYZ, 15}

SaveData("FileName1", "ABCDEFGHIJKLMO", 20)

file name: FileName1

Content: ABCDEFGHIJKLMO

Timestamp: 20

Data stored:

{0, ABCDEFGH, 5}, {1, IJKLMNOP, 5}, {2, QRSTUVWX, 5}, {3, YZ, 10}, {4, XXXXYZ, 15}, {4, IJKLMO, 20}

PrintData(fileName, timestamp)

PrintData(FileName1, 10)

ABCDEFGHIJKLMONPQRSTUVWXYZ

PrintData(FileName1, 20)

ABCDEFGHIJKLMO

repository: chanukyamoorthy/filemangementsystem

repo_name: chanukyamoorthy/filemangementsystem


text:
# D2Go

D2Go is a production ready software system from FacebookResearch, which supports end-to-end model training and deployment for mobile platforms.

## What's D2Go

- It is a deep learning toolkit powered by [PyTorch](https://pytorch.org/) and [Detectron2](https://github.com/facebookresearch/detectron2).
- State-of-the-art efficient backbone networks for mobile devices.
- End-to-end model training, quantization and deployment pipeline.
- Easy export to TorchScript format for deployment.

## Installation

Install PyTorch Nightly (use CUDA 10.2 as example, see details at [PyTorch Website](https://pytorch.org/get-started/)):

```bash
conda install pytorch torchvision cudatoolkit=10.2 -c pytorch-nightly
```

Install Detectron2 (other installation options at [Detectron2](https://github.com/facebookresearch/detectron2/blob/master/INSTALL.md)):

```bash
python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'
```

Install mobile_cv:

```bash
python -m pip install 'git+https://github.com/facebookresearch/mobile-vision.git'
```

Install d2go:

```bash
git clone https://github.com/facebookresearch/d2go
cd d2go & python -m pip install .
```

## Get Started

- [Getting Started with D2Go](./demo).

- See our [model zoo](./MODEL_ZOO.md) for example configs and pretrained models.

## License

D2Go is released under the [Apache 2.0 license](LICENSE).


text:
# Introduction to Julia

Contents:

- [Introduction to Julia](#Introduction-to-Julia)
  - [GitHub](#GitHub)
  - [Work Environment](#Work-Environment)
  - [Jupyter Basics](#Jupyter-Basics)
  - [Julia Basics](#Julia-Basics)

In this lab we will:

(1) Load the lab GitHub repository locally;

(2) Set up our work environment by installing Julia and Jupyter;

(3) Cover Jupyter basics;

(4) Go over some Julia basics.

The goal is to set up a stable work environment for the rest of the quarter while getting a taste of the Julia language.

Today's material borrows quite a lot from [Quantitative Economics with Julia by QuantEcon](https://julia.quantecon.org/), so I encourage everyone to check it out for more details.

---
## GitHub

As you might have already noticed, all lab materials for this quarter live on GitHub. I've done so to allow for easy version control and replicability.

Here I will guide you through the bare minimum of the necessary GitHub functionality required to maintain up-to-date access to lab materials.

Go to the [GitHub website](https://github.com/) to create an account and register for a [student/educator discount](https://education.github.com/pack/offers) (this last part isn't necessary, but can't hurt).

After finishing the above step, download and install [GitHub Desktop](https://desktop.github.com/). Open it and log into your GitHub account locally.

In the GitHub Desktop interface, go to `File > Clone Repository`:

![](images/github_clone_repo.png)

The following box should pop up, in which we need to navigate to the `URL` tab:

![](images/github_clone_url.png)

Paste `https://github.com/gionikola/spring2021_core_macro_lab.git` into the URL, and then choose a folder in which you would like to keep the lab materials.

Once this is done, make sure that the `Current repository` is set to `spring2021_core_macro_lab`:

![](images/github_current_repo.png)

Now we can automatically keep all lab documents up-to-date by periodically opening up GitHub Desktop and going to `Repository > Pull`:

![](images/github_pull.png)

---
## Work Environment

To set up the necessary work environment for these labs, we need to do the following three things:

- Download and install Julia;
- Download and install Jupyter;
- "Connect" Jupyter with Julia.

We will first need to download and install the current stable version of Julia by visiting [this page](https://julialang.org/downloads/) and following the instructions for the appropriate operating system.

Then we will need to visit [this page](https://www.anaconda.com/products/individual) to download and install Anaconda, which essentially a package management tool that includes Jupyter. To learn more about Anaconda, read the [Wikipedia page](https://en.wikipedia.org/wiki/Anaconda_(Python_distribution)).

Once both Julia and Anaconda are installed, we can perform the final step of linking Jupyter with Julia. Open Julia -- we should see the following window pop up:

![](images/repl.png)

This is the Julia REPL (Read-Evaluate-Print Loop).

Now type `]` to enter package mode, then type `add IJulia`.

![](images/repl_package_mode.png)

We then click `Enter` on the keyboard to run the command. This will install the IJulia kernel, which should automatically link Julia with Jupyter. After the installation is complete, we click `Backspace` on the keyboard to leave package mode.

We can now type `using IJulia; notebook()` into the REPL to launch Jupyter Notebook. If this running this command leads to an error or the prompt `install Jupyter via Conda, y/n? [y]:`, then we can alternatively launch Jupyter Notebook by finding where it lives on our machine and/or typing `jupyter notebook` into the Anaconda Prompt. In any case, we should see something similar to the following tab open in our web browser (but in a different directory):

![](images/jupyter_repository.png)

We can now manually navigate to our preferred directory (folder) and create new notebooks, along with other types of documents. When we open a document, it will open as a new tab in our web browser.

At this point we are all set-up and ready to code!

---
## Jupyter Basics

The keyboard shortcuts window says it best: "The Jupyter Notebook has two different keyboard input modes. Edit mode allows you to type code or text into a cell and is indicated by a green cell border. Command mode binds the keyboard to notebook level commands and is indicated by a grey cell border with a blue left margin."

At any given point in time in the Jupyter interface we are either in edit mode or command mode. We can tell which mode we are in using two indicators: (1) the color of the outline around a selected cell, and (2) the presence/absence of a pencil symbol next to the active kernel name at the upper-right corner of the notebook.

If we are in edit mode, then we should see the following:

![](images/jupyter_edit_mode.png)

Otherwise we should see the following:

![](images/jupyter_cmd_mode.png)

The following is a list of some very useful shortcuts:

- `Esc` triggers command mode if currently in edit mode;
- `Enter` triggers edit mode if currently in command mode;
- `Ctrl-Enter` runs selected cell;
- `Shift-Enter` runs selected cell and selects the next cell below;
- `Alt-Enter` runs selected cell and creates a new cell below;
- `D, D` deletes selected cell.

Check out more shortcuts by going into command mode and clicking the `H` key.

There are three types of cells (1) code, (2) Markdown, and (3) raw. We will mostly be interested in code and Markdown cells, so we disregard the final category. We can tell what type of cell we are dealing with by either (1) looking up at our toolbar

![](images/jupyter_toolbar.png)

or by (2) looking to the left of the cell to check whether there is a `In [ ]:` present -- if so, then we have ourselves a code cell, otherwise it is a Markdown cell.

A couple more useful shortcuts:

- `M` changes a selected cell type to Markdown if in command mode;
- `Y` changes a selected cell type to code if in command mode.

Jupyter is simple and intuitive enough that the above should be enough to get us going.

---
## Julia Basics

### Packages

Let us start by loading Julia packages.

```julia
using LinearAlgebra # Load LinearAlgebra package
using Statistics # Load Statistics package
```

Suppose we do not have the `LinearAlgebra` package available in our environment.

In this case, loading LinearAlgebra with `using` will not work, since we first need to install the package using the following code:

```julia
using Pkg # Load `Pkg`, which can be used to install packages
Pkg.add("LinearAlgebra") # Install `LinearAlgebra` package
using LinearAlgebra
```

      Resolving package versions...
    No Changes to `C:\Users\gioni\Project.toml`
    No Changes to `C:\Users\gioni\Manifest.toml`

We could have alternatively opened the REPL, entered package mode by clicking `]`, and ran `add LinearAlgebra`.

![](images/repl_linearalgebra.png)

### Simple Data Types

Now that basic package management is out of the way, we can start talking about data types.

Suppose we have randomly-generated variables $x$ and $y$, and would like to check whether $x > y$. Let's call this condition A. We can store the status of condition A as a Boolean variable $z$.

```julia
x = randn() # Generate random value `x`
y = randn() # Generate random value `y`
z = x > y   # Store status of condition A as variable `z`
message = "x = $(x) and y = $(y), " * "therefore condition A is $(z)."
message
```

    "x = -1.1073262538888373 and y = -1.7756952833062138, therefore condition A is true."

Notice that we used the `randn()` function to generate a random $x,y \in \mathbb{R}$.
Furthermore, we generated a string called `message` by concatenating two separate strings using `*`.
We print the values of variable `x`, `y`, and `z` in `message` using `$`.

Let's confirm that `message` is indeed a string:

```julia
typeof(message)
```

    String

Let's also check the data type of `x` (and equivalently `y`):

```julia
typeof(x)
```

    Array{Int64,1}

Now let's check the data type of `z`:

```julia
typeof(z)
```

    Bool

So far we've seen strings, floats, and booleans, but we also have integers:

```julia
y = 5
typeof(y)
```

    Int64

Since we have a float `x` and an integer `y`, let's test out some basic arithmetic operations with them:

```julia
@show x + y
@show x - y
@show x * y
@show x / y
@show x - (-y)
@show 3x - 4y
@show x^(-1);
```

    MethodError: no method matching +(::Array{Int64,1}, ::Int64)
    For element-wise addition, use broadcasting with dot syntax: array .+ scalar
    Closest candidates are:
      +(::Any, ::Any, ::Any, ::Any...) at operators.jl:538
      +(::Missing, ::Number) at missing.jl:115
      +(::Base.CoreLogging.LogLevel, ::Integer) at logging.jl:116
      ...

    Stacktrace:

     [1] top-level scope at show.jl:641

     [2] include_string(::Function, ::Module, ::String, ::String) at .\loading.jl:1091

Notice that we used the `@show` macro the print out equations.
Furthermore, we ended the last line of the cell with a `;` to supress the redundant printing of the output of `x^(-1)`.
See what happens when you construct a similar cell without `;` at the end.

But did you know we can also apply arithmetic operations to booleans?

```julia
z1 = true # define a true boolean
z2 = false # define a false boolean
@show z1 + 0
@show z2 + 0
@show z1 + z2
@show x + z1
@show x * z1;
```

    z1 + 0 = 1
    z2 + 0 = 0
    z1 + z2 = 1

    MethodError: no method matching +(::Array{Int64,1}, ::Bool)
    For element-wise addition, use broadcasting with dot syntax: array .+ scalar
    Closest candidates are:
      +(::Any, ::Any, ::Any, ::Any...) at operators.jl:538
      +(::Missing, ::Number) at missing.jl:115
      +(::Bool, ::Bool) at bool.jl:96
      ...

    Stacktrace:

     [1] top-level scope at show.jl:641

     [2] include_string(::Function, ::Module, ::String, ::String) at .\loading.jl:1091

Let's give `x` and `y` imaginary components, and play around with the resulting imaginary numbers.

```julia
@show x = x + 3im
@show y = y + 2im
@show x + y
@show x - y
@show x * y
@show x / y
@show x - (-y)
@show 3x - 4y
@show x^(-1);
```

    MethodError: no method matching +(::Array{Int64,1}, ::Complex{Int64})
    For element-wise addition, use broadcasting with dot syntax: array .+ scalar
    Closest candidates are:
      +(::Any, ::Any, ::Any, ::Any...) at operators.jl:538
      +(::Missing, ::Number) at missing.jl:115
      +(::Bool, ::Complex) at complex.jl:293
      ...

    Stacktrace:

     [1] top-level scope at show.jl:641

     [2] include_string(::Function, ::Module, ::String, ::String) at .\loading.jl:1091

### Introduction to Arrays

We can also collect data into arrays.

```julia
a = [10, 20, 30]
```

    3-element Array{Int64,1}:
     10
     20
     30

The output here tells us that `a` is a one-dimensional array containing `Int64` data.

```julia
b = [1.0, 2.0, 3.0]
```

    3-element Array{Float64,1}:
     1.0
     2.0
     3.0

`b` seems to have the same dimensions as `a`, but it contains `Float64` data instead of integer data.

Suppose we try to include multiple data types in a single array:

```julia
c = [1, 1.0, true, "text"]
```

    4-element Array{Any,1}:
        1
        1.0
     true
         "text"

This is undesirable, but as we can see -- it works.

Let's check out the dimensions and size of array/vector `b`:

```julia
@show ndims(b) # Show dimensions of vector `b`
@show size(b); # Show size of vector `b`
```

    ndims(b) = 1
    size(b) = (3,)

The above output essentially tells us that `b` is a vector with 3 entries.

I say that `b` is a vector since a one-dimensional array is equivalent to a vector, while a two-dimensional array is equivalent to a matrix.

We confirm this in the following cell:

```julia
@show Array{Float64, 1} == Vector{Float64}
@show Array{Float64, 2} == Matrix{Float64};
```

    Array{Float64, 1} == Vector{Float64} = true
    Array{Float64, 2} == Matrix{Float64} = true

There are two different ways we can create a column vector:

```julia
col1 = [1, 2, 3]
col2 = [1; 2; 3]
col1 == col2 # Test if both are column vectors
```

    true

We can also create row vectors in the following manner:

```julia
row1 = [1 2 3]
```

    1×3 Array{Int64,2}:
     1  2  3

Let's check the dimensions and size of the row vector `row1`:

```julia
@show ndims(row1)
@show size(row1);
```

    ndims(row1) = 2
    size(row1) = (1, 3)

According to the above output, row vectors are 2-dimensional -- in other words, they are matrices.

Furthermore, unlike column vectors, row vectors are not flat. The above output shows that `row1` has one row and three columns.

Let's create a more traditional-looking matrix `A`:

```julia
A = [1 2; 3 4]
```

    2×2 Array{Int64,2}:
     1  2
     3  4

Now let's transpose `A`:

```julia
A'
```

    2×2 Adjoint{Int64,Array{Int64,2}}:
     1  3
     2  4

We can access the second column of `A` in the following manner:

```julia
A[:,2]
```

    2-element Array{Int64,1}:
     2
     4

Analogously, we may access the second row of `A` in the following manner:

```julia
A[2,:]
```

    2-element Array{Int64,1}:
     3
     4

Notice that when we accessed the second row of `A`, the output was a flat array (column vector).

To may simply transpose it to obtain it in row vector form:

```julia
A[2,:]'
```

    1×2 Adjoint{Int64,Array{Int64,1}}:
     3  4

What if we want to access the $(2,2)$ entry of `A`?

```julia
A'[2,2]
```

    4

Now let's talk briefly about array creation.

A nice way of creating zero vectors is using the `zeros()` function:

```julia
zeros(2)
```

    2-element Array{Float64,1}:
     0.0
     0.0

Similarly, we can use `zeros()` to create matrices:

```julia
zeros(2,2)
```

    2×2 Array{Float64,2}:
     0.0  0.0
     0.0  0.0

More generally, we may use the `fill()` function to create arrays with customized uniform entries:

```julia
fill(3, 2, 4)
```

    2×4 Array{Int64,2}:
     3  3  3  3
     3  3  3  3

How do we copy an array? Let's try simply using an equality:

```julia
x = fill(1, 3) # create vector `x`
y = x # bind `y` to vector `x`
y[2] = 0 # change second entry of `y` to zero
x
```

    3-element Array{Int64,1}:
     1
     0
     1

What we did in the above cell is create a vector `x`, bind a new variable `y` to `x`, and then change the second entry of `y`.

We would expect this to alter `y` and not `x`, but it turns out this is not the case!

So we can't quite "copy" arrays using a simple equality. Instead we may use the `copy()` function:

```julia
x = fill(1, 3)
y = copy(x)
y[2] = 0
x
```

    3-element Array{Int64,1}:
     1
     1
     1

Notice that `x` didn't change, as desired.

What if we don't want to copy an array exactly, but instead create an array of the same data type and size?

For this we can use the `similar()` function:

```julia
x = fill(1, 3)
y = similar(x)
y
```

    3-element Array{Int64,1}:
     230286480
     387663984
     387664016

We may also use `similar()` to change the size while keeping the same data type:

```julia
x = fill(1, 3)
y = similar(x, 4)
y
```

    4-element Array{Int64,1}:
     1
     0
     0
     0

We may also create a matrix similar to a vector:

```julia
x = fill(1, 3)
y = similar(x, 2, 2)
y
```

    2×2 Array{Int64,2}:
     230286480  399933536
     399900928  165547088

### Array Operations

Let's create a random vector and then check out some of its properties:

```julia
v = [-1, 0, 1] # create vector `v` w/ 100 draws from N(0,1) distribution

@show length(v)
@show sum(v)
@show mean(v)
@show std(v)
@show var(v)
@show maximum(v)
@show minimum(v)
@show extrema(v);
```

    length(v) = 3
    sum(v) = 0
    mean(v) = 0.0
    std(v) = 1.0
    var(v) = 1.0
    maximum(v) = 1
    minimum(v) = -1
    extrema(v) = (-1, 1)

We can sort the values in vector `v`:

```julia
w = sort(v, rev = true)
```

    3-element Array{Int64,1}:
      1
      0
     -1

The way in which we sorted vector `v` above does not modify `v` itself.

We can, however, also modify the original vector using the `sort()` function:

```julia
w = sort!(v, rev = true)
v == w
```

    true

Notice that in the above cell we tested whether vectors `w` and `v` have the same values, and found that they do! This means that the original vector `v` was successfully modified.

How to do matrix multiplication?

```julia
A = [1 2; 2 1]
B = ones(2,2)
c = [1, 3]

@show A*B
@show A*B'
@show A*c
@show c'*A;
```

    A * B = [3.0 3.0; 3.0 3.0]
    A * B' = [3.0 3.0; 3.0 3.0]
    A * c = [7, 5]
    c' * A = [7 5]

Let's solve $A \, x = c$:

```julia
x1 = A \ c # the best approach
x2 = inv(A) * c # alternative
x1 ≈ x2
```

    true

In the above cell I show two equivalent ways of solving for $x$.

I check their equivalence by testing whether `x1` is approximately equal to `x2` using the $\approx$ operator.

You may write $\approx$ in a code cell by typing "\approx" ($\LaTeX$ syntax) and hitting 'Tab' on your keyboard.

In fact, you can write many other characters this way:

```julia
α = 1 # \alpha + Tab
β = 2 # \beta + Tab
γ = α + β # \gamma + Tab
γ
```

    3

We can also easily do elementwise operations.

For example, suppose we would like add 1 to every element of a vector or a matrix.

To do so, we **broadcast** the addition (`+`) operator by also including a period (`.+`):

```julia
A = ones(2,2)
x = ones(2)
@show A .+ 1
@show x .+ 1;
```

    A .+ 1 = [2.0 2.0; 2.0 2.0]
    x .+ 1 = [2.0, 2.0]

Similarly, we can squared every entry of a vector/matrix:

```julia
@show (A .+ 1).^2
@show (x .+ 1).^2
```

    (A .+ 1) .^ 2 = [4.0 4.0; 4.0 4.0]
    (x .+ 1) .^ 2 = [4.0, 4.0]

    2-element Array{Float64,1}:
     4.0
     4.0

In fact, we can broadcast pretty much any function. Here's another example:

```julia
@show log.(A .+ 1)
@show log.(x .+ 1)
```

    log.(A .+ 1) = [0.6931471805599453 0.6931471805599453; 0.6931471805599453 0.6931471805599453]
    log.(x .+ 1) = [0.6931471805599453, 0.6931471805599453]

    2-element Array{Float64,1}:
     0.6931471805599453
     0.6931471805599453

Broadcasting is a very handy tool that you should end up using a lot to write efficient code.

Lastly, some basic linear algebra tools:

```julia
A = [1 2; 2 1]
```

    2×2 Array{Int64,2}:
     1  2
     2  1

```julia
det(A)
```

    -3.0

```julia
tr(A)
```

    2

```julia
eigvals(A)
```

    2-element Array{Float64,1}:
     -1.0
      3.0

```julia
rank(A)
```

    2

### Tuples

Tuples are essentially **immutable** data containers.

Let's create one:

```julia
tup = ("macro", 3.0, 2021)
```

    ("macro", 3.0, 2021)

Now let's try to change the second entry of `tup` to `1.0` (something that we often do with arrays):

```julia
tup[2] = 2.0
```

    MethodError: no method matching setindex!(::Tuple{String,Float64,Int64}, ::Float64, ::Int64)

    Stacktrace:

     [1] top-level scope at In[148]:1

     [2] include_string(::Function, ::Module, ::String, ::String) at .\loading.jl:1091

We obtained an error precisely because of the **immutability** of tuples -- once we define a tuple, we cannot change its entry values.

Let's try to store tuple entries as distinct variables:

```julia
class, quarter, year = tup
"Year: $year, Quarter: $quarter, Class: $class"
```

    "Year: 2021, Quarter: 3.0, Class: macro"

We may also create **named tuples** and merge them:

```julia
tup3 = (q1 = "Fall", )
tup4 = (q2 = "Winter", q3 = "Spring")
tup5 = merge(tup3, tup4)
```

    (q1 = "Fall", q2 = "Winter", q3 = "Spring")

Notice that to define a tuple with only one entry, we still need to include a comma with nothing to its right in the declaration.


text:
# ExportCommand

## Main

- export passwords

### Help

```
Usage:
  export
```

##### export

![export](http://assest.dowte.com/imgs/pass-cli/export.jpg)

repository: fwhat/password

repo_name: fwhat/password


text:
# squad-server
Squad Dedicated Server Docker Image

Build to create a containerized version of Squad Dedicated Server
https://store.steampowered.com/app/393380/Squad/

Build by hand
```
git clone https://github.com/antimodes201/squad-server.git
docker build -t antimodes201/squad-server:latest .
```

Docker Pull
```
docker pull antimodes201/squad-server
```

Docker Run with defaults
change the volume options to a directory on your node and maybe use a different name then the one in the example

```
docker run -it -p 42433-42434:42433-42434/udp -v /app/docker/sandstorm:/sandstorm \
    -e INSTANCE_NAME="T3stN3t A Fancy Sandstorm Server in a Docker Container" \
    --name sandstorm \
    antimodes201/squad-server:latest
```

Currently exposed environmental variables and their default values
- INSTANCE_NAME default
- GAME_PORT_1 7787
- GAME_PORT_2 7788
- QUERY_PORT_1 27165
- QUERY_PORT_2 27166
- RCON_PORT 21114
- ENV TZ America/New_York


text:
---
title: Reproducible computing: An initial walkthrough
layout: page
---
# make a new directory and cd into it
mkdir BBSRC-git-demo
cd BBSRC-git-demo

# create a new git repository
git init
git status
# create a file called somecode.R containing the following lines:
# this will load the data from the Lewandowsky et al. study
# (http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0075637)

df=read.table('http://data.bris.ac.uk/datasets/swyt56qr4vaj17op9cw3sag7d/LskyetalPLOSONE.csv',
              header=TRUE,sep=',')
head(df)

# run the R script using the "source" button in Rstudio

# This worked, so let's check our file into the repo
git status
git add somecode.R
git status
git commit -m"initial add"
git status

# let's run a linear regression model to see if
# performance is related to age
# - add the following code and source the file:
lm.result=lm(conspiracist_avg~age,data=df)
summary(lm.result)

# this should also complete.  Let's go ahead and check in again
git add somecode.R
git commit -m"adding lm"

# let's put this into a repository on github so that we can
# share it with others

1. log into github.com
2.  create a new repository (+ sign at top right)
- give it the same name as your directory (BBSRC-git-demo)
- just use the defaults (it should be public) and
click "create repository"
3. There will be a set of commands in the section titled
"…or push an existing repository from the command line"
- copy those and paste them into the terminal inside
the directory with your git repository - somethign like:

git remote add origin git@github.com:poldrack/BBSRC-git-demo.git
git push -u origin master

Click on the repository link at the top of the page
to go to the main repo page. you should see "somecode.R"
in the list.

# let's go ahead and set up circleci to automatically
# run a smoke test for us
# create a file called circle.yml and add the following lines:

dependencies:
  pre:
    - sudo apt-get update && sudo apt-get -y install r-base
test:
  override:
    - Rscript somecode.R

# then add to repo and commit and push to github

git add circle.yml
git commit -m"initial add"
git push origin master

# next we have to hook this up to the circleci system

1. go to circleci.com and log in using your github account
2. click on the "Projects" button (with the + sign)
Choose your github account, and then click on the
"build project" button for your repo
It will then take you to a page showing the status
of the build.  for an overview, click on the "builds"
button which will take you to a list of builds.

after a couple of minutes it should show that the build
succeeded

# look at the log to see what we've done so far
git log

# we are a bit surprised that there is no relation
# between age and performance, so let's have a look
# add the following code and source the file

plot(df$age,df$performance)

git add somecode.R
git commit -m"adding plot"

# let's say that we decided that we don't want the
# plot in the file. We can go back to the previous commit:

git revert <commit ID>

the change in the file should show up immediately in
the RStudio editor window

#it's clear from the plot that something is wrong
# there is an outlier in the age distribution
# let's first add a test to check for age outliers
# in this study, subjects were supposed to be between
# 20 and 60
# add the following code above the lm command:

stopifnot(max(df$age)<60)
stopifnot(min(df$age)>20)

# when you source the file, you should see an error.
# let's see what happens when we push this to github
git add somecode.R
git commit -m"adding assertion test"
git push origin master

# a few seconds later, you will see that the automated
# test starts on circleci
# you will see that the test fails due to the error

# let's add some code to clean up the outliers
# above the assertion tests, add:

df=subset(df,age>20&age<60)

# run the code again - this time it succeeds
# and we now see a strong effect of age

# push it back to github and check circleci again
git add somecode.R
git commit -m"adding outlier removal"
git push origin master

# add a fancy badge to your github page to show off
- go to the builds page and click the gear next to your repo
- click on "status badges" and copy the text under "embed code"
[![CircleCI](https://circleci.com/gh/poldrack/BBSRC-git-demo.svg?style=svg)](https://circleci.com/gh/poldrack/BBSRC-git-demo)
- add this into a README file on github

repository: drewwint/reproducible-workflows

repo_name: drewwint/reproducible-workflows


text:
---
title: Bug Tracking With The Help Of An Entomologist
author: Andrew Wafaa
layout: post
kind: article
created_at: 2011-03-04
permalink: /bug-tracking-with-the-help-of-an-entomologist/
dsq_thread_id:
  - 639907989
categories:
  - OpenSource
tags:
  - Android
  - Bugs
  - CrossPlatform
  - Desktop
  - HackWeek
  - openSUSE
  - Qt
  - Software
---
#

I like to think of myself as being fairly consistent, and I’ve had one request for a long time, a desktop bug tracker client. I even [asked][1]  if someone more adept in the ways of The Code could help.  Luckily [Matt Barringer][4] (from those crazy cooks in the [SUSE Studio][5] team) heard my call. He picked up his code machine gun and blasted away o/

 [3]: http://news.opensuse.org/2011/01/19/hackweek-vi/ "Novell / openSUSE HackWeek VI"
 [4]: http://en.opensuse.org/User:Mbarringer "Znggo the Entomologist"
 [5]: http://susestudio.com/ "Appliances made easy"

A few days after HackWeek VI ended he enlightened me to his new masterpiece – [Entomologist][6].  So what is it? As you may have guessed, it is a desktop bug tracker client.  It’s cutely written in Qt, isn’t final yet (is any software?) but works fairly well.  Why only fairly well? Simples, it only supports Bugzilla at the minute, but there are plans afoot to add others (please see the [ToDo][7] list). Once more bug backends are added it will be great (yes, this is a subtle hint for you to contribute code ;-) ). It is being put forward as one of the [GSOC][9] projects so fingers crossed ;-)

 [6]: http://entomologist.sourceforge.net/ "Desktop bug tracker client"
 [7]: http://entomologist.sourceforge.net/#todo "what needs to be done"
 [9]: http://en.opensuse.org/openSUSE:GSOC_2011_Ideas#Cross_Platform_Bug_Tracker_Client "Part of openSUSE's submission"

Of course you need some obligatory screenshots, so feast your eyes :-)

![Entomologist on the desktop][11]

 [11]: https://lh6.googleusercontent.com/_ObpIMjebLDY/TXC_A6xeQLI/AAAAAAAAAOY/FDjKJM3lPV8/s912/entomologist-desktop.png

Seeing your bugs on the desktop

![Seeing your bugs on the move][12]

 [12]: https://lh5.googleusercontent.com/_ObpIMjebLDY/TXC_Aq_GPaI/AAAAAAAAAOU/qEmmPWywluo/s640/entomologist-android1.png

Seeing your bugs on the move.

Yes that’s right, entomologist works on Android!! Sure the UI is a bit butt ugly on the platform, but the app works ;-)

I’ll do a post on how I ported entomologist to Android shortly, it wasn’t too difficult.

So if you want to give it a whirl, then head over to the homepage and choose your flavour of openSUSE, Fedora, Mandriva, Ubuntu or Debian (thanks [OBS][13])

 [13]: http://en.opensuse.org/Portal:Build_Service "The openSUSE Build Service"

**UPDATE:**

For those that want to minimise the number of clicks the list of repos are:

*   [openSUSE 11.4][14]
*   [openSUSE 11.3][15]
*   [openSUSE 11.2][16]
*   [openSUSE Factory][17]
*   [SLED 11 SP1][18]
*   [Fedora 14][19] (untested)
*   [Fedora 13][20] (untested)
*   [Fedora 12][21] (untested)
*   [Mandriva 2010][22] (untested)
*   [Mandriva 2010.1][23] (untested)

 [14]: http://software.opensuse.org/ymp/home:mbarringer/openSUSE_11.4/entomologist.ymp?base=openSUSE:11.4&query=Entomologist
 [15]: http://software.opensuse.org/ymp/home:mbarringer/openSUSE_11.3/entomologist.ymp?base=openSUSE:11.3&query=entomologist
 [16]: http://software.opensuse.org/ymp/home:mbarringer/openSUSE_11.2/entomologist.ymp?base=openSUSE:11.2&query=entomologist
 [17]: http://software.opensuse.org/ymp/home:mbarringer/openSUSE_Factory/entomologist.ymp?base=openSUSE:Factory&query=Entomologist
 [18]: http://software.opensuse.org/ymp/home:mbarringer/SLE_11_SP1/entomologist.ymp?base=SUSE:SLE-11:SP1&query=entomologist
 [19]: http://software.opensuse.org/ymp/home:mbarringer/Fedora_14/entomologist.ymp?base=Fedora:14&query=Entomologist
 [20]: http://software.opensuse.org/ymp/home:mbarringer/Fedora_13/entomologist.ymp?base=Fedora:13&query=Entomologist
 [21]: http://software.opensuse.org/ymp/home:mbarringer/Fedora_12/entomologist.ymp?base=Fedora:12&query=Entomologist
 [22]: http://software.opensuse.org/ymp/home:mbarringer/Mandriva_2010/entomologist.ymp?base=Mandriva:2010&query=Entomologist
 [23]: http://software.opensuse.org/ymp/home:mbarringer/Mandriva_2010.1/entomologist.ymp?base=Mandriva:2010.1&query=Entomologist

.deb packages are also provided:

*   [(K)Ubuntu 10.10][24]
*   [(K)Ubuntu 10.04][25] (untested)
*   [Debian 6.0][26] (untested)

 [24]: http://download.opensuse.org/repositories/home:/mbarringer/xUbuntu_10.10/
 [25]: http://download.opensuse.org/repositories/home:/mbarringer/xUbuntu_10.04/
 [26]: http://download.opensuse.org/repositories/home:/mbarringer/Debian_6.0/

repository: awafaa-zz/EuropeanWafaa

repo_name: awafaa-zz/EuropeanWafaa


text:
# Codebase for "TabNet: Attentive Interpretable Tabular Learning"

Authors: Sercan O. Arik, Tomas Pfister

Paper: https://arxiv.org/abs/1908.07442

This directory contains an example implementation of TabNet on the Forest Covertype dataset (https://archive.ics.uci.edu/ml/datasets/covertype).

First, run `python -m download_prepare_covertype.py` to download and prepare the Forest Covertype dataset.
This command creates train.csv, val.csv and test.csv files under the "data/" directory.

To run the pipeline for training and evaluation, simply use `python -m experiment_covertype.py`.
For debugging in a low-resource environment, you can use `python -m test_experiment_covertype.py`.

To modify the experiment to other tabular datasets:
- Substitute the train.csv, val.csv, and test.csv files under "data/" directory,
- Modify the data_helper function with the numerical and categorical features of the new dataset,
- Reoptimize the TabNet hyperparameters for the new dataset.

repository: Tarkiyah/googleResearch

gha_language: Jupyter Notebook

repo_name: Tarkiyah/googleResearch


text:
# Pano-AVQA

Official repository of PanoAVQA: Grounded Audio-Visual Question Answering in 360° Videos (ICCV 2021)

![Data_fig](https://hs-yn.github.io/assets/img/panoavqa_data.png)

### [[Paper]](https://openaccess.thecvf.com/content/ICCV2021/html/Yun_Pano-AVQA_Grounded_Audio-Visual_Question_Answering_on_360deg_Videos_ICCV_2021_paper.html) [[Poster]](https://hs-yn.github.io/assets/pdf/2021iccv_panoavqa_poster.pdf) [Video]

## Getting Started

This code is based on following libraries:

* `python=3.8`
* `pytorch=1.7.0` (with cuda 10.2)

To create virtual environment with all necessary libraries:

```bash
conda env create -f environment.yml
```

By default data should be saved under `data/feat/{audio,label,visual}` directory and logs (w/ cache, checkpoint) are saved under `data/{cache,ckpt,log}` directory. Using symbolic link is recommended:

```bash
ln -s {path_to_your_data_directory} data
```

We use single TITAN RTX for training, but GPUs with less memory are still doable with smaller batch size (provided precomputed features).

## Dataset

We plan to release the Pano-AVQA dataset public within this year, including Q&A annotation, precomputed features, etc. Please stay tuned!

## Model

### Training

Default configuration is provided in `code/config.py`. To run with this configuration:

```bash
python cli.py
```

To run with custom configuration, either modify `code/config.py` or execute:

```bash
python cli.py with {{flags_at_your_disposal}}
```

### Inference

Model weight is saved under `./data/log` directory. To run inference only:

```bash
python cli.py eval with ckpt_file=../data/log/{experiment}/{ckpt}.pth
```

## Citation

If you find our work useful in your research, please consider citing:

```tex
@InProceedings{Yun2021PanoAVQA,
    author = {Yun, Heeseung and Yu, Youngjae and Yang, Wonsuk and Lee, Kangil and Kim, Gunhee},
    title = {Pano-AVQA: Grounded Audio-Visual Question Answering on 360$^\circ$ Videos},
    booktitle = {ICCV},
    year = {2021}
}
```

## Contact

If you have any inquiries, please don't hesitate to contact us via heeseung.yun at vision.snu.ac.kr.

repository: HS-YN/PanoAVQA

repo_name: HS-YN/PanoAVQA


text:
qcache
======
[![Build Status](https://api.travis-ci.com/github/andrasq/node-qcache.svg?branch=master)](https://travis-ci.com/github/andrasq/node-qcache?branch=master)
[![Coverage Status](https://coveralls.io/repos/github/andrasq/node-qcache/badge.svg?branch=master)](https://coveralls.io/github/andrasq/node-qcache?branch=master)

Some useful caches: time-to-live, least-recently-used, multi-value.

Caches implement the Store semantics of `set`, `get`, and `delete`, but unlike
Store, caches are free to discard contents according to their retention
heuristics.

Overview
--------

    const qcache = require('qcache');
    const TtlCache = qcache.TtlCache;
    const LruCache = qcache.LruCache;
    const MultiValueCache = qcache.MultiValueCache;

    var cache = new LruCache();
    cache.set('a', 1);
    cache.set('b', 2);
    var a = cache.get('a');         // => 1
    var b = cache.get('b');         // => 2

Benchmark
---------

    $ node benchmark.js

    package versions:
      qcache 0.7.1
      node-cache 4.2.0
      memory-cache 0.2.0
      lru-cache 5.1.1

    ---------------- TTL cache:
    qtimeit=0.21.0 node=8.11.1 v8=6.2.414.50 platform=linux kernel=4.9.0-0.bpo.4-amd64 up_threshold=false
    arch=ia32 mhz=4184 cpuCount=8 cpu="Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz"
    timeGoal=1 opsPerTest=10000 forkTests=false
    name                           speed           rate
    lru-cache                  3,874,722 ops/sec   1937 >>>>>>>>>>
    node-cache                   678,685 ops/sec    339 >>
    qcache.TtlCache            8,532,265 ops/sec   4266 >>>>>>>>>>>>>>>>>>>>>

    ---------------- LRU cache:
    timeGoal=1 opsPerTest=10000 forkTests=false
    name                           speed           rate
    lru-cache 1% lru           1,101,544 ops/sec    551 >>>
    lru-cache 25% lru          1,263,517 ops/sec    632 >>>
    qcache.LruCache 1% lru     5,034,081 ops/sec   2517 >>>>>>>>>>>>>
    qcache.LruCache 25%        4,668,964 ops/sec   2334 >>>>>>>>>>>>

TtlCache
========

Key-value store with a time-to-live (ttl) timeout limit.

### new qcache.TtlCache( options )

        var TtlCache = qcache.TtlCache;
        var cache = new TtlCache();

Options:

- `ttl` - milliseconds before the stored value expires (default 1 year)
- `capacity` - the most items to cache at any time (default 10,000)
- `currentTimestamp` - function that returns a millisecond timestamp
(default is a built-in timeout timer)

Properties:

- `ttl` - the configured ttl
- `capacity` - the configured capacity
- `count` - number of items currently stored.  Read-only; do not write this value.
- `max` - same as capacity

### get( key )

return the value stored under key.  Missing keys and expired contents read as
`undefined`.  An item expires when the current timestamp exceeds the item
ttl (time-to-live) value.

### set( key, value [,ttl] ), put

store the value under key for the configured ttl milliseconds.  If the cache
has already reached its capacity, one or more of the stored values will be
discarded to make room.

If provided, the optional time-to-live `ttl` will be used instead of the cache
instance configured default.

### delete( key ), del

remove the value from the cache.  Removed items read as undefined.

### gc( )

remove expired elements from the cache to free the occupied storage space

### keys( )

return the currently tracked cache keys, including keys that may be expired or been deleted.

LruCache
========

Least-recently-used replacement policy cache with a capacity limit.

As of<EMAIL_ADDRESS>the cache was reimplemented using a fast doubly-linked list,
which is not only 2-3x faster but for large datasets can be orders of magnitude faster.
The old version is still available as `qcache.LruCache1`.

### new qcache.LruCache( options )

create a new lru cache

Options:

`capacity` - max number of elements to cache at one time.  Once the cache
capacity is reached, setting a new key causes the least recently used value to
be discarded.  Default is `Infinity`, unlimited number of entries.

Properties:

- `capacity` - the configured capacity
- `count` - the number of items in the cache

### set( key, value )

store the value under key.  This key/value becomes the most recently used.
If this value would exceed the configured cache `capacity`, the oldest (least recently
accessed) value is displaced from the cache..

### get( key )

get the value stored under key.  This key/value becomes the most recently used.
Missing keys and displaced values read as `undefined`.

### delete( key ), del

remove the key from the cache.

### keys( )

return the currently tracked cache keys, including keys that may be expired or been deleted.

MultiValueCache
===============

The multi-value cache holds multiple values indexed by the same key in
oldest-first insertion order.  There are no ttl and capacity limits
implemented, and since delete semantics are not immediately obvious, there is
no delete yet.

### new qcache.MultiValueCache( )

create a new multi-value cache

### set( key, value )
### push( key, value )

store the value under key as the newest value

### get( key )
### shift( key )

return the oldest value stored under key

### unshift( key, value )

store the value under key as the oldest value.  This value will be the one
returned by the next call to get().

### pop( key )

return the newest value stored under key

### isEmpty( key )

test whether there are any values stored under key

### getLength( key )

return the count of items stored under `key`

### keys( )

return the currently tracked cache keys, including keys that may be expired or been deleted.

Change Log
----------

- 0.8.3 - faster insert/delete by more efficient list updates, return lru keys in sorted order
- 0.8.1 - avoid delete for much faster evictions, fix list breakage when re-inserting last node
- 0.8.0 - `keys` method on TtlCache, LruCache, MultiValueCache, test with newer `qnit`
- 0.7.2 - faster LruCache
- 0.7.1 - LruCache much faster rewrite, fix LruCache1 count
- 0.6.2 - improve lrucache tests
- 0.6.1 - write LruCache unit tests, fix LruCache1 delete; 100% test coverage
- 0.6.0 - optional ttl param to TtlCache set()
- 0.5.0 - move lib/timebase into qtimebase-1.0.0
- 0.4.2 - ttlcache: adjust benchmark to simulate reuse
- 0.4.0 - ttlcache: speed up deletes, 100% unit test coverage, benchmark script
- 0.3.6 - timebase: track timestamps more closely; mvcache: getLength method; test with qnit
- 0.3.4 - make timebase track current time even during blocking burst fetches
- 0.3.3 - added ttl cache gc() method; consider an item that expires this ms as still live
- 0.3.2 - switch to an adaptive timebase for timestamps
- 0.3.0 - LruCache; MultiValueCache
- 0.2.0 - TtlCache

Todo
----

repository: andrasq/node-qcache

repo_name: andrasq/node-qcache


text:
# Add Enemies

## Intro @unplugged

Now let's add some of Wonder Woman's biggest enemies to the maze!
![Enemies!](/static/recipes/wonder-woman-1984/enemies-preview.png)

## Step 1 : Create your villain

Go to the tilemap and click on **My Tiles**. Click on the **+** and draw the enemy you want Wonder Woman to face.
You could draw her enemy from the upcoming movie: **Cheetah**.
From ``||scene||`` get the ``||scene: on sprite of kind overlaps||`` block.
Click on the grey square and find your villain!

```blocks
scene.onOverlapTile(SpriteKind.Player, myTiles.null, function (sprite, location) {

})
```

## Step 2 : Place your artifacts

Now, go back to the tilemap. Go to **My Tiles** and select your enemy. Place the enemy tile on your maze in all the places you want your enemy to appear.
![Enemies!](/static/recipes/wonder-woman-1984/enemies-preview.png)

## Step 3 : Lose points when Wonder Woman runs into enemies

Now, let's make Wonder Woman lose points when she runs into an enemy.
First, we need to create a score tracker for the game.
Get the ``||info: set score to||`` block at put it in the ``||loops: on start||`` loop.
Now when you start the game, your score will be set to 0.

Next, get the ``||info: change score by||`` block and put it in the ``||scene: on sprite of kind overlaps||`` block.
Now, when Wonder Woman overlaps with an enemy, the player will **lose one point**.

```blocks
scene.onOverlapTile(SpriteKind.Player, myTiles.null, function (sprite, location) {
    info.changeScoreBy(1)
})
```

## Step 4 : Add a sound when for the enemy

In ``||music||`` , get the ``||music: play sound||`` block. In the drop-down, choose the sound that will play when you lose a point.

```blocks
scene.onOverlapTile(SpriteKind.Player, myTiles.null, function (sprite, location) {
    info.changeScoreBy(1)
    music.baDing.play()
})
```

## Step 5 : Make enemies disappear

Once you've already seen an enemy, you want it to disappear. So, from the "tile" section of ``||scene||``, get a ``||scene: set at tilemap||`` block.
Click on the grey box and choose the tile that matches the background of your maze.
Replace the ``||variable: tilemap col row||`` with ``||variable: location||`` by dragging it from the ``||scene: on sprite of kind overlaps||`` block that's already in your workspace.

```blocks
scene.onOverlapTile(SpriteKind.Player, myTiles.null, function (sprite, location) {
    info.changeScoreBy(1)
    music.baDing.play()
    tiles.setTileAt(location, myTiles.null)
})
```

## Step 6 : Add more enemies

Repeat these steps to create more enemies to add to your maze or start playing!

repository: jacobcarpenter/pxt-arcade

repo_name: jacobcarpenter/pxt-arcade


text:
RUL
===

Version 1.1.0
This library is designed to simplify randomization and the creation of procedurally generated content. RUL is capable of generating
pseudo-random numbers, vectors, colors and noise, as well as randomly modifying existing objects.

Rul is split into four modules :
Rul,
RulVec (for vector randomization),
RulCol (for color randomization) and
RulNoise (for creating noise) .
All these modules are implemented as static classes in the RUL namespace.

Other versions
---------------------
This version of RUL comes with custom vector and color types, but ports for [Unity](https://github.com/CaptainBubbles/RUL_Unity) and [MonoGame](https://github.com/CaptainBubbles/RUL_MonoGame) exist.

Setting up RUL
----------------------
The easiest way to get started with RUL is to check out the tutorials in the [wiki](https://github.com/CaptainBubbles/RUL/wiki). They cover everything from downloading the source to generating Perlin noise.

Code samples :
----------------------
    using RUL;
    ...
    long myLong = Rul.RandLong(700,1000); // Creates random long between 700 and 1000

    long myInt = Rul.RandInt(5,10,InclusionOptions.Upper); //Returns random int between 6 and 10

    string name = Rul.RandElement("Jon","Ned","Bran"); //Returns one of the given elements

    int probablyOne = Rul.RandElement(new int[] {1,2,3},0.9F); //Returns 1 in nine out of ten cases

    Vec3 unitVec = RulVec.RandUnitVec3(); //Returns a random 3D vector with length 1

    Col lightColor = RulCol.RandCol(200); //Returns a light color

    Col darkRed = RulCol.RandCol(Hues.Red, LuminosityTypes.Dark); //Returns a dark shade of red

    float[,] noise = RulNoise.RandSimplexNoise2(400,400); //Returns simplex noise

License
-----------
This library is published under the very permissive MIT license. See [http://opensource.org/licenses/MIT](http://opensource.org/licenses/MIT) for information on what you can and cannot do with this software.

repository: MStadlmeier/RUL

repo_name: MStadlmeier/RUL


text:
# 2do - Interactive todo CLI

2do is an interactive todo CLI.

## The Story:
todo

## Main features:
todo

## Installation
todo

## Usage
todo

## Compatibility

2do is cross-platform → Windows, MacOSX, Linux.

🤓 - **Windows users will get best experience with Windows Terminal / WSL**.

## Built with

- [click](https://github.com/pallets/click)
- [tinydb](https://github.com/msiemens/tinydb)
- [inquirer](https://github.com/CITGuru/PyInquirer)
- [rich](https://github.com/willmcgugan/rich)

## Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
Please make sure to update tests as appropriate.

### Local dev environment (MacOSX / Linux)

- Clone this project
- cd into project directory
- Run: poetry install
- Run: poetry env info -p
- Add [bash alias](https://linuxize.com/post/how-to-create-bash-aliases/):

```bash
alias 2do='PYTHONPATH=<LOCAL 2DO PROJECT DIR> <POETRY VENV PATH>/bin/python3 <LOCAL 2DO PROJECT DIR>/2do/cli.py'
```

- Now you can run 2do from bash and code changes in local 2do dir will apply.

## License

[MIT](https://choosealicense.com/licenses/mit/)