text:
# p4-jenkins

## Requirements

* Jenkins 1.509 or greater.
* P4D 12.1 or greater.
* Minimum Perforce Protection of 'open' for the Jenkins user.
If you wish to use the Review Build feature you will need Swarm.  Swarm 2014.2 or greater is required to support Jenkins authentication.

## Install

1. Open Jenkins in a browser; e.g. http://jenkins_host:8080
2. Browse to 'Manage Jenkins' --> 'Manage Plugins' and Select the 'Available' tab.
3. Find the 'P4 Plugin' or use the Filter if needed
4. Check the box and press the 'Install without restart' button

If you are unable to find the plugin, you may need to refresh the Update site.

1. Select the 'Advanced' tab (under 'Manage Plugins')
2. Press the 'Check now' button at the bottom of the page.
3. When 'Done' go back to the update centre and try again.

## Building

To build the plugin and run the tests use the following:

	mvn package

Note: for the tests to run you must have p4d in your PATH, to skip tests use the -DskipTests flag.

## Manual install

1. Open Jenkins in a browser; e.g. http://jenkins_host:8080
2. Browse to 'Manage Jenkins' --> 'Manage Plugins’ and Select the 'Advanced' tab.
3. Press the 'Choose File' button under the 'Upload Plugin' section
4. Find the location of the 'p4.hpi' file and Select Upload
5. Choose the 'Download now and install after restart' button (this might be different on the newer version)

## Credentials

The plugin makes use of the Jenkins Credential store making it easier to manage the Perforce Server connection for multiple Jenkins jobs.  Perforce Server credentials must be added to the Global or a user defined domain, using one of the two supported Perforce Credentials: 'Perforce Password Credential' or 'Perforce Ticket Credential'.

![Global credentials](docs/images/1.png)

To add a Perforce Credential:

1. Navigate to the Jenkins Credentials page (select 'Credentials' on the left hand side)
2. Select 'Global credentials' (or add domain if needed)
3. Select 'Add Credentials' from the left hand side
4. Choose 'Perforce Password Credential' from the 'Kind' drop-down select
5. Enter a Description e.g. local test server
6. Enter the P4Port e.g. localhost:1666
7. Enter a valid username and password
8. Press the 'Test Connection' button (you should see Success)
9. Click 'Save' to save.

![Perforce Credential](docs/images/2.png)

The Perforce Ticket Credential supports using a ticket file (such as the default P4TICKETS file) or a ticket value (returned by the command p4 login -p).  If Ticket authentication is used for remote builds the Ticket must be valid for the remote host (either login on the remote host or use p4 login -a).

All Perforce Credential types support SSL for use on Secured Perforce Servers; to use just check the SSL box and provide the Trust fingerprint.

![P4Trust Credential](docs/images/3.png)

## Workspaces

Perforce workspaces are configured on the Jenkin Job configuration page and support the following behaviours:

* Static

The workspace specified must have been previously defined.  The Perforce Jenkins user must either own the workspace or the spec should be unlocked allowing it to make edits.  The workspace View remains static, but Jenkins will update other fields such as the workspace root and clobber option.

![Static config](docs/images/4.png)

* Spec File

The workspace configuration is loaded from a depot file containing a Client workspace Spec (same output as p4 client -o and the Spec depot '.p4s' format).  The name of the workspace must match the name of the Client workspace Spec.

![Spec config](docs/images/spec.png)

* Manual

This allows the specified workspace to be created (if it does not exist) or update the spec by setting the various options.  Jenkins will fill out the workspace root and may override the clobber option.

![Manual config](docs/images/5.png)

* Template & Stream

In this mode the workspace View is generated using the specified template workspace or stream.  The name of the workspace is generated using the Workspace Name Format field and makes it an ideal choice for matrix builds.

![Stream config](docs/images/6.png)

## Populating

Perforce will populate the workspace with the file revisions needed for the build, the way the workspace is populated is configured on the Jenkin Job configuration page and support the following behaviours:

* Automatic cleanup and sync

Perforce will revert any shelved or pending files from the workspace; this includes the removal of files that were added by the shelved or pending change.  Depending on the two check options boxes Perforce will then clean up any extra files or restore any modified or missing files.  Finally, Perforce will sync the required file revisions to the workspace populating the 'have' table.

![Automatic populate](docs/images/auto_pop.png)

* Forced clean and sync

Perfore will remove all files from under the workspace root, then force sync the required file revisions to the workspace.  If the populating the 'have' table options is enabled then the 'have' list will be updated.

![Force populate](docs/images/force_pop.png)

This method is not recommended as the cost of IO resources on server and client are high.  Apart from exceptional circumstances the Automatic cleanup and sync option will produce the same result.

* Sync only

Perforce will not attempt to cleanup the workspace; the sync operation will update all files (as CLOBBER is set) to the required set of revisions.  If the populating the 'have' table options is enabled then the 'have' list will be updated.

![Sync populate](docs/images/sync_pop.png)

## Building

Building a Jenkins Job can be triggered using the SCM polling option, Build Now button or calling the build/ URL endpoint.

To enable SCM polling, check the 'Poll SCM' option and provide a Schedule using the Cron format.  For example every 10 minutes Monday to Friday, the 'H' is a time offset (calculated using a Hash of the Job name).

![Build trigger](docs/images/trigger.png)

To build immediately select the Build now button...

![Build now](docs/images/now.png)

Or use the call the build/ URL endpoint e.g. http://jenkins_host:8080/job/myJobID/build

(where myJobID is the name of your job and jenkins_host the name or IP address of your Jenkins server).

### Building at a change

A Jenkins job can build at any point in the codes history, identified by a Perforce change or label.

The Jenkins job can be _pinned_ to a Perforce change or label by setting the `Pin build at Perforce Label` field under the Populate options.  Any time the Jenkins job is trigged, it will only build upto the pinned point.

Alternativly, a change or label can be passed using the `Build Review` paramiters or URL end point (see the _Build Review_ chapter for details)

## Filtering

When polling is used, changes can be filtered to not trigger a build; the filters are configured on the Jenkin Job configuration page and support the following types:

* Exclude changes from user

Changes owned by the Perforce user specified in the filter will be excluded.

![User Filter](docs/images/userF.png)

* Exclude changes from Depot path

Changes where all the file revision's path starting with the String specified in the filter will be excluded.

![Path Filter](docs/images/pathF.png)

For example, with a Filter of "//depot/main/tests":

Case A (change will be filtered):

    Files:
        //depot/main/tests/index.xml
        //depot/main/tests/001/test.xml
        //depot/main/tests/002/test.xml

Case B (change will not be filtered, as build.xml is outside of the filter):

    Files:
        //depot/main/src/build.xml
        //depot/main/tests/004/test.xml
        //depot/main/tests/005/test.xml

## Review

The plugin supports a Build Review Action with a review/build/ URL endpoint.  Parameters can be passed informing Jenkins of Perforce shelf to unshelve and changelist to sync to.  There are also Pass/Fail callback URLs for use with Swarm.

An example URL that would build the review in the shelved change 23980:

http://jenkins_host:8080/job/myJobID/review/build?status=shelved&review=23980

The Build Review Action support the following parameters:
* status (shelved or submitted)
* review (the pending shelved change)
* change (the submitted change)
* label (a Perforce label, instead of change)
* pass (URL to call after a build succeeded)
* fail (URL to call after a build failed)

*Please note these paramiter are stored in the Environment and can be used with variable expansion e.g. ${label}; for this reason please avoid these names for slaves and matrix axis.*

The Build Review Action can be invoked manually from within Jenkins by selecting the Build Review button on the left hand side.  This provides a form to specify the parameters for build.

![Build review](docs/images/review.png)

![Build manual](docs/images/manual.png)

## Changes Summary

After a build Jenkins provides the ability to see the details of a build.  Select the build of interest from the Build History on the left hand side.  You then get a Perforce change summary for the build and clicking on the View Detail link for specific files.

![Change summary](docs/images/summaryC.png)

Detailed view...

![Change detail](docs/images/detailC.png)

## Tagging Builds

Jenkins can tag builds automatically as a Post Build Action or allow manual tagging of a build.  The Tags are stored in Perforce as Automatic Labels with the label view based on the workspace at the time of tagging.

Tagging with Post Build Action

![Post Action tag](docs/images/tag.png)

Manual Tagging

* Select the build that you wish to tag from the project page.

![Manual tag](docs/images/manualT.png)

* Click on the 'Label This Build' link on the left hand panel, if the build has already been tagged the link will read 'Perforce Label'.

![Label tag](docs/images/labelT.png)

* Update the label name and description as required and click 'Label Build' to add the label to Perforce.

![Update tag](docs/images/updateT.png)

* Once the build is labeled you will see the label details appear in a table above.  New labels can be added to the same build or labels can be updated by providing the same label name.

## Publishing Build assets

Jenkins can automatically shelve or submit build assets to Perforce.  Select the 'Add post-build action' and select the 'Perforce: Publish assets' from the list.  Select the Credentials and Workspace options, you can connect to a different Perforce server if required.  Update the description if required, ${variables} are expanded.

Shelving with Post Build Action

![Shelve Asset](docs/images/ShelveAsset.png)

Submitting with Post Build Action

![Submit Asset](docs/images/SubmitAsset.png)

## Repository Browsing

Repository browsing allows Jenkins to use an external browser, like Swarm, P4Web, etc... to navigate files and changes associated with a Jenkins build.

To enable the feature select the Repository browser from the Job Configuration page and provide the full URL to the browser.

![Repo list](docs/images/repos.png)

Link to change in Swarm

![Repo list](docs/images/swarm.png)

## Troubleshooting

### Connection issues

**Error**: Setting up a SSL Credentials connection to Perforce.

```
Unable to connect: com.perforce.p4java.exception.ConnectionException:
Error occurred during the SSL handshake:
invalid SSL session
```

**Solution**: Due to current US export control restrictions for some countries, the standard JDK package only comes with 128 bit encryption level cyphers. In order to use P4Java to connect to an SSL-enabled Perforce server, those living in eligible countries may download the unlimited strength JCE (Java Cryptography Extension) package and replace the current default cryptography jar files on the build server with the unlimited strength files.

The libraries can be downloaded from:

   http://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.html

Installation instructions can be found in the file 'README.txt' in the JCE download.

---

**Error**: Perforce login error part way through a build with an Edge/Commit setup.

```
Perforce password (P4PASSWD) invalid or unset.
- no such file(s).
ERROR: Unable to update workspace:
com.perforce.p4java.exception.AccessException: Perforce password (P4PASSWD) invalid or unset.
```

**Solution**: The following configurables must be set to allow the Edge to forward the login information to the Commit server.

`p4 configure set cluster.id=myID`
`p4 configure set myEdge#rpl.forward.login=1`

## Release notes

### Release 1.2.6 (major features/fixes)

[@15293](swarm.workshop.perforce.com/changes/15293) - Add retry attempts to Perforce Tasks. If a task fails due to an exception then the task will retry based on the value specified in the connection Credential.

[@15249](swarm.workshop.perforce.com/changes/15249) - Null protection if Label Owner is not set.  Fall back to “unknown” for user.

[@15138](swarm.workshop.perforce.com/changes/15138) - StreamName not shown in Manual Workspace config.

### Release 1.2.5 (major features/fixes)

[@14838](swarm.workshop.perforce.com/changes/14838) - Check if the workspace exists before cleanup. JENKINS-29030

[@14779](swarm.workshop.perforce.com/changes/14779) - Add shelved changes to built changes list. JENKINS-25724

[@14173](swarm.workshop.perforce.com/changes/14173) - Support P4D 15.1 'reconcile -m'. Client workspace MODTIME option is no longer required with -m.

[@14150](swarm.workshop.perforce.com/changes/14150) - URL Encode/Decode the depot path for changes. Filenames with ampersands was causing Jelly to break when showing the change detail. JENKINS-29017

[@14040](swarm.workshop.perforce.com/changes/14040) - Delay polling if a build is in progress.

[@14035](swarm.workshop.perforce.com/changes/14035) - Publish on Success option. Added a checkbox to the Publish step to only shelve/submit change if the build succeeded.

[@13994](swarm.workshop.perforce.com/changes/13994) - Make TaskListener as transient.

### Release 1.2.4 (major features/fixes)

[@13800](swarm.workshop.perforce.com/changes/13800) -
Updated P4Java to 15.1

[@13795](swarm.workshop.perforce.com/changes/13795) -
(matthauck) Fix JENKINS-28760: Set line endings explicitly for template workspaces

[@13777](swarm.workshop.perforce.com/changes/13777) -
(matthauck) Fix JENKINS-28726: Allow for default matrix execution strategy

[@13701](swarm.workshop.perforce.com/changes/13701) -
Move Labelling into a Task.

[@13681](swarm.workshop.perforce.com/changes/13681) -
Abstracted Expand class from Workspace.  Added support for Label variable expansion in the name and description.

[@13676](swarm.workshop.perforce.com/changes/13676) -
Added support for `p4 clean`. If the Perforce server is 14.1 or greater then the `-w` flag is used (p4 clean), otherwise the original auto clean up code.

### Release 1.2.3 (major features/fixes)

[@13619](swarm.workshop.perforce.com/changes/13619) -
Document building at a change. JENKINS-28301

[@13604](swarm.workshop.perforce.com/changes/13604) -
Improved error handling and fixed test case issue.

[@13603](swarm.workshop.perforce.com/changes/13603) -
Improved Error for Publish step when connection is down.

### Older Releases

Please refer to the [Activity](https://swarm.workshop.perforce.com/projects/p4-jenkins/activity) feed.

## Known limitations
* One Jenkins Job per Swarm branch


text:
== Kinetic Request CE Submission Search
Searches a kapp or form for submissions and returns any matching submission objects in the
specified return format.

=== Parameters
[Error Handling]
  Determine what to return if an error is encountered.
[Space Slug]
  The space the submission is being retrieved from (defaults to info value if not provided).
[Kapp Slug]
  The slug of the Kapp to search for submissions in
[Form Slug]
  The slug of the Form to search for submissions in
[Include]
  Comma-separated list of properties to include in the response. Options include 'details', 'activities', 'children', 'descendants', 'form', 'origin', 'parent', 'type', 'values', 'values.raw', and 'values[FIELD NAME]'
[Core State]
  Searches for submissions that have a core state that matches this parameter. Options
  are 'Draft', 'Submitted', 'Closed'.  If no value is provided, the results will
  contain submissions in all core states.
[Timeline]
  Date property to search by. Options are 'createdAt','submittedAt','updatedAt', and
  'closedAt'. The default value is 'createdAt'.
[Direction]
  Result Set sorting direction. Options are 'ASC' and 'DESC' The default value is
  descending (DESC).
[Start]
  Start date/time of the timeline. This value should be used to both refine and limit
  the search results. Format: yyyy-MM-ddTHH:mm:ssZ
[End]
  End date/time of the timeline. This value should be used to both refine and limit
  the search results. Format: yyyy-MM-ddTHH:mm:ssZ
[Limit]
  Limit the number of results returned. If not provided, the server will limit the
  results to 25 submissions.  Maximum value 1000.
[Query]
  The query that will be used to search the submissions. Ex: values[company]=Kinetic
[Page Token]
  The value to use as the offset for the page of submissions to return. The submission that
  matches this value will not be included in the results.
[Return Type]
  The format that the results should be returned in.

=== Sample Configuration
Error Handling:         Error Message
Space Slug:
Kapp Slug:              catalog
Form Slug:              test-form
Include:
Core State:
Timeline:
Direction:
Start:
End:
Limit:                  1000
Query:                  include=values&limit=100&q=values[Any Text] IN ("Testing","String")
Page Token:             70b37e5f-d64c-11e8-bbc0-ebb51a7fa935
Return Type:            JSON

=== Results
[Handler Error Message]
  Error message if an error was encountered and Error Handling is set to "Error Message".
[Count]
  The number of submissions that have been returned.
[Next Page Token]
  The ID of the last submission retrieved by the API if the Limit input is reached.  This can be
  used in a subsequent call to continue retrieve results.
[Result]
  List of submissions that match the query --JSON,XML, or ID List

=== Detailed Description
Given a Request CE Space, Kapp, and optional Form, searches for submissions with a given query. The
query parameter allows anything that can be passed into the query parameter in the /submissions
endpoint of the Request CE API (including the limit= and q= parameters). Before being sent to the
Request CE instance, the query is escaped by running the URI.escape() command on the inputted query
string. After all the submissions that match the query have been returned, the handler formats it
into one of three possible output formats: a JSON string, an XML string, or an XML list of
submission ids.

If ID List is selected as the Return Type, the output looks like:

<ids><id>f4cd6acf-e552-11e5-9c32-97bbc6bf7f84</id><id>bbc6bf7f-...</id></ids>

Note: The limit by default for submissions in 25, so if you want to retrieve more submissions than
that add a limit=n (where n is an integer representing the new limit) to the query string

license: MIT

repository: kineticdata/peripherals-kinetic-core

branch_name: refs/heads/master

extension: md

gha_language: Ruby

gha_license_id: MIT

license_type: permissive

repo_name: kineticdata/peripherals-kinetic-core


text:
# Pleolpidium cuculus E.J. Butler, 1907 SPECIES

#### Status
SYNONYM

#### According to
The Catalogue of Life, 3rd January 2011

#### Published in
Memoirs of the Dept. Agric. India, Bot. Ser. 1(5): 125 (1907)

#### Original name
Pleolpidium cuculus E.J. Butler, 1907

### Remarks
null

repository: mdoering/backbone

path: /life/Fungi/Chytridiomycota/Chytridiomycetes/Chytridiales/Olpidiaceae/Rozella/Rozella cuculus/ Syn. Pleolpidium cuculus/README.md

repo_name: mdoering/backbone


text:
### 4.1.0
- **[IMPORTANT]** Comments are now a separate Source (if you are using Comments for any Post Type Source you *will need to edit your Engine and rebuld your index*, this is not done automatically and should be planned for when updating)
- **[New]** Refactored indexer for stability, reliability, and performance optimization
- **[New]** `searchwp\index\update_entry` action when an Entry is updated in the Index
- **[New]** `searchwp\debug\front_end_summary` filter to control whether the HTML comment summary is output when debugging is enabled
- **[New]** `wp searchwp reindex` WP-CLI command
- **[New]** `wp searchwp diagnostics` WP-CLI command
- **[New]** Primary keys added to database tables *for new installs* where previously there were none
- **[New]** Adds `$query` property to `\SWP_Query` to reference underlying query object
- **[New]** `searchwp\source\comment\parent_attribution\strict` hook to control whether SearchWP is strict about parent attribution which can introduce overhead in some cases
- **[New]** SearchWP will now generate more expected variations of regex pattern matches
- **[New]** `searchwp\tokens\generate_parttern_match_variations` filter to control whether additional pattern match variations are generated during indexing
- **[New]** Ability to enter custom ignored queries for Statistics (with wildcard `*` support)
- **[New]** Ability to automatically trim Statistics logs
- **[New]** Notice when WordPress available memory can be improved
- **[New]** Export/import now optionally includes Settings, Stopwords, and Synonyms
- **[New]** All ACF fields are displayed in the Custom Fields dropdown when applicable, not only ACF "Repeatables"
- **[New]** Synonyms actions: Sort ASC, Sort DESC, Clear
- **[Change]** Refines default data indexed for Taxonomy Terms to be: taxonomy name, term name, term slug, and term description
- **[Change]** Abstracts Statistics Dashboard Widget from `jquery-ui-tabs`
- **[Improvement]** Indexer auto-scaling when server load is high
- **[Improvement]** Handling of HTML-formed content during indexing
- **[Improvement]** Integration of partial matches and keyword stemming
- **[Improvement]** Indexer delta trigger specificity in some cases
- **[Improvement]** Document processing handling, footprint
- **[Improvement]** Highlighting when a suggested search has been returned
- **[Fix]** Prevent duplicate results during weight transfer in some cases
- **[Fix]** `AND` logic restriction, performance in some cases
- **[Fix]** Persistent dismissal of missing integration notice
- **[Fix]** Synonyms management in some cases
- **[Update]** Dependencies
- **[Update]** Translation source
- **[Update]** Interface updates and refinements

### 4.0.34
- **[Fix]** Fixes regression introduced by Shortcodes fix in `4.0.33` (proper fix in Shortcodes `1.8.2`)
- **[Update]** Updated updater

### 4.0.33
- **[New]** `searchwp\swp_query\mods` filter to add Mods to `SWP_Query`
- **[Fix]** PHP Warning introduced in 4.0.32
- **[Fix]** Prevention of duplicate indexing processes in some cases
- **[Fix]** Display issue on Statistics screen
- **[Fix]** PHP Warning when Admin color schemes have been removed
- **[Fix]** PDF parsing taking place unnecssarily in some cases
- **[Improvement]** Disable `searchwp_search_results` Shortcode when generating excerpts so as to prevent unwanted loop

### 4.0.32
- **[Fix]** Duplicate indexer processes in some cases
- **[Fix]** Inaccurate batch size handling in some cases
- **[Fix]** Prevent redundant search suggestion output in some cases
- **[Improvement]** Post stati validation when parent attribution is enabled
- **[Improvement]** Attachment status handling over time
- **[New]** `RAND(seed)` support by suffixing `random` `Mod` `order_by` with a colon and seed e.g. `random:10` will be `RAND(10)`

### 4.0.31
- **[Improvement]** Adds `NOT IN` option to Media File Type Rule
- **[Improvement]** Debug HTML comment block output during Admin requests
- **[Fix]** Custom Attribute Options not returning proper Label after saving
- **[Fix]** Relocate `searchwp\query\search_string` hook to fire earlier
- **[Fix]** Issue when performing cross-site Multisite search
- **[Fix]** `searchwp\query\tokens\limit` default value
- **[New]** Action `searchwp\query\core_mods_out_of_bounds` fires when core Mods are considered out of bounds

### 4.0.30
- **[Fix]** UI changes introduced by WordPress 5.5
- **[Fix]** Global excerpt generation from search suggestions
- **[Fix]** Synonym migration from SearchWP 3.x

### 4.0.29
- **[Fix]** Token handling in some cases
- **[Fix]** Document content handling when using alternate indexer in some cases
- **[Improvement]** Tokenization of HTML in some cases
- **[New]** `searchwp\entry\update_data\before` action fired before `Entry` data is retrieved
- **[Update]** Bundle dependencies

### 4.0.28
- **[Fix]** Prevent inapplicable comment edit events from triggering delta updates
- **[Improvement]** Reduced index method checks
- **[Improvement]** Reactivity when observing meta updates

### 4.0.27
- **[Fix]** File Content meta box display in some cases
- **[Fix]** Entries not being reintroduced after failing when using alternate indexer
- **[Fix]** Display of Source Attribute Options when statically defined
- **[Fix]** UI display edge cases
- **[Change]** Token handling chunked in more cases so as to avoid issues when hosts limit query character length

### 4.0.26
- **[Fix]** Handling of `SWP_Query` `tax_query` argument
- **[New]** Advanced setting checkbox to control whether stored document content is purged and re-indexed during index rebuilds
- **[Update]** Translation source

### 4.0.25
- **[Fix]** Regression introduced in 4.0.24 when utilizing PDF Metadata
- **[Improvement]** Note displayed in SearchWP Document Content meta box when document is queued but not yet processed
- **[Update]** Translation source

### 4.0.24
- **[Fix]** Handling of PDF metadata that includes invalid characters
- **[Fix]** Searching of hierarchical post types in the Admin
- **[Improvement]** Performance when handling documents outside the indexing process
- **[Update]** Bundle dependencies

### 4.0.23
- **[Fix]** Utilize previously extracted PDF metadata instead of parsing it repeatedly
- **[Change]** Updated default batch size for Media to 3, can be customized with `searchwp\indexer\batch_size\post.attachment` hook
- **[Improvement]** Handling of urlencoded tokens in some cases

### 4.0.22
- **[New]** Query parameter support for `post_type` when using `SWP_Query` (additional parameter support is planned)
- **[Fix]** Issue with partial matching yielding zero results in some cases
- **[Fix]** Quoted search support for `WP_Post` Content, Document Content
- **[Improvement]** Reduced debug log volume (logs should be deleted once you're done debugging)

### 4.0.20
- **[New]** New filter `searchwp\source\post\db_where` to customize global `WHERE` limits per post type
- **[New]** License key is automatically activated when provided via constant or hook
- **[Fix]** Error on uninstall when removing all data
- **[Fix]** Issue where Mods were not applied to `SWP_Query` in some cases
- **[Change]** No longer relying on `excerpt_more` when working with excerpts, now using ellipsis filtered by `searchwp\utils\excerpt_more`
- **[Improvement]** Handling of rare cases where index would need to be woken up repeatedly in order to build
- **[Improvement]** Omits redundant Entry retrieval in some cases
- **[Improvement]** Significant performance retrieval when generating excerpts (e.g. Highlighting)
- **[Improvement]** Advanced Custom Fields integration support

### 4.0.19
- **[Notice]** `Mod`s have in part been cleaned up and refined in this release, which may affect your usage. Please review any `Mod`s you are using by testing this update on a staging server. If you are manipulating relevance weight based on date, it is likely you will need to update your hooks. Snippets have been updated on the KB article [https://searchwp.com/?p=222848](https://searchwp.com/?p=222848) for review. Please also ensure your SearchWP Extensions are up to date as well.
- **[Fix]** Source `Mod` `WHERE` clauses causing errors in some cases
- **[Fix]** Raw `Mod` `WHERE` clauses had no local alias to utilize
- **[Fix]** `Mod` `JOIN` claus order was not retained causing errors in some cases
- **[Improvement]** Optimized `Mod` handling in `SWP_Query`
- **[Improvement]** Disable integration extension checks when doing AJAX

### 4.0.18
- **[Fix]** Error when using `mod` argument of `\SearchWP\Query` parameters array
- **[Improvement]** Control over Settings page navigation

### 4.0.17
- **[Note]** Rebuilding your index using the Rebuild Index button on the Engines tab of the SearchWP settings screen is recommended after updating
- **[Fix]** Delta update regression introduced in `4.0.13`
- **[Fix]** Error when applying delta update to Source that no longer exists
- **[Improvement]** Handling of delta update process during failures
- **[Improvement]** Delta update queue handling during index rebuild

### 4.0.16
- **[Fix]** Invalid range in character class introduced in 4.0.15 for PHP 7.3+
- **[Change]** `searchwp\tokens\whitelist\only_full_matches` retagged as `searchwp\tokens\regex_patterns\only_full_matches`
- **[Change]** `searchwp\tokens\apply_rules_to_whitelist` retagged as `searchwp\tokens\apply_rules_to_pattern_matches`
- **[Change]** `searchwp\tokens\whitelist_regex_patterns` retagged as `searchwp\tokens\regex_patterns`
- **[Change]** Regex pattern matches are now tokenized during indexing (but remain exclusive when searching by default when applicable)
- **[New]** Filter `searchwp\tokens\tokenize_pattern_matches\indexing` to disable new tokenizing of pattern match behavior during indexing

### 4.0.15
- **[New]** New filter `searchwp\tokens\string` to customize strings before tokenization
- **[Fix]** Handling of synonyms when finding partial matches
- **[Fix]** Implementation and handling of regex pattern match tokenization setting
- **[Improvement]** Dash/hyphen and word match regex patterns
- **[Improvement]** `searchwp\source\post\excerpt_haystack` filter now passes arguments array
- **[Update]** Translation source

### 4.0.14
- **[Fix]** Issue where partial matches from keyword stems were not found in some cases
- **[Fix]** Partial match token processing limited to applicable site(s)
- **[Fix]** Excerpt generation when handling unregistered Shortcodes

### 4.0.13
- **[Fix]** Delta update routine when using alternate indexer that caused unwanted exit
- **[Fix]** `searchwp\document\content` implementation
- **[Improvement]** Index integrity check when rebuilding
- **[Improvement]** Source hook management in multisite
- **[Improvement]** Upgrade routine process

### 4.0.12
- **[Fix]** Inability to filter `searchwp\settings\capability`
- **[Fix]** Issue with Highlighting in some cases
- **[Fix]** Document Content not properly considered for global excerpt in some cases
- **[Fix]** Warning when reacting to invalid `Entry` during indexing
- **[Fix]** Namespace issue with PDF parsing in some cases
- **[Fix]** Unnecessary provider reset when switching to the same site in multisite
- **[Update]** Updated updater

### 4.0.11
- **[Fix]** Loss of tokens when applying partial match logic in some cases
- **[Update]** Revised MySQL minimum to 5.6 because of `utf8mb4_unicode_520_ci` collation requirement

### 4.0.9
- **[Fix]** Regression introduced in 4.0.6 that prevented non `WP_Post` results from returning

### 4.0.8
- **[Fix]** Issue where taxonomy Rules for Media were not applied correctly in some cases

### 4.0.7
- **[Fix]** Mod `WHERE` clauses not restricted to `Source` when defined

### 4.0.6
- **[Change]** Post is now returned when parent weight transfer is enabled but Post has no `post_parent`
- **[Improvement]** Excerpt handling for native results
- **[Improvement]** Additional prevention of invalid `WP_Post` results being returned in one case

### 4.0.5
- **[New]** Filter to control stemmer locale `searchwp\stemmer\locale`
- **[Improvement]** Token stems/partial matches are considered during `AND` logic pass
- **[Fix]** String not sent to `searchwp\stemmer\custom`
- **[Change]** `searchwp\query\partial_matches\buoy` is now opt-in

### 4.0.4
- **[Fix]** Issue where `AND` logic would not apply in some cases
- **[Fix]** Issue where additional unnecessary query clauses are added in some cases
- **[Fix]** Issue with delta updates not processing when HTTP Basic Auth is active
- **[Fix]** Minimum PHP version requirement check (which is 7.2)

### 4.0.3
- **[Fix]** Issue where tokens table was not reset during index rebuild

### 4.0.2
- **[New]** Support for `BETWEEN`, `NOT BETWEEN`, `LIKE`, and `NOT LIKE` compare operators for `Mod` `WHERE` clauses
- **[Fix]** Handling of `Mod` `WHERE` clauses in some cases
- **[Fix]** Handling of REST parameters when returning search results

### 4.0.1
- **[New]** Check for remnants of SearchWP 3 that were not removed as per the Migration Guide
- **[New]** `searchwp\source\post\attributes\comments` action when retrieving Post comments
- **[Fix]** Handling of empty search strings in some cases

### 4.0.0
- **[New]** Complete rewrite of SearchWP

repository: Threespot/threespot-wp-sandbox

gha_language: PHP

repo_name: Threespot/threespot-wp-sandbox


text:
# Environment-configurable docker image with Nginx and automatic Let's Encrypt

The image is fully configurable via environment variables and
does not require the creation of configuration files.

Inspired by [nginx-le/nginx-le](https://github.com/nginx-le/nginx-le).

## Example `docker-compose.yml`

You only need to set `SERVER_NAMES`, `LE_EMAIL`
and `PROXY_PASS` OR `FASTCGI_PASS` to get started.

Configuration files are generated every time the container starts,
so there is no need to rebuild for the changes to take effect.

```yaml
version: '3.5'
services:
    nginx:
        image: avzd/nginx-le:latest

        environment:
            ## Space-separated list of domains to host
            SERVER_NAMES: api.example.com api2.example.com

            ## https://letsencrypt.org/docs/expiration-emails/
            LE_EMAIL<EMAIL_ADDRESS>
            ## Uncomment this to run certbot with --dry-run
            # LE_DRY_RUN: "yes"

            ## Simple proxy HTTP
            # PROXY_PASS: http://host:8080

            ## OR

            ## FastCGI (e.g. for php-fpm):
            # FASTCGI_PASS: host:9000
            # FASTCGI_PARAM_SCRIPT_FILENAME: /sources/index.php

            ## ...
            ## See all options in Dockerfile ENV

        # volumes:
            ## To make dhparams persistent
            # - /path/to/ssl:/etc/nginx/ssl

            ## To make certificates persistent
            # - /path/to/letsencrypt:/etc/letsencrypt

        restart: always

        logging:
            driver: json-file
            options:
                max-size: "10m"
                max-file: "5"

        ports:
            - "80:8080"
            - "443:4430"
```

## See also

 * [Traefik](https://containo.us/traefik/)
 * [nginx-le/nginx-le](https://github.com/nginx-le/nginx-le)
 * [gilyes/docker-nginx-letsencrypt-sample](https://github.com/gilyes/docker-nginx-letsencrypt-sample)

repository: avz/nginx-le

repo_name: avz/nginx-le


text:
---
title: Separator Whitespace
toc: false
---

Checks that whitespace is present or absent around a separators.

### Configuration

```json
{
    "type": "SeparatorWhitespace",
    "props": {
        "severity": "WARNING",
        "dotPolicy": "around",
        "commaPolicy": "none",
        "semicolonPolicy": "none"
    }
}
```

| Policy                 | Operators
| ---------------------- | ------------- |
| `dotPolicy`            | `.`           |
| `commaPolicy`          | `,`           |
| `semicolonPolicy`      | `;`           |

Available policy values are:

| Option    | Description |
| --------- | ----------- |
| `around`  | enforce whitespace before and after operator |
| `before`  | enforce whitespace before and no whitespace after operator |
| `after`   | enforce no whitespace before and whitespace after operator |
| `none`    | enforce no whitespace before and after operator |
| `ignore`  | skip checks |

repository: HaxeCheckstyle/docs

branch_name: refs/heads/gh-pages

gha_language: CSS

repo_name: HaxeCheckstyle/docs


text:
---
title: Genocide
process: atrocity-crimes
published: true
sort_order: 4
---

Genocide is criminal acts committed with the intent to destroy in whole or in part a national, ethnic, racial, or religious group, and such acts include killing or imposing measures to prevent births, for instance. Unlike CAH or War Crimes, Genocide does not require a particular context to be happening simultaneously. However, like CAH, Genocide can happen during war or peace.


text:
# nmt-wizard-docker

The goal of this project is to encapsulate MT frameworks in Docker containers and expose a single interface for preprocessing, training, translating, and serving models. The project targets advanced users who are looking to automate the production and validation of translation models.

The [available Docker images](https://hub.docker.com/u/nmtwizard) extend the original frameworks with the following features:

* Data weighting, sampling, and preprocessing from raw training files.
* Files synchronization from and to remote storages such as Amazon S3, Swift, or any server via SSH.
* Metadata on model history such as parent model, training data that was used, training time, etc.
* Regular HTTP request to an URL to declare running status.

It supports the [OpenNMT-py](https://github.com/OpenNMT/OpenNMT-py/) and [OpenNMT-tf](https://github.com/OpenNMT/OpenNMT-tf/) training frameworks, and provides translate-only frameworks using online translation API from DeepL, Google, Baidu, and others.

## Usage

We recommend using the Docker images that are available on [Docker Hub](https://hub.docker.com/u/nmtwizard).

The Docker image entrypoint is a Python script that exposes the same command line interface for all frameworks. For example, run the commands below to download the latest OpenNMT-tf image and list the available options:

```bash
docker pull nmtwizard/opennmt-tf
docker run nmtwizard/opennmt-tf -h
```

[JSON configuration files](#configuration) are used to define training data, data preprocess, and framework specific options (model, hyperparameters, etc).

As an example, let's train an English-German Transformer model with OpenNMT-tf.

**1\. Prepare the data.**

For simplicity, we assume that the training data is already tokenized and the current directory contains a `data/` subdirectory with the following structure:

```
$ tree data/
.
├── corpus
│   ├── train.de
│   └── train.en
└── vocab
    └── shared-vocab.txt
```

where:

* `train.de` and `train.en` are tokenized training files
* `shared-vocab.txt` contains one token per line and no special tokens

**2\. Define the configuration.**

The JSON configuration file is used to describe where to read the data (`data`) and how to transform it (`preprocess`). The `options` block is **specific to each framework** and defines the model and training hyperparameters. See [Configuration](#configuration) for more details.

```json
{
    "source": "en",
    "target": "de",
    "data": {
        "sample_dist": [
            {
                "path": "/data/corpus",
                "distribution": [
                    ["train", "*"]
                ]
            }
        ]
    },
    "preprocess": [
        {
            "op":"tokenization",
            "source": {
                "mode": "space"
            },
            "target": {
                "mode": "space"
            }
        }
    ],
    "vocabulary": {
        "source": {
            "path": "/data/vocab/shared-vocab.txt"
        },
        "target": {
            "path": "/data/vocab/shared-vocab.txt"
        }
    },
    "options": {
        "model_type": "Transformer",
        "auto_config": true
    }
}
```

**3\. Train the model.**

The `train` command is used to start the training:

```bash
cat config.json | docker run -i --gpus all \
    -v $PWD/data:/data -v $PWD/models:/models nmtwizard/opennmt-tf \
    --model_storage /models --task_id my_model_1 --config - train
```

This command runs the training for 1 epoch and produces the model `models/my-model-1`. The model contains the latest checkpoint, the JSON configuration, and all required resources (such as the vocabulary files).

You can run the next epoch by passing the model name as argument:

```bash
docker run --gpus all -v $PWD/data:/data -v $PWD/models:/models nmtwizard/opennmt-tf \
    --model_storage /models --task_id my_model_2 --model my_model_1 train
```

Alternatively, you can run the full training in one command by setting the training step option available in the training framework.

**4\. Translate test files.**

Once a model is saved, you can start translating files with the `trans` command:

```bash
docker run --gpus all -v $PWD/data:/data -v $PWD/models:/models nmtwizard/opennmt-tf \
    --model_storage /models --model my_model_2 trans -i /data/test.en -o /data/output.de
```

This command translates `data/test.en` and saves the result to `data/output.de`.

**5\. Serve the model.**

At some point, you may want to turn your trained model into a translation service with the `serve` command:

```bash
docker run -p 4000:4000 --gpus all -v $PWD/models:/models nmtwizard/opennmt-tf \
    --model_storage /models --model my_model_2 serve --host <IP_ADDRESS> --port 4000
```

This command starts a translation server that accepts HTTP requests:

```bash
curl -X POST http://localhost:4000/translate -d '{"src":[{"text": "Hello world !"}]}'
```

See the [REST translation API](docs/rest_api.md) for more details.

To optimize the model size and loading latency, you can also `release` the model before serving. It will remove training-only information and possibly run additional optimizations:

```bash
docker run -v $PWD/models:/models nmtwizard/opennmt-tf \
    --model_storage /models --model my_model_2 release
```

This command produces the serving-only model `models/my_model_2_release`.

## Remote storages

Files and directories can be automatically downloaded from remote storages such as Amazon S3, Swift, or any server via SSH. This includes training data, models, and resources used in the configuration.

Remote storages should be configured in a JSON file and passed to the  `--storage_config` command line option:

```json
{
    "storage_id_1": {
        "type": "s3",
        "bucket": "model-catalog",
        "aws_credentials": {
            "access_key_id": "...",
            "secret_access_key": "...",
            "region_name": "..."
        }
    },
    "storage_id_2": {
        "type": "ssh",
        "server": "my-server.com",
        "basedir": "myrepo",
        "user": "root",
        "password": "root"
    }
}
```

*See the supported services and their parameters in [SYSTRAN/storages](https://github.com/SYSTRAN/storages).*

Paths on the command line and in the configuration can reference remote paths with the syntax `<storage_id>:<path>`, where:

* `<storage_id>` is the storage identifier in the JSON file above (e.g. `storage_id_1`)
* `<path>` is a file path on the remote storage

Files will be downloaded in the `/root/workspace/shared` directory within the Docker image. To minimize download cost, it is possible to mount this directory when running the Docker image. Future runs will reuse the local file if the remote file has not changed.

## Configuration

The JSON configuration file contains all parameters necessary to train and run models. It has the following general structure:

```json
{
    "source": "string",
    "target": "string",
    "data": {},
    "preprocess": [
        {
            "op":"tokenization",
            "source": {
                "mode": "string"
            },
            "target": {
                "mode": "string"
            }
        }
    ],
    "vocabulary": {
        "source": {
            "path": "string"
        },
        "target": {
            "path": "string"
        }
    },
    "options": {},
    "serving": {}
}
```

### Description

#### `source` and `target`

They define the source and target languages (e.g. "en", "de", etc.).

#### `data`

The `data` section of the JSON configuration can be used to select data based on file patterns. The distribution is a list where each element contains:

* `path`: path to a directory where the distribution applies
* `distribution`: a list of filename patterns and weights

For example, the configuration below will randomly select 10,000 training examples in the directory `data/en_nl/train` from files that have `News`, `IT`, or `Dialog` in their name:

```json
{
    "data": {
        "sample": 10000,
        "sample_dist": [
            {
                "path": "/data/en_de/train",
                "distribution": [
                    ["News", 0.7],
                    ["IT", 0.3],
                    ["Dialog", 0.5]
                ]
            }
        ]
    }
}
```

Weights define the relative proportion that each pattern should take in the final sampling. They do not need to sum to 1. The special weight `"*"` can be used to force using all the examples associated with the pattern.

**Source and target files should have the same name and be suffixed by the language code.**

#### `preprocess`

This block applies preprocess operations, such as tokenization, to the data.

Tokenization operator accepts any tokenization options from [OpenNMT/Tokenizer](https://github.com/OpenNMT/Tokenizer/blob/master/docs/options.md).

#### `vocabulary`

This block contains the vocabularies used by translation framework.

The vocabulary file must have the following format:

* one token par line
* no special tokens (such as `<s>`, `</s>`, `<blank>`, `<unk>`, etc.)

We plan to add a `buildvocab` command to automatically generate it from the data.

#### `options`

This block contains the parameters that are **specific to the selected framework**: the model architecture, the training parameters, etc.

See the file `frameworks/./README.md` of the selected framework for more information.

#### `serving`

Serving reads additional values from the JSON configuration:

```json
{
    "serving": {
        "max_batch_size": 64
    }
}
```

where:

* `max_batch_size` is the maximum batch size to execute at once

These values can be overriden for [each request](docs/rest_api.md).

### Overriding the model configuration

When a model is set on the command line with `--model`, its configuration will be used. You can pass a partial configuration to `--config` in order to override some fields from the model configuration.

### Environment variables

Values in the configuration can use environment variables with the syntax `${VARIABLE_NAME}`.

This is especially useful to avoid hardcoding a remote storage identifier in the configuration. For example, one can define a data path as `${DATA_DIR}/en_de/corpus` in the configuration and then configure the storage identifer when running the Docker image with `-e DATA_DIR=storage_id_1:`.

## Add or extend frameworks

This repository consists of a Python module `nmtwizard` that implements the shared interface and extended features mentionned above. Each framework should then:

* extend the `nmtwizard.Framework` class and implement the logic that is specific to the framework
* define a Python entrypoint script that invokes `Framework.run()`
* define a `Dockerfile` that includes the framework, the `nmtwizard` module, and all dependencies.

Advanced users could extend existing frameworks to implement customized behavior.

## Run tests

```bash
# Install test requirements:
pip install -e .[tests]

# Run unit tests:
pytest test/

# Automatically reformat code:
black .

# Check code for errors:
flake8 .
```

repository: OpenNMT/nmt-wizard-docker

gha_language: Python

repo_name: OpenNMT/nmt-wizard-docker


text:
![Metafacture](https://raw.github.com/wiki/culturegraph/metafacture-core/img/metafacture.png)

Metafacture is a toolkit for processing semi-structured data with a focus on library metadata. It provides a versatile set of tools for reading, writing and transforming data. Metafacture can be used as a stand-alone application or as a java library in other applications. The name Metafacture is a portmanteau of the words *meta* data and manu*facture*.

Metafacture consists of a core library and a number of plugin packages and satellite projects which build on the core library and extend it with additional tools and features. This page describes the core package. Have a look at the [Plugins and Tools page](https://github.com/culturegraph/metafacture-core/wiki/Plugins-and-Tools) on the wiki for an overview of the supplementary packages and projects.

Originally, Metafacture was developed as part of the [Culturegraph](http://culturegraph.org) platform but its now used by others, too: [see who uses Metafacture](https://github.com/culturegraph/metafacture-core/wiki/Who-uses-Metafacture).

# Build

[![Build Status](https://secure.travis-ci.org/culturegraph/metafacture-core.svg?branch=master)](https://travis-ci.org/culturegraph/metafacture-core/) [![Dependency Status](https://www.versioneye.com/user/projects/5592673839656100200000d7/badge.svg)](https://www.versioneye.com/user/projects/5592673839656100200000d7) [![Maven Central](https://img.shields.io/maven-central/v/org.culturegraph/metafacture-core.svg)]() [![Javadocs](http://www.javadoc.io/badge/org.culturegraph/metafacture-core.svg?color=blue)](http://www.javadoc.io/doc/org.culturegraph/metafacture-core)

See the [.travis.yml](./.travis.yml) file for details on the CI config used by Travis.

# Getting started

You can use Metafacture either as a stand-alone application or as a Java library in your own projects.

## Metafacture as a stand-alone application

If you are only interested in running Flux scripts without doing any Java programming this is the way to go. The instructions assume that you are using a *nix-like shell.

1. Download the latest distribution package from the [metafacture-core/releases](https://github.com/culturegraph/metafacture-core/releases) page. Make sure that you do download a distribution package and _not_ a source code package.

2. Extract the downloaded archive:
   ```bash
   $ tar xzf metafacture-runner-VERSION-dist.tar.gz
   ```
   This will create a new directory containing a ready-to-use metafacture distribution.
3. Change into the newly created directory:
   ```bash
   $ cd metafacture-runner-VERSION
   ```
4. Run one of the example scripts:
   ```bash
   $ ./flux.sh examples/read/marc21/read-marc21.flux
   ```
   This example will print a number of marc21 records on standard out.

The _examples_ folder contains many more examples which provide a good starting point for learning metafacture. If you have any questions please join our [mailing list](http://lists.dnb.de/mailman/listinfo/metafacture) or use our issue-based discussion forum over at [metafacture-documentation](https://github.com/culturegraph/metafacture-documentation).

## Using Metafacture as a Java libary

If you want use Metafacture in your own Java projects all you need to do is to include the metafacture-core package as a dependency in your project. metafacture-core is available from [Maven Central](http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.culturegraph%22). To use it, add the following dependency declaration to your `pom.xml`:

    <dependency>
    	<groupId>org.culturegraph</groupId>
    	<artifactId>metafacture-core</artifactId>
    	<version>4.0.0</version>
    </dependency>

Our integration server automatically publishes successful builds of the master branch as snapshot versions on [Sonatype OSS Repository](https://oss.sonatype.org/index.html#nexus-search;quick~culturegraph).

<!--
TODO: Add

* Link to getting started tutorial
* Mention the application-archetype
-->

# Building metafacture-core from source

Building metafacture-core from source is easy. All you need is git and [maven](http://maven.apache.org/):

1. Clone the metafacture-core repository:

        $ git clone https://github.com/culturegraph/metafacture-core.git

2. Build and install in your local repository:

        $ mvn clean install

    It is important to perform this step *before* importing the project into your IDE because it generates the lexer and parser sources for Flux from antlr grammar definitions. Otherwise your IDE will fail to compile the sources if it has integrated background compilation (like Eclipse has, for instance).

See [Code Quality and Style](https://github.com/culturegraph/metafacture-core/wiki/Code-Quality-and-Style) on the wiki for further information on the sources.

<!--
TODO: Include a link to a page which explains how to write plugins
-->

# Stay updated

For support and discussion join the [mailing list](http://lists.dnb.de/mailman/listinfo/metafacture).

repository: cboehme/metafacture-core

repo_name: cboehme/metafacture-core


text:
Build status | Description
:--- | :---
[![CircleCI](https://circleci.com/gh/cypress-io/cypress/tree/develop.svg?style=svg)](https://circleci.com/gh/cypress-io/cypress/tree/develop) | `develop` branch

# Cypress

This is the Cypress monorepo, containing all packages that make up the Cypress app. See [Issue #256](https://github.com/cypress-io/cypress/issues/256) for details.

This monorepo is made up of various packages, all of which are found under the `packages` directory. They are discrete modules with different responsibilities, but each is necessary for the Cypress app and is not necessarily useful outside of the Cypress app.

Some, like `https-proxy` and `launcher`, run solely in node and support the Cypress server. Others, like `desktop-gui` and `runner`, create the GUI parts of the Cypress app. All packages can require each other using `const server = require('@packages/server')` syntax.

## Documentation

- [Main Docs](https://on.cypress.io)
- [Roadmap](https://on.cypress.io/roadmap)
- [Changelog](https://on.cypress.io/changelog)

## Development

Please see our [Contributing Guideline](/CONTRIBUTING.md) which explains repo organization, linting, testing, and other steps.

## Deployment

Please see out [deployment document](DEPLOY.md).

## License

This project is licensed under the terms of the [MIT license](/LICENSE.md).

## Badges

Let the world know your project is using Cypress.io to test with this cool badge

[![badge](https://img.shields.io/badge/cypress.io-tests-green.svg?style=flat-square)](https://cypress.io)

```
[![Cypress.io tests](https://img.shields.io/badge/cypress.io-tests-green.svg?style=flat-square)](https://cypress.io)
```

repository: smartmanru/cypress

branch_name: refs/heads/develop

gha_language: CoffeeScript

repo_name: smartmanru/cypress


text:
---
tags: meeting, infrastructure
project: infrastructure
---
<!-- markdownlint-disable MD026-->

# 2022-06-21

[:movie_camera: Zoom Meeting Room](https://zoom.us/j/92454301214?pwd=aEVoUi9EanpaakN3L1ZxRlpDQk5Ddz09)
[:calendar: Shared Calendar](https://jenkins.io/event-calendar/)
[:speech_balloon: IRC Chat Room #jenkins-infra](https://jenkins.io/chat/#jenkins-infra)
[:email: Google Group (mailing list) jenkins-infra](https://groups.google.com/g/jenkins-infra)
[🧠 Previous Meeting Notes](https://github.com/jenkins-infra/documentation/blob/main/meetings/2022-06-14.md)

## Attendees

* @hlemeur (Hervé Le Meur)
* @MarkEWaite (Mark Waite)
* @smerle33 (Stéphane Merle)
* @poddingue (Bruno Verachten)

<!--
* @dduportal (Damien Duportal)
* @zvW_c6ROSOOuJDTOracA7Q (Tim Jacomb)
-->

## Announcements :loudspeaker:

1. Weekly and 2 LTS releases - security release
    * Tomorrow, Jun 22, 2022
    * 2.356
    * 2.332.4
    * 2.346.1
3. Announcement

## Notes :book:

* Done (https://github.com/jenkins-infra/helpdesk/milestone/21?closed=1)
  * [Maven 3.8.6 upgrade](https://github.com/jenkins-infra/helpdesk/issues/2986)
      * Switched tooling to use tags instead of GitHub releases
      * Using the packer image as definition point for Maven version
  * [Unable to sign up to accounts.jenkins.io](https://github.com/jenkins-infra/helpdesk/issues/3008)
      * Mark Waite helped
  * [Puppet: enable rotation of the apache error logs](https://github.com/jenkins-infra/helpdesk/issues/2969)
      * 10's of GB of logs not lingering now
  * [Renew update center certificate (crawler and update-center builds failing on trusted.ci.jenkins.io)](https://github.com/jenkins-infra/helpdesk/issues/2950)
      * Renewed by @dduportal, error report when checking plugin signature
      * Tool versions provided by crawler no longer had a valid signature
      * Proposed to add a health check for the crawler file signing
      * [JENKINS-66788](https://issues.jenkins.io/browse/JENKINS-66788) Certificate expired report from user but unclear if report is valid
  * [Auto notify people based on service routing rules](https://github.com/jenkins-infra/helpdesk/issues/2827)
  * [Enable timestamper for pipeline jobs](https://github.com/jenkins-infra/helpdesk/issues/2990)

* Work in Progress (https://github.com/jenkins-infra/helpdesk/milestone/21)
  * [[INFRA-3100] Migrate updates.jenkins.io to another Cloud](https://github.com/jenkins-infra/helpdesk/issues/2649)
      * @smerle working the issue, progressing
  * [[ci.jenkins.io] Digital Ocean sponsorship runs out of credits: stopping the cluster](https://github.com/jenkins-infra/helpdesk/issues/2917)
      * @lemeurherve and @dduportal met with DigitalOcean last week
      * DigitalOcean has donated $2700 additional
      * Granting Jenkins membership in their top tier of open source projects
  * [[ci.jenkins.io] Provides powershell on the Windows Docker images used on ACI](https://github.com/jenkins-infra/helpdesk/issues/2974)
      * @lemeurherve plans to install most recent powershell on all agents
          * Assure that prior powershell is also available
          * Likely need improvements to the documentation to use `pwsh`
  * [[JENKINS-49707] Evaluate retry conditions to improve the stability of the builds](https://github.com/jenkins-infra/helpdesk/issues/2984)
      * Jesse Glick has implemented a retry facility for infra outages
      * Will test drive it on ci.jenkins.io later this week
  * [enable Development integration in Jira](https://github.com/jenkins-infra/helpdesk/issues/2985)
      * Automatically link to GitHub PR's from inside Jira issues
      * Moved to next milestone
  * [Replace s390x Ubuntu 18.04 agent with s390x Ubuntu 20.04 agent](https://github.com/jenkins-infra/helpdesk/issues/2983)
      * Mark Waite create a new issue for ppc64le agent update
  * [DockerHub rate limiting](https://github.com/jenkins-infra/helpdesk/issues/2961)
      * Waiting for response from Docker
  * [Upgrade to Kubernetes 1.22](https://github.com/jenkins-infra/helpdesk/issues/2930)
      * No change since last week

* New/Importants [`infra-team-sync-next`](https://github.com/jenkins-infra/helpdesk/milestone/10)
  * [Migrate from workflow-cps-global-lib to pipeline-groovy-lib](https://github.com/jenkins-infra/helpdesk/issues/3011)
      * Install the new plugin
      * Remove the deprecated plugin
  * [[plugins.jenkins.io] Top sub-menus do not work](https://github.com/jenkins-infra/helpdesk/issues/3009)
      * Plugin site, likely needs Gavin Mogan
      * May be part of Google Summer of Code pipeline steps doc generator improvements
      * Moved issue to plugin-site repository
  * [weekly.ci's default view description diverged from the defined one](https://github.com/jenkins-infra/helpdesk/issues/2999)
      * Needs review of the configuration as code definition
      * Next week
      * Timestamper plugin not installed on weekly.ci.jenkins.io
          * Added to plugins.txt for Docker images
          * Plugin is installed on weekly.ci but is not active by default

* ToDo (next milestone) (https://github.com/jenkins-infra/helpdesk/milestone/22)

## Action Points :muscle:

* [x] Tomorrow is a good day to avoid infrastructure changes

<!-- How To: https://github.com/jenkins-infra/runbooks/tree/main/meetings -->
* [x] Publish this meeting note on [jenkins-infra/documentation](https://github.com/jenkins-infra/documentation)
* [x] Publish this meeting record on Youtube - https://youtu.be/VxqDIh1nqvw
* [x] Publish a topic in community.jenkin.io with the tag `meeting` - https://community.jenkins.io/t/infrastructure-team-meeting-june-21-2022/2863

Next meeting note - [2022-06-28](https://github.com/jenkins-infra/documentation/blob/main/meetings/2022-06-28.md)

repository: jenkins-infra/documentation

branch_name: refs/heads/main

repo_name: jenkins-infra/documentation


text:
# Basic sample for writing C/C++ GTest tests and running them on Android emulators / devices.

This project uses the Gradle build system. You don't need an IDE to build and execute it but Android Studio is recommended.

1. Download the project code, preferably using `git clone`.
2. In Android Studio, select *File* | *Open...* and point to the `./build.gradle` file.
3. Check out the relevant code:
   * The C++ sources and tests are in `src/main/cpp`
   * The device tests (which wrap the native tests) are in `src/androidTest/java`
4. Create and run the Instrumented test configuration
   * Open the `AdderTest` file, and click the run icon in the gutter, or
   * Manually create a configuration.
      * Open *Run* menu | *Edit Configurations*
      * Add a new *Android Instrumented Tests* configuration
      * Choose the `app` module
      * Connect a device or start an emulator
      * Run the newly created configuration

If you are using Android Studio, the *Run* window will show the test results.

repository: android/testing-samples

gha_language: Java

repo_name: android/testing-samples


text:
# `ngraph_jll.jl` (v0.0.1+0)

This is an autogenerated package constructed using [`BinaryBuilder.jl`](https://github.com/JuliaPackaging/BinaryBuilder.jl).

For more details about JLL packages and how to use them, see `BinaryBuilder.jl` [documentation](https://juliapackaging.github.io/BinaryBuilder.jl/dev/jll/).

## Sources

The tarballs for `ngraph_jll.jl` have been built from these sources:

* git repository: https://github.com/NervanaSystems/ngraph.git (revision: `81ca5be950bb62b97c5f6f96616c9de5b39ebf45`)
* git repository: https://github.com/Kitware/Cmake.git (revision: `dee2eff2cfb4d0743c5b2c3468e2b9227baff102`)

## Platforms

`ngraph_jll.jl` is available for the following platforms:

* `Linux(:x86_64, libc=:glibc)` (`x86_64-linux-gnu`)

## Dependencies

The following JLL packages are required by `ngraph_jll.jl`:

* `OpenSSL_jll)`

## Products

The code bindings within this package are autogenerated from the following `Products`:

* `LibraryProduct`: `libcodegen`
* `LibraryProduct`: `libcpu_backend`
* `LibraryProduct`: `libdnnl`
* `LibraryProduct`: `libeval_backend`
* `LibraryProduct`: `libinterpreter_backend`
* `LibraryProduct`: `libiomp5`
* `LibraryProduct`: `libmklml_intel`
* `LibraryProduct`: `libngraph`
* `LibraryProduct`: `libnop_backend`


text:
---
title: "subset_spec.proto"
weight: 5
---

<!-- Code generated by solo-kit. DO NOT EDIT. -->

### Package: `options.gloo.solo.io`
#### Types:

- [SubsetSpec](#subsetspec)
- [Selector](#selector)
- [Subset](#subset)

##### Enums:

	- [FallbackPolicy](#fallbackpolicy)

##### Source File: [github.com/solo-io/gloo/projects/gloo/api/v1/options/subset_spec.proto](https://github.com/solo-io/gloo/blob/main/projects/gloo/api/v1/options/subset_spec.proto)

---
### SubsetSpec

See envoy docs for details:
https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/cluster/v3/cluster.proto#config-cluster-v3-cluster-lbsubsetconfig

```yaml
"selectors": []options.gloo.solo.io.Selector
"fallbackPolicy": .options.gloo.solo.io.FallbackPolicy
"defaultSubset": .options.gloo.solo.io.Subset

```

| Field | Type | Description |
| ----- | ---- | ----------- |
| `selectors` | [[]options.gloo.solo.io.Selector](../subset_spec.proto.sk/#selector) | Defines the set of subsets of the upstream. |
| `fallbackPolicy` | [.options.gloo.solo.io.FallbackPolicy](../subset_spec.proto.sk/#fallbackpolicy) | The behavior used when no endpoint subset matches the selected route’s metadata The default value is ANY_ENDPOINT. |
| `defaultSubset` | [.options.gloo.solo.io.Subset](../subset_spec.proto.sk/#subset) | Specifies the default subset of endpoints used during fallback if fallback_policy is DEFAULT_SUBSET. |

---
### Selector

```yaml
"keys": []string
"singleHostPerSubset": bool

```

| Field | Type | Description |
| ----- | ---- | ----------- |
| `keys` | `[]string` | A subset is created for each unique combination of key and value. |
| `singleHostPerSubset` | `bool` | Selects a mode of operation in which each subset has only one host. Default is false. |

---
### Subset

```yaml
"values": map<string, string>

```

| Field | Type | Description |
| ----- | ---- | ----------- |
| `values` | `map<string, string>` | Any host that matches all key/value pairs is part of this subset. |

### FallbackPolicy

Description:

| Name | Description |
| ----- | ----------- |
| ANY_ENDPOINT | Any cluster endpoint may be returned (default) |
| DEFAULT_SUBSET | Load balancing over the endpoints matching the values from the default_subset field |
| NO_FALLBACK | A result equivalent to no healthy hosts is reported |

<!-- Start of HubSpot Embed Code -->
<script type="text/javascript" id="hs-script-loader" async defer src="//js.hs-scripts.com/5130874.js"></script>
<!-- End of HubSpot Embed Code -->

repository: solo-io/gloo

gha_language: Go

repo_name: solo-io/gloo


text:
# Amazon Linux 2 Container Image Pipeline with Example "hello world" Component Document

This is a sample template that demonstrates how to use EC2 Image Builder CloudFormation resources to build an Amazon Linux 2 Docker container image with a sample "hello world" component document and then publish the image to the specified Amazon Elastic Container Registry (ECR) repository.

***Internet connectivity is required in your default VPC*** to pull the source image by digest from a Docker Hub repository. If you do not have a default VPC, or want to use a custom VPC, you will need to specify a subnet ID and one or more security group IDs in the VPC as parameters when you create a stack based on this template.

## How this Stack Works

First, the stack will create an [AWS::S3::Bucket](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket.html) resource that is used to capture logs.

By default, AWS Services do not have permission to perform actions on your instances. So, the stack will create an [AWS::IAM::Role](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-iam-role.html) which grants AWS Systems Manager (SSM) and EC2 Image Builder the necessary permissions to build an image.

The instance also needs access to the bucket created by the stack, so a policy is added to the newly created role that allows the instance to use ```s3:PutObject``` to save logs to the logging bucket.

Then, an [AWS::IAM::InstanceProfile](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-iam-instanceprofile.html) is created, which passes the instance role to the EC2 instance.

Next, an [AWS::ImageBuilder::InfrastructureConfiguration](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-imagebuilder-infrastructureconfiguration.html) resource is created, and the Instance Profile is specified as one of its parameters. This parameter tells EC2 Image Builder to use the specified profile with the EC2 instance during the build.

The [AWS::ImageBuilder::ContainerRecipe](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-imagebuilder-containerrecipe.html) ties together the amazonlinux:latest parent image and the sample "hello world" component. This example demonstrates using the Dockerfile template default settings, which provides variables for your parent image, environments, and components. These variables will be replaced with Image Builder generated scripts at build time.

The resource [AWS::ImageBuilder::DistributionConfiguration](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-imagebuilder-distributionconfiguration.html) allows you to specify the name and description of your output container image and settings for tagging and sharing to a target ECR repository in a specific region.

The [AWS::ImageBuilder::Image](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-imagebuilder-image.html) represents the built image.

**Note:** *Optionally, you can uncomment the [AWS::ImageBuilder::ImagePipeline](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-imagebuilder-imagepipeline.html), which creates an automation pipeline for your container image builds. The pipeline is associated with the container recipe and can also be associated with an infrastructure configuration and distribution configuration. You can also use a schedule to configure how often and when a pipeline will automatically create a new image. In this example, the pipeline is scheduled to run a build at 9:00AM Coordinated Universal Time (UTC) on the first day of every month.*

## Walkthrough

It takes approximately 20 minutes for the stack build to complete.

This solution can be deployed using both the AWS Management Console or the Command Line Interface (CLI).

### AWS Management Console:
1. Upload the ```amazon-linux-2-container-image.yml``` template to CloudFormation.
2. You will see a checkbox informing you that the stack creates IAM resources. Read and check the box.
3. Wait for the stack to build.
4. Note the AWS::ImageBuilder::Image resource ```IBImage``` will show ```CREATE_IN_PROGRESS``` while the image is being created, and will later show ```CREATE_COMPLETE``` when complete.

### CLI:
1. Ensure that your YAML template and JSON parameters file are located within your current directory.
2. Modify the parameters in ```amazon-linux-2-container-image.json``` as necessary.
3. Run the following command from your terminal:
```
aws cloudformation create-stack \
--stack-name sample-ec2-ib-container-image \
--template-body file://amazon-linux-2-container-image.yml \
--parameters file://amazon-linux-2-container-image.json \
--capabilities CAPABILITY_NAMED_IAM \
--region us-east-1
```

## Troubleshooting

While the stack is building, you will see an EC2 instance running. This is either the build or test instance. AWS Systems Manager (SSM) Automation will also run. You can observe this automation to see the steps EC2 Image Builder takes to build your image.

If the stack fails, check the CloudFormation events. These events include a description of any failed resources.

## Cleanup

To delete the resources created by the stack:

1. Delete the contents of the S3 bucket created by the stack (if the bucket is not empty, the stack deletion will fail). To keep the bucket, add a ```Retain``` deletion policy to the CloudFormation bucket resource. See https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-deletionpolicy.html for more information on ```DeletionPolicy``` attributes.
2. Delete the container image within your ECR repository created by the stack (if the repo is not empty, the stack deletion will fail).
3. Delete the stack in the CloudFormation console, or by using the CLI/SDK.


text:
Assembly
========

### Common Functions and Code written in Assembly Language

- Intel(x86-32)
- ATT&T
- MIPS

### Compiling x86 on Mac OSX and FreeBSD

#### Steps:

- nasm -o hello.tmp -f macho hello.s
- ld -arch i386 -macosx_version_min 10.6 -no_pie -e _main -o hello.o hello.tmp
- ./hello.o

#### One-Liner:

- nasm -o hello.tmp -f macho hello.s && ld -arch i386 -macosx_version_min 10.6 -no_pie -e _main -o hello.o hello.tmp && ./hello.o

### Important:
##### Note: I use .tmp in order to go from (.s) - assembly -> (.tmp) - temporary -> (.o) - binary
##### Note: I also use "&&" in order to stop execution if anything goes wrong

#### For nasm:
- -o hello.tmp - outfile
- -f macho - specify format
	- Linux - elf or elf64
	- Mac OSX - macho

#### For ld:
- -arch i386 - specify architecture (32 bit assembly)
- -macosx_version_min 10.6 (Mac OSX - complains about default specification)
- -no_pie (Mac OSX - removes ld warning)
- -e main - specify main symbol name (Mac OSX - default is start)
- -o hello.o - outfile

#### For Shell:
- ./hello.o - execution

### Compilation Issues:

#### Mac OSX

"ld: warning: PIE disabled. Absolute addressing (perhaps -mdynamic-no-pic) not allowed in code signed PIE, but used in _start from hello.o. To fix this warning, don't compile with -mdynamic-no-pic or link with -Wl,-no_pie"
- Add the "-no_pie" option to "ld" when linking

"ld: warning: -macosx_version_min not specified, assuming 10.6"
- Add the "-macosx_version_min 10.6" option to "ld" when linking

#### Other

"ld: warning: option -m is obsolete and being ignored"
- Older option

"ld: warning: option -s is obsolete and being ignored"
- Older option

repository: vincentlao/assembly-example

repo_name: vincentlao/assembly-example


text:
# OT-2 Protocols

This repository contains customizable OT-2 protocols written in [OT-2 APIv2](https://docs.opentrons.com). These protocols are (hopefully) user-friendly to those who need more customizations than what the [Opentrons Protocol Designer](https://designer.opentrons.com/) can offer. They are also written with modularity in mind, and should serve as boilerplate scripts for more experienced OT-2 users who just need to tweak existing protocols for **medium** throughput process.

## How these protocols work

For most protocols you find here, the basic idea here is that, the information of *what* to transfer and *how* to transfer are separated. Consider the case of resuspending lyophilized oligonucleotides with water (aka primer resuspension):

* *what* = 20 tubes of primers, each tube receives a different volume water. (This changes every time)
* *how* = Transfer the water using the P300 single channel pipette. When dispensing, position the bottom of the tip 5 mm below the rim of the tube so that the same tip can be reused. (This never changes)

Therefore, every time we do primer resuspension we can reuse the protocol `primer_resuspension.py`. We only need to designate the *paths* and *volumes*, and also the *deck layout* and if necessary, the *pipette setup*, but we do not change the *behavior* of the pipetting steps.

### Working out transfer paths and volumes

When it comes to **paths** and **volumes**, say in the context of primer resuspension again, we think in terms of *what goes in what*:

1. "oligo001 resuspended with 324 µL of water",
2. "oligo002 resuspended with 275 µL of water", and etc.

But the API doesn't care about the actual identities of the samples. It only cares about *from where to where*:

1. "Transfer 324 µL from Slot 1, A1 (water) to Slot 2, A1 (primer tube)",
2. "Transfer 275 µL from Slot 1, A1 to Slot 2, A2"

The conversion from *what* to *where* is easy if you only have a few transfers, but it soon becomes mentally demanding as the number goes up. This is where the **InstructionWriter** comes into play. Instead of working out all the paths ourselves, we supply the InstructionWriter with an **intuitive Excel spreadsheet** stating what samples are in what wells, and where the liquids should go and the volumes. The InstructionWriter then process these information and write out parasable instructions, like:

```python
"324$1_A1->2_A1", # = move 302 µL from slot 1 well A1 to slot 2 well A1
"275$1_A1->2_A2"
# ... and the list goes on
```
which you can copy and paste inside a list in the protocol file. The protocol will then execute these instructions with a tested and establisehd *behavior*.

This feature is indispensable when we need to deal with medium-throughput work, e.g. combinatorial Golden Gate Assembly. It can also be applied to high-throughput, though, at that point we probably have to change the paradigm and set up the experiment differently.

## Quick Example

We will exmplify with a hypothetical scenario of setting up combinatorial Golden Gate Assemblies.

1. Install Python or Anaconda (recommended) and `pip install opentrons`.
2. Clone / Download this repository onto your computer
3. Check the contents of an Excel file under the folder `/instructions_io/` that is named `example_GGA.xlsx`
4. Close the Excel file (else `PermissionError` will raise)
5. Open an Anaconda terminal, navigate into the downloaded folder
6. Execute `>python -m InstructionWriter ./instructions_io/example_GGA.xlsx`
7. A file now appears under `/instructions_io/` that is called `example_GGA_instructions.txt`
8. Open this text file, copy all
9. Use a text editor (Spyder / Notepad), open `combinatorial_pipetting.py`
10. Paste the instructions under `inst_list = [` and before `]`

It should now look like:

```python
inst_list = [
'2$1_A2->2_A1',
'1.0$1_B1->2_A1',
'1$1_C2->2_A1',
'3$1_A3->2_A2',
'1.0$1_B1->2_A2',
'1$1_C2->2_A2',
'1$1_A1->2_A3',
'1.5$1_B2->2_A3',
#... many more lines here
]
```

11. Configure the deck layout and the pipette to be used as follow

```python
    slots_map = {
            '1':'opentrons_24_tuberack_eppendorf_1.5ml_safelock_snapcap',
            '2':'starlabpcrplateonws_96_wellplate_350ul'
            }

    # Configure tip racks and pipette

    r_pipette_name = 'p10_single'
    r_tiprack_slots = ['3']
    r_tiprack_name = 'geb_96_tiprack_10ul'
```

12. Make sure the right pipette (right mount) is used in the execution section

```python
for inst in inst_list:

    # Decodes information from each instruction line
    vol, path = inst.split('$')
    vol = float(vol)
    source, dest = path.split('->')
    source_slot, source_well = source.split('_')
    dest_slot, dest_well = dest.split('_')

    # Execution of transfer
    r_pipette.transfer(vol,
                       labware_items[source_slot].wells_by_name()[source_well],
                       labware_items[dest_slot].wells_by_name()[dest_well],
                       new_tip='always'
                       )
```

SaveLog: simulate the protocol and check whether the steps are correct.
Then save the log file for troubleshooting.
This function directly builds upon the Opentrons `simulate` module

13. Open an Anaconda terminal in this dowloaded folder
14. Execute `>python -m SaveLog combinatorial_pipetting.py`
15. Open the folder `protocol_log` and open the text file named `combinatorial_pipetting_log.txt` to check the simulated protocol

If your protocol employs any custom [labware definitions](https://support.opentrons.com/en/articles/3136504-creating-custom-labware-definitions), you need to put the custom labware definition files, in json formats, under the `./labware_def/` folder.

## The InstructionWriter and setting up the Excel files (short ver.)

The InstructionWriter takes an Excel file that adheres to certain formats, and does the job of converting human-readable instructions, e.g. "Transfer 5 µL of primer from tube primer001 to PCR tube", into machine-parsable, intermediate instructions with volume and path information, e.g. `5$1_A1->2_A1`. Users only need to specify the locations of source samples on one or more Excel spreadsheets, and what samples or liquids goes into destination wells in another Excel spreadsheets, and the InstructionWriter will perform the mapping. Then, these instructions can be directly copied and paste into a template protocol file, which will decode these intermediate instructions and feed them into the OT-2 APIv2 to execute the transfers.

The purpose of the InstructionWriter is offload the mentally demanding tasks of working out transfer paths for humans. It is **not** an OT-2 protocol writer.

The InstructionWriter also supports different layouts for destination formats.
Below a number of hypothetical cases are illustrated:

### intuitive
Example in patterning a plate in an human-trackable manner, and creating the characters "OT 2" on a 96-well plate:

This is an example of defining transfers from an `intuitive` layout of Opentrons 4-in-1 50 mL tube rack to another `intuitive` layout of a 96-well plate, as such it was designated so under the `slot_setup` sheet. The destination sheet format supports receiving multiple liquids, i.e. samples "a" and "b" and "c", shown on slot 2 well A1. Each sample were prefixed with a `+` sign to tell the InstructionWriter to add each component to that well. For `intuitive` format, InstructionWriter does not support assigning different volumes to different transfers. Rather, a `global_volume` is given on the `slot_setup` sheet so every sample will receive 200 µL. That would of course means that slot 2 well A1 will receive 600 µL and will overflow. Volume handling is beyond the scope of the InstructionWriter.

### df_variable_volume
Example in primer resuspension:

### df_variable_sample
Example in setting up multiple PCR reactions during cloning. Volumes of primers and templates are fixed at 5 µL and 1 µL, respectively, but the identity of primers and templates vary from one destination PCR tube to another:

### df_variable_sample_n_volume
A most customizable table for combinatorial pipetting. The following example illustrates how it can be used for Gibson Assembly involving different constructs:

### Further instructions on using the InstructionWriter

- You can find templates for these layouts under `/instructions_io/_layout_templates.xlsx`
- For a complete guide of how to set up the Excel file, see [`Manual_on_InstructionWriter.pdf`](./Manual_on_InstructionWriter.pdf).
- The InstructionWriter can be called in a script using
```python
import InstructionWriter
instructions = InstructionWriter.write_instructions("your_excel_file.xlsx")
```

### Applications
Protocols I performed using the InstructionWriter:
- Mixing different plasmids for co-transformation (intuitive -> intuitive)
- Standardizing cell mass for whole-cell lysates preparation for Western Blot analysis (intuitive -> df_variable_sample_n_volume)
- Cherry picking strains that showed desirable traits when screening libraries of mutated variants (dataframe -> df_variable_sample)
- Moving PCR products from 96-well PCR plate into DNA purification columns (dataframe -> df_variable_sample)
- Migrating column purified DNA back to 96-well PCR for submission of Sanger sequencing (dataframe -> df_variable_sample)
- Moving bacterial cultures from 96-well deep plate into microcentrifuge tubes for saving glycerol stocks (intuitive -> intuitive)

### Feature and Advantage
1. The InstructionWriter allows comments to be inserted on the Excel spreadsheet so that users can write "metadata" regarding the source or desination samples. This is illustrated in the 3 examples above.
2. If there is an accident and you need to restart a protocol. It is easy to comment out the instructions that were done. If you use the many-to-many `pipette.transfer()` function from the API. You will need to comment out items on at least 2 separate lists.

### Limitations
1. The InstructionWriter mandates that each slot can only be either a "source" slot or a "destination" slot. It also does not accept a source sample to appear twice on the deck.
2. The characters `+`, `$` and `_` are forbidden characters for sample names.
3. The string "Unnamed:" is forbidden for column headers.
4. The characters `#` and `.` are discouraged for sample names.
5. The InstructionWriter does not know what the actual labware is, and will not figure out whether too much or too little volumes are in the source/destination wells.
6. The InstructionWriter will **not** work out how to add the samples most efficiently by doing one-aspirate-many-dispense actions. It was not designed to handle this problem. Users who wish to seed many wells on the plate with water should rely on programming through the API directly.
7. It takes a lot more time to compile for loops on instructions than to compile a  many-to-many `pipette.transfer()` on the OT-2's Raspberry Pi, resulting in longer time for the protocol to be loaded.

## Descriptions on Boilerplate Protocols
Protocols deposited are briefly described below. Please refer to the docstrings for detail descriptions.

| Protocol filename             | Requires InstructionWriter | Description                                                                                |
|-------------------------------|----------------------------|----------------------------------------------------------------------------------------------------------------------------------------|
| combinatorial_pipetting.py    | Yes                        | Basic but versatile protocol that handles most complex pipetting needs                                                                 |
| combinatorial_pipetting_al.py | Yes                        | A variant protocol that handles pipetting samples with high alcohol contents                                                           |
| colony_plating.py             | No                         | Plate transformed bacterial onto rectangular one-well agar plate                                                                       |
| plate_induction_2con.py       | No                         | Dilute overnight grown bacterial cultures from a 96-well plate into 2 different 96-well plates, one with inducer and the other without |
| primer_rehydration.py         | Yes                        | Add water to lyophilized oligonucleotides                                                                                |
| primer_dilution.py            | Yes                        | Dilute 100 µM of resuspended oligonucleotides from screw-cap tubes to 10 µM in snap-cap microcentrifuge tubes                          |

## Questions/Issues/Suggestions
Please contact me through email since I do not actively or passively monitor my GitHub. Thank you.

## License
Licensed under MIT license


text:
dropDrive
=========

- `dropdrive-cmd` is simple java command line application for uploading file(s) to Dropbox.
- `dropdrive-core` is core library which provide ability for uploading file(s) to Dropbox.

dropDrive as command line application
-------------------------------------

### Usage
usage: `dropdrive [-a <code>] [-d <dir>] [-f <file>] [-h] [-l] [-p <props>] [-v]`

 `-a <code>` - process authorization

 `-d <dir>` - directory for upload; creates new one if no directory exists; default is dropdrive-uploads

 `-f <file>` - upload files

 `-h` - show this help

 `-l` - display authorization link

 `-p <props>` - path to dropDrive properties file

 `-v` -show dropDrive version

### How to authorize application
1. generate authorization URL:

   `dropdrive -p dropdrive.properties -l`
2. copy and paste URL to your browser to receive authorization code
3. authorize application with received authorization code:

   `dropdrive -p dropdrive.properties -a <code>`

### How to upload files
Upload file(s) to Dropbox:

   `dropdrive -p dropdrive.properties -f <file1> <file2> <file3>`

Files are uploaded to `dropdrive-uploads` directory by default.

If you want to change upload directory:

- change `dropdrive.uploadDir` property in _properties_ file
- or pass directory in `-d <dir>` argument:

   `dropdrive -p dropdrive.properties -f <file> -d <path>/<to>/<dir>`

### How to use properties file
- `dropdrive.accessToken` - Dropbox access token; this property is updated automatically by dropDrive
- `dropdrive.uploadDir` - path to dir where files will be uploaded: `<path>/<to>/<dir>`

dropDrive as core library
-------------------------
dropDrive core can be used in any other application to provide ability for uploading file(s) to Dropbox.

### Usage
1. add implementation of `IDropDriveCredential` interface to spring context
   or use `SimpleDropDriveCredential` or extend `AbstractDropDriveCredential`
2. provide client key as `dropdrive.core.drive.clientKey` property in spring context
3. provide client secret as `dropdrive.core.drive.clientSecret` property in spring context
4. import dropDrive spring context with `<import resource="classpath:dropdrive-core-context.xml" />` or using annotation `@Import(net.czpilar.dropdrive.core.context.DropDriveCoreContext.class)`
5. autowire `IFileService` and use file uploading methods

License
=======

    Copyright 2019 David Pilar

    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.

repository: czpilar/dropdrive

repo_name: czpilar/dropdrive


text:
# TiffLibrary

A C# library for decoding and encoding images from and to TIFF files.

[![Build Status](https://dev.azure.com/jinyi0679/yigolden/_apis/build/status/yigolden.TiffLibrary?branchName=master)](https://dev.azure.com/jinyi0679/yigolden/_build/latest?definitionId=1&branchName=master)

## Features

* Implemented completely in C# code. No native dependencies.
* Runs on asynchronous APIs. Suitable for server-side applications.
* Supports both standard TIFF and BigTIFF format.
* Supports both stripped TIFF and tiled TIFF.
* Capable of reading and writing TIFF file of massive size.

## Supported Runtimes

* .NET Framework 4.6.1+
* .NET Core 3.1+
* Runtimes compatible with .NET Standard 2.0

## Getting Started

### **Install TiffLibrary Package**

Install the latest version from NuGet.

NuGet: [![NuGet](https://img.shields.io/nuget/v/TiffLibrary.svg)](https://www.nuget.org/packages/TiffLibrary/)

```
dotnet add package TiffLibrary --version <VERSION>
```

Add the following using statement to your source files.
``` csharp
using TiffLibrary;
```

### **Opens a TIFF File for Reading**

Open a TIFF file using its file name:
``` csharp
using TiffFileReader tiff = await TiffFileReader.OpenAsync(@"C:\Data\test.tif");
```

Read from an existing `Stream`:
``` csharp
Stream stream = GetStream();
using TiffFileReader tiff = await TiffFileReader.OpenAsync(stream, leaveOpen: false);
```

Read from byte buffer:
``` csharp
byte[] buffer = GetBuffer();
using TiffFileReader tiff = await TiffFileReader.OpenAsync(buffer, 0, buffer.Length);
```

### **Read Image File Directory and Its Entries**

An image file directory (IFD) is a structure in TIFF file. It contains information about the image, as well as pointers to the actual image data. A single TIFF file can contains multiple IFDs (multiple images). They are stored in the TIFF file like a linked-list. Each IFD contains an pointer to the next, while the TIFF file header contains an pointer to the first IFD.

The following example shows how to read the first IFD and its entries.
``` csharp
using TiffFileReader tiff = await TiffFileReader.OpenAsync(@"C:\Data\test.tif");

// ReadImageFileDirectoryAsync method with no parameter will read the list of all the entries in the first IFD.
TiffImageFileDirectory ifd = await tiff.ReadImageFileDirectoryAsync();

// Tests whether an entry with the specified tag exists
bool exists = ifd.Contains(TiffTag.RowsPerStrip);

// Gets information of an entry.
TiffImageFileDirectoryEntry entry = ifd.FindEntry(TiffTag.RowsPerStrip);
TiffFieldType fieldType = entry.Type; // The data type of the values. In this case, it can be TiffFieldType.Short.
long count = entry.ValueCount; // Each entry contains an array of elements of the specific type. ValueCount is the length of the array.
// To read actual values from the entry, TiffFieldReader should be used, which will be covered later in this documentation.
```

The following example shows how to enumerate all the IFDs.
``` csharp
using TiffFileReader tiff = await TiffFileReader.OpenAsync(@"C:\Data\test.tif");

TiffStreamOffset ifdOffset = tiff.FirstImageFileDirectoryOffset;
while (!ifdOffset.IsZero)
{
    // Read the list of entries in this IFD.
    TiffImageFileDirectory ifd = await tiff.ReadImageFileDirectoryAsync(ifdOffset);

    // Operations on the IFD goes here.

    // Move to the next IFD.
    ifdOffset = ifd.NextOffset;
}
```

### **Read the Value of an IFD Entry**

To read actual values from the IFD Entry, a `TiffFieldReader` instance should be created. The following example shows how to read value from the specified tag.
``` csharp
using TiffFileReader tiff = await TiffFileReader.OpenAsync(@"C:\Data\test.tif");
using TiffFieldReader fieldReader = await tiff.CreateFieldReaderAsync();
TiffImageFileDirectory ifd = await tiff.ReadImageFileDirectoryAsync();

// Read the values of BitsPerSample tag.
TiffImageFileDirectoryEntry entry = ifd.FindEntry(TiffTag.BitsPerSample);
TiffValueCollection<ushort> bitsPerSample = await fieldReader.ReadShortFieldAsync(entry);
int count = bitsPerSample.Count;
ushort bitsPerSample0 = bitsPerSample.GetFirstOrDefault(); // or bitsPerSample[0]
ushort bitsPerSample1 = bitsPerSample[1];
ushort bitsPerSample2 = bitsPerSample[2];

// Alternatively, you can use the TiffTagReader helper if you are trying to read well-known tags.
TiffTagReader tagReader = new TiffTagReader(fieldReader, ifd);
TiffValueCollection<ushort> bitsPerSample = await tagReader.ReadBitsPerSampleAsync();

// TiffValueCollection<T> is a array-like structure for optimizing memory usage. It is allocation-free if the value collection contains no or only one elements.
```

Note that when the `TiffFieldReader` is created, a `TiffFileContentReader` instance is created (along with a `Stream` instance if we are reading from file) and kept alive in the life time of `TiffFieldReader` instance. As a result, to save system resources, reuse the `TiffFieldReader` instance if possible or cache the values read from the IFD so that `TiffFieldReader` is not created over and over again.

### **Decode an Image from an IFD**

Each IFD contains at most one image. The following code decode the image from the first IFD into an array.

``` csharp
using TiffLibrary.PixelFormats;

using TiffFileReader tiff = await TiffFileReader.OpenAsync(stream, leaveOpen: false);
TiffImageFileDirectory ifd = await tiff.ReadImageFileDirectoryAsync();

// Create the decoder for the specified IFD.
TiffImageDecoder decoder = await tiff.CreateImageDecoderAsync(ifd);

// Create an array to store the pixels
TiffRgba32[] pixels = new TiffRgba32[decoder.Width * decoder.Height];
TiffMemoryPixelBuffer<TiffRgba32> pixelBuffer = new TiffMemoryPixelBuffer<TiffRgba32>(pixels, decoder.Width, decoder.Height, writable: true);

// Decode the image. Note that this call will create a `Stream` instance for reading pixel data if we are reading from file. It will be disposed before DecodeAsync completes.
await decoder.DecodeAsync(pixelBuffer);
```
When creating `TiffImageDecoder` instance, `TiffFieldReader` is used. Therefore, reuse the `TiffImageDecoder` instance if possible to save system resources.

For massive-sized TIFF, you may not allocate enough buffer for the entire image at once. You may want to read a sub-region from the entire region. The following code show how to do that.
``` csharp
...

TiffImageDecoder decoder = await tiff.CreateImageDecoderAsync(ifd);

// Buffer for a sub-region.
TiffRgba32[] pixels = new TiffRgba32[1024 * 1024];
TiffMemoryPixelBuffer<TiffRgba32> pixelBuffer = new TiffMemoryPixelBuffer<TiffRgba32>(pixels, 1024, 1024, writable: true);

// Read a sub-region starting from (2048, 2048).
await decoder.DecodeAsync(new TiffPoint(2048, 2048), pixelBuffer);
```

If you want to decode the image into pixel buffer other than `Memory<TPixel>` (eg. An `Image<Rgba32>` instance from ImageSharp library), you can refer to **[Working with Pixel Buffer](./Documentation/working-with-pixel-buffer.md)** page in advanced topics.

For other usages of `TiffImageDecoder`, see **[Using Image Decoder APIs](./Documentation/using-image-decoder-apis.md)** in advanced topics.

### **Encode an Image into TIFF file**

The following example shows how to encode an RGBA image into the first IFD of an TIFF file.

``` csharp
TiffRgba32[] pixels = new TiffRgba32[1024 * 1024];

// Operations to fill pixels.
...

// Build The image encoder.
var builder = new TiffImageEncoderBuilder();
builder.PhotometricInterpretation = TiffPhotometricInterpretation.RGB;
builder.EnableTransparencyForRgb = true;
builder.IsTiled = true;
builder.TileSize = new TiffSize(256, 256);
builder.Compression = TiffCompression.Deflate;
builder.ApplyPredictor = TiffPredictor.HorizontalDifferencing;
TiffImageEncoder<TiffRgba32> encoder = builder.Build<TiffRgba32>(); // The encoder instance can be reused.

// The pixel buffer to read from.
var pixelBuffer = new TiffMemoryPixelBuffer<TiffRgba32>(pixels, 1024, 1024, writable: false);

// Opens the file for writing.
using var writer = await TiffFileWriter.OpenAsync(@"C:\Data\test2.tif", useBigTiff: false);

// Encode the image into an IFD.
TiffStreamOffset ifdOffset;
using (var ifdWriter = writer.CreateImageFileDirectory())
{
    await encoder.EncodeAsync(ifdWriter, pixelBuffer);

    // Write other properties here (eg, XResolution, YResolution)
    await ifdWriter.WriteTagAsync(TiffTag.XResolution, TiffValueCollection.Single(new TiffRational(96, 1)));
    await ifdWriter.WriteTagAsync(TiffTag.YResolution, TiffValueCollection.Single(new TiffRational(96, 1)));
    await ifdWriter.WriteTagAsync(TiffTag.ResolutionUnit, TiffValueCollection.Single((ushort)TiffResolutionUnit.Inch));

    ifdOffset = await ifdWriter.FlushAsync();
}

// Set this IFD to be the first IFD and flush TIFF file header.
writer.SetFirstImageFileDirectoryOffset(ifdOffset);
await writer.FlushAsync();
```

## Adapter for SixLabors.ImageSharp

This project adds TIFF decoding and encoding supports into [SixLabors.ImageSharp](https://github.com/SixLabors/ImageSharp) library.

Add the following code to the start up process of your application (eg. Main method) to add TIFF support to SixLabors.ImageSharp.
``` csharp
SixLabors.ImageSharp.Configuration.Default.Configure(new TiffLibrary.ImageSharpAdapter.TiffConfigurationModule());
```

Now you should be able to decode or encode TIFF files using ImageSharp APIs.

``` csharp
using (Image<Rgba32> image = Image.Load<Rgba32>(@"C:\Data\test.tif"))
{
    image.Save(@"C:\Data\test2.tif");
    // Or
    image.Save(@"C:\Data\test2.tif", new TiffEncoder
    {
        PhotometricInterpretation = TiffPhotometricInterpretation.RGB,
        EnableTransparencyForRgb = true,
        Compression = TiffCompression.Deflate,
        DeflateCompressionLevel = TiffDeflateCompressionLevel.Default,
        Predictor = TiffPredictor.HorizontalDifferencing
    });
}
```

## Advanced Topics

* [Supported TIFF Features](./Documentation/supported-tiff-features.md)
* [Working with Pixel Buffer](./Documentation/working-with-pixel-buffer.md)
* [Using Image Decoder APIs](./Documentation/using-image-decoder-apis.md)

repository: yigolden/TiffLibrary

gha_language: C#

repo_name: yigolden/TiffLibrary