text:
## 1. Leo Overview

The [VINCI](http://www.hsrd.research.va.gov/for_researchers/vinci/)-developed Natural Language Processing (NLP) infrastructure, is a set of services and libraries that facilitate the rapid creation and deployment of Apache UIMA-AS annotators focused on NLP. Leo is separated into three main components; Leo-client, Leo-core, and Leo-service.

**leo-client** contains the components that send requests to the services. Setting up a client is as simple as picking the collection reader and listener required.

**leo-core** contains the core tools that have been developed in conjunction with Leo to facilitate various NLP and annotation needs.

**leo-service** contains the core server functionality for launching UIMA_AS services. All three packages can be found together in leo-base.

Leo also contains a collection of packages that provide additional functionalities for creating custom pipelines for distributed NLP systems. The Basic architecture of Leo is shown in the following image with the flow beginning at the reader.

<img src="images/IntroFigure1.png" alt="" style="width: 750px;"/>

###Documentation Overview
The documentation is separated into two main sections; in the first sections of this guide you'll find a description of Leo as well as associated terminology and concepts. The next section goes into detail about setting up your computer, installing and configuring everything needed in order to run a sample Leo Project. Once the Leo project is up and running, you will find a description of the steps needed in order to adapt Leo to your specific project needs. For specific questions about the use of various classes and methods within Leo see the Javadoc page.

If you're already familiar with UIMA and Maven, you can go to the Download page [here](download.html) for the POM dependencies required.
A list of relevant tools and packages used by Leo can be found in the [Components and Tools](components.html) page.

###1.1 Leo-Client
The client is used to define the readers and listeners needed to execute a pipeline via the client API for UIMA_AS. In Leo-client will initialize and execute a pipeline via the client API for UIMA AS which has been created via the Service object. Setting up a client is as simple as picking the collection reader and listener required; plugging in the input directory you're using, and specifying the output listener that you'd like to use. An example of the code requiring your input can be found in the "Understanding and Adapting the Leo-Example Project" section of the User-guide.

#### CollectionReaders: Fountains of knowledge

UIMA-AS services analyze and tag data stored in CAS (Common Analysis Structure) objects.
CAS objects are sent from the client to the service after the client has populated the CAS
with document text.  CollectionReaders populate the document text in each CAS object.  The
source of the document text varies between readers.  Some common sources include text
files and relational database tables.

---

##### Leo Readers

Leo provides complete CollectionReaders for the some of the most common data sources. The readers that are included in the core leo packages are the `BatchDatabaseCollectionReader`, `DatabaseCollectionReader`, `SQLServerPagedDatabaseCollectionReader`, `FileCollectionReader`, and `XmiFileCollectionReader`. There are also other readers not included in the core leo packages for some common specific formats such as Knowtator or Siman pre-annotated document information.

##### Complete Leo Readers

The `DatabaseCollectionReader` pulls in one record per row of query results. The standard java database connection information must be provided such as the jdbc driver, url, username, password, and the query to execute. The `ID` and `Note` column names are also required so that the reader knows from which fields in the query to pull the record ID and document text.  See the javadocs for more information on expected parameters.

```java
DatabaseCollectionReader reader = new DatabaseCollectionReader("com.mysql.jdbc.Driver", "jdbc:mysql://localhost/test",
                                 "testuser", "password", "select id, document from example_document", "id", "document");
```

The `BatchDatabaseCollectionReader` extends the `DatabaseCollectionReader`. It allows the user to pull the data in batches based on some sequential identifier, such as a row number field in the table. This is handy for large tables as well as environments with less reliable database connections. The information required to initialize the `BatchDatabaseCollectionReader` is the same as for the `DatabaseCollectionReader` with the addition of the minimum and maximum identifier fields and a batch size field. The query can optionally include a `{min}` and `{max}` tag set which will be replaced by the minimum and maximum record identifiers of each batch as in the following code:

```java
BatchDatabaseCollectionReader reader = new BatchDatabaseCollectionReader("com.mysql.jdbc.Driver", "jdbc:mysql://localhost/test",
                                      "testuser", "password",
                                      "select id, document from example_document where rowno >= {min} and rowno < {max}",
                                      "id", "document", 0, 100, 10);
```

The `SQLServerPagedDatabaseCollectionReader` extends the `DatabaseCollectionReader`. It allows the user to pull table data in batches from a MS SQL Server table. Parameters include a first row offset for processing and the number of records to pull in each batch or page size as in the following code:

```java
SQLServerPagedDatabaseCollectionReader reader = new SQLServerPagedDatabaseCollectionReader("com.microsoft.sqlserver.jdbc.SQLServerDriver",
                                                "jdbc:mysql://localhost/test",
                                                "testuser", "password",
                                                "select id, document from example_document order by id",
                                                "id", "document", 50000, 0);
```

The `FileCollectionReader` reads a collection of files from an input directory. The reader can be flagged to recurse into sub-directories. By default the reader will only process files with a .txt extension however a custom file extension filter can also be specified. See the javadocs for more information.

```java
FileCollectionReader reader = new FileCollectionReader(new File("data/input"), false);
```

The `XmiFileCollectionReader` is a special case of a `FileCollectionReader` which reads in the IBM xmi output format for CAS serialization. The `XmiFileCollectionReader` does not allow the user to specify a custom extension, just the input directory and recurse flag.

```java
XmiFileCollectionReader reader = new XmiFileCollectionReader(new File("data/xmi-input"), false);
```

##### Base Leo Readers

Leo also provides Base versions of common types of CollectionReaders so that custom implementations are easier to create. The base readers are the `BaseDatabaseCollectionReader` and `BaseFileCollectionReader`. Each of these readers extend the `BaseLeoCollectionReader` as well as providing most of the functionality required to implement a reader of that type. In each case the only required abstract method is the `getNext(CAS aCas)` method for supplying the actual document text. For more information see the Javadocs of the base reader implementations.

---

##### BaseLeoCollectionReader

Leo provides an abstract base object that implements the most common aspects of the UIMA CollectionReader API called the BaseLeoCollectionReader. There are some methods however that are required by the API for the purpose of communicating the progress through a collection of data as well as the action of retrieving the data. These methods must be implemented by extending classes.

`hasNext()` returns a boolean which when true indicates that there is more data for processing in the collection.

```java
@Override
public boolean hasNext() throws IOException, CollectionException {
    return currentString < numberOfStrings;
}
```

`getNext(CAS aCAS)` populates the document text in the CAS object provided.

```java
@Override
public void getNext(CAS aCAS) throws CollectionException, IOException {
    int length = random.nextInt(MAX_STRING_LENGTH);
    aCAS.setDocumentText(RandomStringUtils.randomAlphanumeric(length));
    currentString++;
}
```

`Progress[] getProgress()` returns an array of UIMA Progress objects, typically with only one element, which record the current record index relative to the size of the data collection.

```java
@Override
public Progress[] getProgress() {
    return new Progress[]{ new ProgressImpl(currentString, numberOfStrings, Progress.ENTITIES) };
}
```

`CollectionReader produceCollectionReader()` creates a new CollectionReaderDescription by creating parameters from class variables marked with the @LeoConfigurationParameter annotation.  Values from those variables are propagated into the descriptor and then a new CollectionReader instance is produced from the descriptor that is created.

Here is the code for a complete CollectionReader implementation that extends the BaseLeoCollectionReader.

```java
/**
 * This reader populates a specified number of CAS objects with random strings. Useful primarily as an example of the
 * implementation of the BaseLeoCollectionReader API.
 *
 * User: Thomas Ginter
 * Date: 10/27/14
 * Time: 09:15
 */
public class RandomStringCollectionReader extends BaseLeoCollectionReader {

    /**
     * Number of strings produced so far.
     */
    protected int currentString = 0;
    /**
     * Number of strings that the reader will create.
     */
    @LeoConfigurationParameter(mandatory = true)
    protected int numberOfStrings = 0;
    /**
     * Generates a random length for each string.
     */
    protected Random random = new Random(System.currentTimeMillis());
    /**
     * Maximum length of the generated string.
     */
    protected static final int MAX_STRING_LENGTH = 1024;

    public RandomStringCollectionReader(int numberOfStrings) {
        this.numberOfStrings = numberOfStrings;
    }

    /**
     * @param aCAS the CAS to populate with the next document;
     * @throws org.apache.uima.collection.CollectionException if there is a problem getting the next and populating the CAS.
     * @see LeoCollectionReaderInterface#getNext(org.apache.uima.cas.CAS)
     */
    @Override
    public void getNext(CAS aCAS) throws CollectionException, IOException {
        int length = random.nextInt(MAX_STRING_LENGTH);
        aCAS.setDocumentText(RandomStringUtils.randomAlphanumeric(length));
        currentString++;
    }

    /**
     * @return true if and only if there are more elements available from this CollectionReader.
     * @throws java.io.IOException
     * @throws org.apache.uima.collection.CollectionException
     * @see LeoCollectionReaderInterface#hasNext()
     */
    @Override
    public boolean hasNext() throws IOException, CollectionException {
        return currentString < numberOfStrings;
    }

    /**
     * Gets information about the number of entities and/or amount of data that has been read from
     * this <code>CollectionReader</code>, and the total amount that remains (if that information
     * is available).
     * <p/>
     * This method returns an array of <code>Progress</code> objects so that results can be reported
     * using different units. For example, the CollectionReader could report progress in terms of the
     * number of documents that have been read and also in terms of the number of bytes that have been
     * read. In many cases, it will be sufficient to return just one <code>Progress</code> object.
     *
     * @return an array of <code>Progress</code> objects. Each object may have different units (for
     * example number of entities or bytes).
     */
    @Override
    public Progress[] getProgress() {
        return new Progress[]{ new ProgressImpl(currentString, numberOfStrings, Progress.ENTITIES) };
    }
}

```

---

##### Third Party Readers

Third party CollectionReaders can be used as easily as LeoCollectionReaders using the ExternalCollectionReader object. The ExternalCollectionReader can be initialized using either an object that extends the CollectionReader_ImplBase or a File object that references the CollectionReader descriptor as in the following examples:

```java
//From reader object
CollectionReader myReader = new MyReader(param1, param2);
ExternalCollectionReader myExternalReader = new ExternalCollectionReader(myReader);

//From descriptor file
ExternalCollectionReader externalReader = new ExternalCollectionReader(new File("path/to/readerDescriptor.xml"));
```

* * *

####Listeners: What was that?

##### UIMA Listeners

UIMA-AS clients send data to services populated in CAS (Common Analysis Structure) objects by CollectionReaders. After the services have finished processing the text in the CAS it is sent back to the client. The client then passes the CAS to one or more registered listener objects. Listeners receive several different events from the client framework during the course of sending and receiving CAS objects to/from the services. The event that receives the completed CAS object back from the service is called the entityProcessComplete event. In this event we receive a reference handle to the CAS as well as a status message. From this event listeners can iterate through the CAS processing results for storage or further processing. In this way CollectionReaders are the source of data input and Listeners are the mechanism for output.
For more information on UIMA listeners check out the documentation found on the [UIMA Site](http://uima.apache.org)

---

##### Leo Listeners

Leo provides two completed listeners and three base listeners for users to extend to create their own listener. The complete listeners are basic examples of listener implementations called the `SimpleCsvListener` and `SimpleXmiListener`.

The `SimpleXmiListener` extends the `BaseListener` class to dump CAS objects to UIMA's XMI format. XMI is essentially an XML representation of the CAS that can be stored and then loaded later for additional processing. CAS output can be filtered by setting the type names in the `annotationTypeFilter`. When types have been added to the filter array then only CAS objects containing one or more of the filter types will be written to the output directory.

The `SimpleCsvListener` writes the annotations found in each CAS to an output CSV file. The user can choose whether or not to include features in the output though by default they are excluded.

---

#####  Base Leo Listeners

There are three base listeners in the Leo client module. They are the `BaseListener`, `BaseCsvListener`, and `BaseDatabaseListener`. The `BaseListener` provides the core listener functionality and will be discussed in the section titled "Create a Custom Listener". The `BaseCsvListener` and `BaseDatabaseListener` extend the `BaseListener` to add specialized functions.

The `BaseCsvListener` provides the capability to write out CSV output with custom separators. Extending classes can set the header data by overriding the `String[] getHeaders()` method to provide the custom header list. The `writeHeaders()` method can then be called by either the client or as part of an initialization event in the extending listener. This method should only be called once as it is used to write the header row to the output CSV file. Extending classes must also implement the `List<String[]> getRows(CAS cas)` method. Each `String[]` in the list that is returned represents a row of data that will be written to the output CSV file.

The `BaseDatabaseListener` provides the ability to write rows to an output database table. The output table can be created by calling one of the public `createTable` methods provided by the base listener. Extending classes must override the `List<Object[]> getRows(CAS aCas)` method. Each `Object[]` represents a row of data to be added to the table. Rows are added as a batch whose size is configurable. The appropriate batch size will vary depending on the database environment being used. If inserts are taking an inordinate amount of time then the batch size may be too large, however if your program execution is delayed because there are too many batches being created then the batch size should be increased.

---

##### Create a Custom Listener

Creating a custom listener other than the ones already listed the user can extend the `BaseListener` class. Extending classes should call `super.entityProcessComplete(aCas, aStatus)` in the overriding `void entityProcessComplete(CAS aCas, EntityProcessStatus aStatus)` method. Doing so allows the `BaseListener` to screen the status object for errors and keep track of the counts of CAS objects sent and received to/from the services. The annotationTypeFilter can also be accessed from the `BaseListener` class for basic output filtering. A good example of an extending class to the `BaseListener` can be found in the `SimpleXmiListener.java` source code.

---

#####Third Party Listeners

The Leo `Client` class will register any listener object that extends the UIMA `UimaAsBaseCallbackListener`object. This allows the Leo `Client` to use any third party listener in addition to listeners that extend the `BaseListener`.

***

###1.2 Leo-Service

The Leo Service class can be used to define type system and annotators for the pipeline programmatically. The Service class deploys the UIMA AS service from the list of primitive descriptor files or Annotator objects provided by the caller. Any modules that will be executed must be in the class path of the Server process, remote modules being the exception once those are implemented. Setting up the Service, like most aspects of Leo, is as simple as editing a few directories to your specific files.

####Analysis Engines

**LeoBaseAnnotator:**

This is a base annotator class for holding functions that all annotators might take advantage of, such as loading resources files, initializing parameters etc...

**LeoCachingAnnotator:**

A base annotator with methods that implement caching via EHCache.

#### Creating an annotator

Leo is geared towards processing machine-readable text. A base annotator class `gov.va.vinci.leo.ae.LeoBaseAnnotator` (which extends `org.apache.uima.analysis_component.JCasAnnotator_ImplBase` ) is provided to simplify and speed up development of an Analysis Engine for text processing.

In order to develop an annotator, write a new class that extends gov.va.vinci.leo.ae.LeoBaseAnnotator. Each AE has to have two main steps – initializing the AE parameters, and processing each document. When a service is deployed (see Service deployment methods section), for each annotator and AE object is created and initialized. When a document reaches the specific annotator in the pipeline, the document JCas is passed to the annotate method.

```java
import gov.va.vinci.leo.ae.LeoBaseAnnotator;
import gov.va.vinci.leo.descriptors.LeoTypeSystemDescription;
import org.apache.uima.analysis_engine.AnalysisEngineProcessException;
import org.apache.uima.jcas.JCas;

public class myAnnotator extends LeoBaseAnnotator {

	@LeoConfigurationParameter
	protected String myStringParam = null;

	@Override
	public void initialize(UimaContext aContext) throws ResourceInitializationException {
		super.initialize(aContext);
		< additional initialization code >
	}

	@Override
	public void annotate(JCas aJCas) throws AnalysisEngineProcessException {
		< additional processing code >
	}
}
```

Annotator parameters are identified with the @LeoConfigurationParameter annotation.  The LeoBaseAnnotator will extract parameter values set in an annotator object and propagate them to the descriptor. In addition values set in the descriptor will be initialized in the annotator when the pipeline is initialized. You should write setter and getter methods for each of the annotator parameters that you wish to set in creating Leo pipelines.

Depending on the needs of your annotator, you may use the default initialize method to validate values or perform additional initialization:

```
initialize(org.apache.uima.UimaContext aContext);
```

When overriding the initialize(UimaContext aContext) method super.initialize(aContext) must be called in order for the base annotator values extraction to function properly.

When extending the base annotator, the only method required to overwrite is `annotate(org.apache.uima.jcas.JCas aJCas)`
The CAS object that is passed by the system contains getDocumentText() method that gives access to the actual text of the document.

---
###1.4 Leo-Core
The Leo-Core package contains the tools, descriptors, types, and extra functionalities of Leo that are helpful for individualized tasks.

####Descriptors
Descriptors are used to define the annotators that make up a service as well as the type system that those annotators will use. The Leo-core package comes with several commonly used descriptors such as an aggregate engine factory, and type system descriptors.

#####LeoAEDescriptor

This is one of the handiest classes in Leo. It allows the user to easily create and modify an AnalysisEngineDescriptor object without needing to directly access the UIMA API. The class extends `gov.va.vinci.leo.descriptors.LeoDelegate`, which also is implemented by LeoRemoteAEDescriptor.

For a complete list of the descriptors available see the descriptors section of the Javadocs.

####Tools
Here you'll find useful tools like an ascii filter, text filters and xml filters. See the Javadocs for complete code and descriptions of each tool.

---

##2. Installation and Configuration of Leo-Example
### Installation and Configuration
The guide below will walk you through the steps needed to configure your computer, set up Leo, and start running a simple Leo pipeline. It is exhaustive; if you're already familiar with UIMA and Maven, you can go to the Download page [here](download.html) for the POM's required.

### **2.1 Install Java**
Leo uses Java 8 and both the JRE and JDK are required.
####2.1.1 Java Runtime Environment (JRE)
You will need Java 8 installed on your system. It can be obtained [here](http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html) or through another preferred download method such as your package manager. Please choose the version appropriate for your operating system and processor.
####2.1.2 Java Development Kit (JDK)
You will also need the Java Development Kit installed on your system. It can be obtained [here](http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html) or through another preferred download method such as your package manager. Please choose the version appropriate for your operating system,  processor, and JRE installation.

---

#### **2.2 Install the UIMA Framework**

####2.1.1 Downloading and Extracting UIMA
1. Visit the [Apache UIMA - Downloads](http://uima.apache.org/downloads.cgi) Official Releases webpage.
2. Download the 2.9.0 binary version of the **UIMA-AS Asynchronous Scale-out**.
3. Extract the contents of the file you just downloaded to any permanent location you choose on your computer. Just don't forget what folder you choose because you will need it later. A reasonable location on a Windows computer would be the root of the C: drive. For Linux or Mac users, you might choose /usr/local/ or /usr/share/.

####2.1.2 Setting the UIMA Environment Variable
You will need to set an environment variable called UIMA_HOME, the value of which should be the directory into which you just extracted the UIMA files. For example, on Windows, this might be C:\apache-uima-as-2.9.0

* For help setting a the environment variable, see these examples for Windows, Mac, and Linux provided by Oracle [here](https://docs.oracle.com/javase/tutorial/essential/environment/paths.html).

---

### **2.3 Install Eclipse or Intellij**

####Windows and Mac OS X
* This example will continue using Eclipse; however, Intellij works excellently and is preferred by some.
* You can find the Intellij download needed [here](http://www.jetbrains.com/idea/download/). The free community edition has all of the functionality required for Leo.
* When you visit the Eclipse [download page](http://www.eclipse.org/downloads/), you are presented with several varieties of Eclipse. The processes described in this document have only been tested with the version called "Eclipse IDE for Java Developers". Download Eclipse and extract it to some location on your hard drive.

---

### **2.4 Install Eclipse Plugins**

After launching Eclipse, in the Preferences window, select "Install/update" -> "Available Software Sites" in the tree on the left. On the far right, there should be an "Add" button as depicted in this image:

<img src="images/Available_software_sites.PNG" alt="Available Software" style="width: 650px;"/>

Using the "Add" button, ensure that the following sites are in your list and are enabled. If Eclipse warns you that any of these already exist, that's fine; you can skip those.

Recommended Name| Location
:----------|:-----------
Eclipse Project Updates | <http://download.eclipse.org/eclipse/updates/3.7>
Mylyn for Eclipse Indigo | <http://download.eclipse.org/mylyn/releases/indigo>
Maven | <http://download.eclipse.org/technology/m2e/releases>
Luna| <http://download.eclipse.org/releases/Mars>

*As a side note, Mars is the name of the current version of Eclipse, the link name may change with newer versions.

Once you have added all of these sites, click OK on the Preferences dialog. Now we can proceed to actually install the plugins. To do this, click "Install New Software..." in the "Help" Menu. In the "Install" dialog that pops up, there is a drop-down menu where you can select a site to work with.

When prompted to restart Eclipse, click "Not Now" until you have installed all of the plugins. You will need to restart Eclipse after the last plugin has been installed.

####**2.4.1 Basics of Maven**

Leo uses Maven dependencies and as such, there needs to be a maven environment configured on your computer. Maven is a dependency and build system. The core of maven is found in the pom.xml that will reside at the top level of every project. The pom.xml describes the dependencies, as well as build, deployment and reporting aspects of the project. With Leo you typically use maven at the very beginning of your project in order to setup dependencies and builds; eclipse is generally used throughout the rest of the project.

In order to use our maven repository, you need to update your settings.xml file. This file is found in (~/.m2/). If there is no settings.xml file create a blank text file with that name and populate it with this code.

```xml
<project ...>
  <repositories>
     <repository>
	 <id>decipher</id>
	 <url>http://decipher.chpc.utah.edu/nexus/content/groups/public</url>
     </repository>
  </repositories>
  ...
</project>
```

Additionally in the menu bar you can go to Window > Preferences.  Select Java > Build Path > Classpath Variables, click on the new button > MR_REPO variable and point it to your Maven repository.

---

### **2.5. Add Libraries**

Enter the Eclipse Preferences window once again and select Java -> Installed JREs. If you have more than one JRE listed, you may want to do this for each one just to be sure.

Select your JRE and click the "Edit" Button on the right.  In the window that pops up, click on the "Add External JARs..." button. Browse to the location where your JDK is installed, looking either for tools.jar or classes.jar (depending on your platform):

* In Windows, look for C:\Program Files\Java\jdk1.8.0_02\lib\tools.jar (the folder version number may vary).
* In Ubuntu Linux, this will likely be something like /usr/lib/jvm/java-8-openjdk/lib/tools.jar
* In Mac OS, look in /System/Library/Frameworks/JavaVM.framework/Classes

Verify that the JAR you just selected (either tools.jar or classes.jar) was added to the list in the "Edit JRE" window and click Finish.

Before closing the Preferences window, select Java -> Build Path -> User Libraries. Click the "New" button, name the library "UIMA-AS-2.9.0", and click OK. You should now see UIMA-AS-2.9.0 in your list of User Libraries. Select UIMA-AS-2.9.0 and click on the "Add JARs..." button. Now browse to the location where you saved the UIMA library way back at the beginning of this document. Within the UIMA directory, browse into the lib directory and select all of the JARs. After you click OK, you should see many entries under UIMA-AS-2.9.0 in the User Libraries dialog. We still need to add more JARs to it, so select UIMA-AS-2.9.0 again and click the "Add JARs..." button. Under the main UIMA directory, look for a directory called apache-activemq-5.14.0. Browse into it, then browse into its lib directory, and select all of the JARs you find. Now click OK on the User Libraries dialog.

---

###**2.6. Install Git**

While it is true that you already installed the Git plugin, the command line tools are still necessary as the plugin simply doesn't give us everything we need.

#### 2.6.1 Windows

A distribution of git can be downloaded [here](https://git-scm.com/downloads). If a GUI interface is desired, a list of available interfaces is [here](https://git-scm.com/download/gui/windows). Both [Eclipse](https://projects.eclipse.org/projects/technology.egit) and [IntelliJ](https://www.jetbrains.com/help/idea/using-git-integration.html) offer plugins to work with git.

#### 2.6.2 Linux

Check your package manager.  Example:

~~~
sudo apt-get install git
~~~

#### 2.6.3 Mac OS X
It's likely that you already have Git installed. You can check by using ```git --version``` in terminal. If for some reason you don't, [click here](http://git-scm.com/downloads).

---

### **2.7. Clone the Leo Example**

Now that Git is installed, we can grab the latest version of Leo. Open up the terminal(or Git Bash on Windows) and cd to your workspace directory. For an easy example pipeline, enter this command:

~~~
git clone http://decipher.chpc.utah.edu/gitblit/git/examples/leo-example.git
~~~

Git has now created an example pipeline that uses Leo, and has the ability to process text, annotate the text through various java classes, and output the data in xmi format. However, the project is not quite ready to be run; at this point you will need to import the project into Eclipse.

---

###**2.8. Import Leo into Eclipse Workspace**
In the step above, we downloaded Leo into a directory contained in our workspace directory. However, Eclipse doesn't know that yet. You can add the project into eclipse by either using maven, or the Eclipse menus.

####2.9.0 Create eclipse Project Using Maven
This step is the quickest and easiest, but for the sake of completeness, the option to create the project through Eclipse can be found in the next step. While in the terminal or command line, make sure that you are currently in the directory for the project (i.e. User/Workspace/leo-example). While in the directory run:

~~~
mvn eclipse:eclipse
~~~

The eclipse project has now been created.

#### 2.8.2  Import project through Eclipse
In the Eclipse File menu, select "Import...". A prompt will now show up, expand the "Git" folder folder and select "Projects from Git", then go to "Existing projects from Git". When you are prompted to select a Git repository, click the "add" button on the right. Then browse to the Leo directory that you just cloned. Make sure the checkbox is checked next to the Leo directory and click OK.  Click "Next".  Now you will be prompted to select a wizard for importing the project, select the second option, which is labeled "Use the New Projects wizard" and click "Finish".

A new dialog window will appear prompting you to select a wizard. Expand the Java folder and pick "Java Project". Click "Next and name the project "leo" or "leo_example" depending on which option you cloned.

 ---

###**2.9 Running the Sample Project**

In the Previous section we set up a Leo environment using a Leo-Sample project. The steps to run a project described in this section are the same whether you are using the sample project, or your own project build.

If you're running the sample project make sure that there is an input and output directory in the "Data" folder on the left side of the eclipse project. If the output folder is not there, create one.

If you're running your own pipeline using Leo components, be sure to stop here and make sure that your Service and Client are configured.

Now you're ready to run the example pipeline, or Leo as a whole pipeline with your own or other annotators.
###  Start Broker
Open terminal/command line, and find your way to the UIMA home directory, then CD to the bin sub-directory. Now run the startBroker script(startBroker.bat on Windows, ./startBroker.sh otherwise). At this point there will be some text scrolling by, just leave this running and switch back to Eclipse.

###  Run the Service
Run the service program by opening its source file (i.e src/main/java/ExampleService.java) and clicking the green Run button at the top of Eclipse.

### Run the Client
Now that the Service is running, it's time to open the client program and click the Run button again. You will now need to open a new terminal window, cd to your UIMA_HOME/bin directory again and run the "annotationViewer" script(./annotationViewer.sh). This dialog will now appear:

<img src="images/annotation_viewer.PNG" alt="Annotation Viewer" style="width: 650px;"/>

Change the first text field "input directory" to the *output directory* that you created.

The second one is a little trickier. Scroll back through the SampleService output looking for a line that looks something like this:

~~~
Aggregate: /var/folders/x9/r87vxc1s1fb6gl80r306hf7h0000gn/T/leoAggregate7687725334298388844.xml
~~~

Copy the path to the aggregate descriptor into the second text field of the Annotation Viewer window.

Once both fields are filled in, click the "View" button. A new window with the analyzed documents will pop up. It will contain a list of xmi files, each one corresponding to one of the files in your input directory. To see results of the simple service, double click on any of the xmi files. If an "Annotation Results..." window appears with colorful text, you're all done!  The pipeline has processed the text according to the annotators provided in the client.java program, and given you the xmi files needed for your research!

 ---

##3. Understanding and Adapting the Leo-Example Project

In the previous section we downloaded the Leo-Example and configured your computer for Leo, UIMA, and Maven. We also ran the sample project as is.  The example pipeline and input/output files are a simple demonstration used in order to ensure that everything you would need for a Leo Pipeline is ready to go.  This section will detail each of the packages and how you can configure Leo for your own projects.

####  Adapting the Client

The Example comes with two possible Clients to use; the first is "CommandLineClient.java" which allows the user to give command line Arguments for configuration if that's the route they prefer. This guide is going to begin with the other Client "ExampleClient.java". The Example Client is where you will specify which Collection Reader and Listener your pipeline will use.  This Leo-Example project only contains two possible Listeners(CSV Listener, XMIListener) and four Readers(BatchDatabase, DatabaseCollection, FileCollection and XmiFileCollection). For more information about other Readers and Listeners as well as learning how to build and adapt your own, see the Listener and Reader Section of the User-guide under "Leo Client".

In the code below, you only need to change which Reader and Listener you plan to use.

```java
new CommandLineClient(  new File("config/ClientConfig.groovy"),
                                new File("config/readers/FileCollectionReaderExample.groovy"),
                                new File[]{new File("config/listeners/SimpleXmiListenerExample.groovy")}
                             ).runClient();
```

This is all that you need to change in the Client file. Now you can go to the respective configuration file for the Reader and Listener you're using in order to specify the paths used in your project. In the "config" folder you will find the configuration files for each Listener and Reader, as well as any resource files required by an annotator. All that is required in the config files is to specify your project specific paths.

Specify the Input files in the Reader(FileCollectionReaderExample.groovy):

```java
reader = new FileCollectionReader(new File("data/input"), false);
```
Specify the Output files in the Listener(SimpleXmiListernExample.groovy):

```java
listener = new SimpleXmiListener(new File("data/output"))
```
The ClientConfig.groovy file requires the user to specify a brokerUrl where the ActiveMQ Broker is running, a CAS pool size to determine how big the input and output queue is, as well as the "endpoint" or project name.

```java
brokerUrl                       = "tcp://localhost:61616"
endpoint                        = "leoExample"
casPoolSize                     = 4
CCTimeout                       = 1000
```

The client has now been configured for your specific project.

###Adapting the Service

**For scalability, multiple instances of a service may be started in their own JVM instances. All services should have the same broker url and service names. In that scenario, the client requests are routed to whichever instance is currently idle.** In gov.va.vinci.leo.examples/ExampleService.java  you will find the code required to change the broker url and name of the input queue.

The ServerConfig.groovy file requires the user to specify a brokerUrl where the ActiveMQ Broker is running, a CAS pool size to determine how big the input and output queue is, as well as the "endpoint" or project name.

```java
brokerUrl                       = "tcp://localhost:61616"
endpoint                        = "leoExample"
jamQueryIntervalInSeconds       = 60
jamResetStatisticsAfterQuery    = true
jamServerBaseUrl                = "http://localhost:8080/jam" // If not specified, the service will not try to register with JAM.
jamJmxPort                      =-1  // If -1, the service will pick an open port automatically.
```

In addition, a service requires a type description and aggregate to actually do work.

```java
 LeoAEDescriptor aggregate =  new ExamplePipeline().getPipeline();
            aggregate.setIsAsync(true);
            aggregate.setNumberOfInstances(1);
```

###Adapting the Pipeline and Selecting Annotators

At this point, the only thing you need to do in order to further adapt your pipeline is to choose which annotators you will be using in gov.va.vinci.leo.examples.pipeline. We will be looking into ExamplePipeline.java. ExampleScalablePipeline provides the structure required if the pipeline you are implementing requires further scalability. PipelineInterface.java is a helper class that allows for interchangeability of multiple pipelines so that the user can easily switch between pipelines if needed. You only need to extend it in order to use the same names from one pipeline to another.

ExamplePipeline.java has the code required to use the regex and Window Annotators. For information on how to use other annotators, go to the "Additional Annotators" section of this User-guide.

```java
public class ExamplePipeline implements PipelineInterface{
    LeoAEDescriptor pipeline = null;

    /**
     * Constructor
     * @throws Exception
     */
    public ExamplePipeline() {
    }

    @Override
    public LeoAEDescriptor getPipeline() throws Exception {
        if(pipeline == null) {
            pipeline = new LeoAEDescriptor();

            /** Add the regex annotator first **/
            pipeline.addDelegate(
                    new RegexAnnotator()
                        .setGroovyConfigFile("config/resources/example-regex.groovy")
                        .setLeoTypeSystemDescription(getLeoTypeSystemDescription())
                        .getLeoAEDescriptor()
            );

            /**
             * Second, add windows around the regexes. The window annotation will extend 1 word (defined as separated by a whitespace
             * character before and after each character string.
             */
            pipeline.addDelegate(
                    new WindowAnnotator(Window.class.getCanonicalName(), RegularExpressionType.class.getCanonicalName())
                        .setWindowSize(1)
                        .setAnchorFeature("Anchor")
                        .setLeoTypeSystemDescription(getLeoTypeSystemDescription())
                        .getLeoAEDescriptor()
            );

        }

        return pipeline;
    }

    @Override
    public LeoTypeSystemDescription getLeoTypeSystemDescription() {
        LeoTypeSystemDescription description = new WindowAnnotator().getLeoTypeSystemDescription();
        description.addTypeSystemDescription(new RegexAnnotator().getLeoTypeSystemDescription());
        return description;
    }
}
```

In order to create your own annotator. the `annotate` method has to be developed.

This line of code already assumes that JcasGen has been called.

```java
@Override
public LeoTypeSystemDescription getLeoTypeSystemDescription() {
    LeoTypeSystemDescription description = new WindowAnnotator().getLeoTypeSystemDescription();
    description.addTypeSystemDescription(new RegexAnnotator().getLeoTypeSystemDescription());
    return description;
}
```

In other projects you will need to call JcasGen in order to generate your java and class files.

```java
if (generateTypes)
{
	types.jCasGen("src/main/java", "target/classes");
}
```

### Running your Project

The process used to finally run a project is the same as described at the end of the previous section.

1. Start the Broker
2. Run the Service
3. Run the Client

From here, you can view the annotations created in your pipeline using the default UIMA annotation viewer, or some of the other tools described in the [Components and Tools](components.html) page.

---

## 4. Additional Annotators

Several specialized annotators have been developed and optimized for general NLP tasks.

### Annotation Librarian

The AnnotationLibrarian is a helper class that performs many of the common functions needed by Annotation Engines. Functionality includes methods such as getAllAnotationsOfType(), getFirstAnnotationOfType(), mergeAnnotations().

### Annotation Pattern Annotator(APA)

Building on Annotation Librarian ideas, the AnnotationPatternAnnotator mirrors regular expression functionality to manipulate annotations. It uses a resource file that contains regular expressions that include tagged annotation types combined with standard regular expressions syntax. Additionally, part of the pattern can be separated as an Anchor feature, where a new annotation is created for the tagged snippet.

The behavior of Annotation Pattern is very similar to the Java Regex Package's Pattern and Matcher classes.
The AnnotationPattern Object is created using the simple syntax :
AnnotationPattern patternName = AnnotationPattern.compile(regex);
where
- compile() - is a private static method which takes the annotationPattern to return an AnnotationPattern Object.
- regex - is a String that will mention the Annotation annotationPattern which the user is expecting. Below are various examples
to show different combination of patterns.

### ConceptLink Annotator

Creates output annotations covering related concepts. Includes a MatchMaker annotator for concept value pair annotations.

### CONTEXT

This project contains a Leo wrapper for ConText application found at http://orbit.nlm.nih.gov/resource/context.

ConText is based on a negation algorithm called NegEx. ConText's input is a sentence with indexed clinical conditions;
ConText's output for each indexed condition is the value for contextual features or modifiers. The initial versions of ConText determines values for three modifiers:

Negation - affirmed or negated.
Temporality - recent, historical, or hypothetical.
Experiencer - patient or other.
The current version of ContextAnnotator has additional output features.

ContextAnnotator outputs an annotation of the outputType type.
The outputType also has an additional feature -- Window -- which contains the annotation of the window in which the concept is found.
The ContextAnnotation will span only the concept itself and not the window. So this feature is a hook to the input type in the output type.

We adopted it as a UIMA annotator, and made several changes to the algorithm. Such as a new rule category "adjacent", which specifies rules for words that are immediately before or immediately after the anchor word; as well as several new rules to the resource file.

### Merge Annotator

The Merge Annotator allows the user to easily merge overlapping or proximal annotations of the same type.

### REGEX Annotator

The Regular Expression Annotator provides a UIMA based annotator for searching a corpus using regular Expressions. The Regular Expression Annotator can be configured via a simple configuration file, for processing a single group of patterns with the same configuration for each pattern, or via a more flexible Groovy configuration file that allows for multiple groups of patterns configured with different parameters per group of patterns. For a more detailed synopsis and download instructions, you can go the the Regular Expression site [here.](http://decipher.chpc.utah.edu/sites/gov.va.vinci/leo-regex/2014.09.1/)

### Sentence Annotator

Annotates sentences found in a covering annotation or on the document as a whole.

### Sherlock
The Sherlock framework provides the base objects for implementing machine learning (ML) algorithms in a UIMA environment. It does not attempt to provide a specific ML implementation but rather the means to accessing existing implementations within a framework of objects that provide a common interface. This is accomplished by recognizing two primary workflows for learning tasks:

(Training) INPUT -> Compute Feature Vector -> Create Vector Classification -> Aggregate Vectors and Classifications -> Train a Learning Model
(Prediction) INPUT -> Compute Feature Vector -> Predict from Trained Model
Sherlock was created from the need to be able to perform the above outlines processes within UIMA utilizing previously created annotations as well as the source text as the INPUT for feature vector creation.

### White Space Annotator
Create a series of tokens that ignore whitespace and identify the basic building blocks of words, newline, number,
and punctuation identifiers. This can be done either character by character using the WhitespaceTokenizer, or regular
expressions using RegexWhitespaceTokenizer.

### Window Annotator

The Window annotator takes an anchor annotation and builds a new annotation that is larger that the anchor annotation based on specified number of tokens to the left and to the right of the anchor annotation. If only a single window number is specified, the left and write window will be at the equal distance from the anchor annotation. A link to the anchor annotation gets stored in an output annotation as a feature.

---
## 5. Terminology

**Aggregate Analysis Engine** An Analysis Engine that is implemented by configuring a collection of component analysis Engines.

**Analysis Engine(AE)** - A program that analyzes artifacts (e.g. documents) and infers information about them, and which implements the UIMA Analysis Engine interface Specification. It does not matter how the program is built, with what framework or whether or not it contains component ("sub") Analysis Engines.

**Annotator** - Not to be confused with manual annotators(people), this is a software component that is used to write relevant metadata over the text being processed.

**CAS** - The UIMA Common Analysis Structure is the primary data structure which UIMA analysis components use to represent and share analysis results. It contains:

* The artifact - This is the object being analyzed such as a text document or audio or video stream. The CAS projects one or more views of the artifact. Each view is referred to as a Subject of Analysis.
* A type system description indicating the types, subtypes, and their features.
* Analysis metadata "standoff" annotations describing the artifact or a region of the artifact
* An index repository to support efficient access to and iteration over the results of analysis.
* UIMA's primary interface to this structure is provided by a class called the Common Analysis System. We use "CAS" to refer to both the structure and system. Where the common analysis structure is used through a different interface, the particular implementation of the structure is indicated, For example, the JCas is a native Java object representation of the contents of the common analysis structure.

**Client** - A component that sends requests to the services.

**Collection Reader** - A component that reads documents from some source, for example a file system or database. Each document is returned as a CAS that may then be processed by Analysis Engines. If the task of populating a CAS from the document is complex, a Collection Reader may choose to use a CAS Initializer for this purpose.

**Descriptor** A descriptor defines the annotators that make up a service, as well as the type system(s) those annotator(s) use.

**Listener**  A Listener is a package that listens to the output provided by the pipeline, and outputs it in the different formats needed per project.(csv, xmi)

**UIMA-AS** - UIMA-AS is substantially different from UIMA and provides a range of additional functionality enabling scalable processing. While UIMA components can be run within UIMA AS with no descriptor changes, additional descriptor files are required to deploy systems based on UIMA AS.

**Service** - The Leo service provides the Core Server functionality for launching UIMA-AS services.

**XMI** - XML Metadata Interchange

**XML** - Extensible Markup Language

---

## 6. References and Helpful Links

[Apache Maven](http://maven.apache.org/download.cgi)

[eHost](http://ehostdoc.com/)

[UIMA-AS 2.9.0 Documentation](https://uima.apache.org/d/uima-as-2.9.0/uima_async_scaleout.html)

[VINCI Home](http://www.hsrd.research.va.gov/for_researchers/vinci/)

repository: department-of-veterans-affairs/Leo

gha_language: Java

repo_name: department-of-veterans-affairs/Leo


text:
---
layout: post
title: "11 things we just learned about how the Apple Watch works"
date: 2014-11-20 17:52:27 +0100
comments: true
categories:
- Apple
external-url: http://www.theverge.com/2014/11/18/7243085/most-important-apple-watchkit-discoveries
---

Ross Miller, The Verge:

> We've been combing through all of [Apple's publicly-released documentation](https://developer.apple.com/watchkit/) and watching the WatchKit introduction video. This is everything we've learned about Apple Watch and how it works.

Interesting.

license: MIT

repository: analogsenses/source

branch_name: refs/heads/master

extension: markdown

license_type: permissive

repo_name: analogsenses/source


text:
IODB design specification
============================

This document outlines design and implementation of IODB database at 0.4 release.

Basic terms
-----------

This spec assumes basic knowledge of log structured stores.
Here are some basic terms

### Log
- Log is sequence of updates organized by time
    - This sequence might not follow actual organization of data, it can be reorganized by rollback or compaction
- Log can be spread over multiple files.
- Each Log Entry contains:
    - list of update keys
    - list of deleted keys (see tombstone)
    - link to previous update

- Key-Value pair is found by traversing Log until key is found
    - rollback or Compaction can reorganize sequence of updates
    - search follows link to previous update in Log Entry, rather than sequentially traversing log file in reverse order

![Log before merge](img/spec/log-before-merge.jpg)

### State of store
- content of store for given VersionID (or most recent update)
- all key-value pairs in store

### Tombstone

- Already inserted data are immutable. It is not possible to delete keys directly from store.
- Deleted keys are indicated by **tombstone** marker
    - it is special type of value
    - in binary storage is indicated by value with length `-1`
- Compaction eventually removes tombstones from store and reclaims the disk space

### VersionID
- Each Update takes VersionID as a parameter
- Each Update creates snapshot of data
- This snapshot is identified (and can be reverted to) by `byte[]` identifier

### Binary Search

- Keys in each update entry are sorted
- To find a key binary search is used
- It compares `byte[]` (or `ByteArrayWrapper`) with content of file
- All keys have equal size, that simplifies the search significantly
- 0.4 version uses `FileChannel` which is very slow
    - memory mapped file can speedup binary search 10x
    - unsafe file access is even faster

### Merge iterator

- state of store (all key-value pairs) is reconstructed by replaying all updates from start to end
- easiest way is to traverse log and insert key-value pairs into  in-memory `HashMap`
    - that consumes too much memory

- IODB uses Merge iterator (lazy-N-way merge) instead

- Inside each Update Entry (such as V1, V2...), keys are stored in sorted order.

- Compaction reads content of all  Updates  in parallel, and produces the result by comparing keys from all updates.

- The time complexity is `O(N*log(U))`, where `N` is the total number of keys in all Updates and `U` is the number of Updates (versions) in merge.

- Compaction is streaming data; it never loads all keys into memory. But it needs to store `U` keys in memory for comparison.

- If `U` is too large, the compaction process can be performed in stages.

- No random IO, as all reads are sequential and all writes are append-only.

![Merge iterator](img/spec/merge-iterator.jpg)

### Merge in log

As the number of updates grows and the linked-list gets longer, Find Key operation will become slow
Also obsolete versions of keys are causing space overhead.

Both problems are solved by merging older updates into a single update.
The merge is typically performed by a compaction process,
which runs on the background in a separate thread.

- Merge process takes N Updates
    - from most recent
    - until previous Merge Entry or start of the log
- It produces Merge Iterator over this data set
- It inserts current state of store into Log Entry

- 0.4 serializes Merge Entry into `byte[]` using `ByteArrayOutputStream`
    - it can run out of memory
    - temporary file should be used instead

![Log before merge](img/spec/log-before-merge.jpg)

![Log after merge](img/spec/log-after-merge.jpg)

### Shards

- If store contains too many Key-Value pairs, Merge or binary search becomes slow
- So the store is split into Shards, each Shard is compacted and managed separately
- Newer data are stored in single log (Journal) latter moved into Shards

Data lifecycle
------------------

IODB moves data around to ensure good performance

### Update

Modifications (updated key-value pairs and keys to delete) are inserted in
batches (here referred as updates).
Each Update is identified by `VersionID` and creates new snapshot which
can be rolled back to.

Over time data (key-value pairs) move following way:

- journal contains most recent updates
- over time journal becomes too long, so Distribute Task is triggered
- at start Distribute Task inserts Distribute Placeholder entry into Journal
- Distribute Task moves data from Journal into Shards
- after Distribute Task finishes, it puts file positions in Shards into Journal
- over time (after some Distribute Tasks are finished) some Shards become too long, this triggers Shard Merge task
- Shard Merge tasks select longest Shard and merges content from multiple Update Entries into single Merge Entry

![Data flow](img/spec/data-flow.jpg)

### find key (get)

`Store.get()` returns value associated with the key, or `None` if key is not found

- search traverses journal, until Distribute Placeholder is found (or key is found)
- from Distribute Entry it takes position in Shard
- search continues by traversing Shards from given position
- search finishes when
    - key is found
    - Merge Entry is found in Shard
    - end of Shard is reached

![find key](img/spec/find-key.jpg)

### get all

`Store.getAll()` returns content of store (all key-value pairs) for given `VersionID` (or msot recent version).

Store content is spread over journal, shards, multiple files.
So the dataset is produced by iterating over multiple files, and using lazy-N-way merge iterator:

- traverse Journal until Distribute Placeholder is found
- take all data in Journal from VersionID until Distribute Placeholder
    - merge them into single data set
    - store it in temporary file (is currently done in-memory and that causes out-of-memory exceptions)
    - this is referred here as Journal content
- iterate over content of shards and merge them with Journal content
    - Shards are not overlapping
    - Journal content is sorted
    - Shard and Journal content is merged on single iteration

![get all](img/spec/get-all.jpg)

Rollback
----------

Rollback reverts store into state at given VersionID. It discards data inserted after
given VersionID.

Log is chain of updates. It is not possible to overwrite new updates during rollback.
So the new update (Offset Alias Log entry) is written into Journal.
It instruct search and other operation to skip irrelevant data while traversing the log.

Rollback is performed in following way
- traverse Journal until matching VersionID is found
- continue Journal traversal until Distribute Entry is found
    - this is first distribute entry after VersionID
    - we need Shard positions for given VersionID
- insert Offset Alias Log Entry into journal
- rollback Shards into state found in Distribe Entry

![Log after rollback](img/spec/log-after-rollback.jpg)

Shards
------

- binary search on large tables is slow, also compaction on large tables is slow
- only most recent data are stored in Journal
- older data are distributed into shards
- there is Distribute Task, it takes data from Journal and distributes them into Shards

### Shard Splitting

- ideally Shards should be created dynamically
    - small or empty store starts with single shards
    - if Shard becomes too big, it is sliced into several smaller Shards
    - if Shard becomes too small, it is merged into its neighbours

- dynamic Shard allocation has following advantages
    - less configuration, number of shard scales with store size
    - Shard Boundaries are self-balancing
        - if Key Space is sliced at constant intervals, and key distribution is not random (for example incremental keys), most keys can end in single interval
        - with dynamic Shard allocation new shards are created by slicing old Shards.

- 0.4 release has **static sharding**
    - `ShardedStore` has `shardCount` parameter
        - this is hardcoded into store and can not be changed
    - key space is sliced at constant interval
    - dynamic Sharding was in 0.3 release, but caused concurrency issues
on

File naming
----------------------

- Log is composed of multiple files
- IODB stores data in directory
    - this dir contains multiple logs (Journal and Shards)
    - it also contains temporary files

- Each file has prefix, file ID and suffix

- Prefix identifies type of file (journal, shard, temp)
    - It also assigns file into its Log (Journal or Shard)

- File ID gives position of file within log
    - TODO File ID should have two parts; log file position and file version

- Suffix is not used
    - TODO drop suffix or use it to identify temp files

- Temporary files are used
    - To produce merged result during compaction and distribute (TODO currently uses `byte[]`)
    - To store large updates
    - To store result of `getAll()` operation (TODO currently loaded into in-memory `TreeMap`)

File handles and File IO
------------------------

- IODB performs following IO operations on files
    - append at end of the file
    - sync; flush write cache on file (temp files are not flushed)
    - binary search read

- IODB keeps number of files open
    - one writeable handle to append to Journal
    - one readonly handle for each file

- File handle is limited resource
    - by default only 64K handles are allowed per process on Linux
    - exhausting file handles can cause JVM to crash (it can not open DLLs or JAR files)

- IODB keeps file handles in `LogStore.fileHandles` map

- File handle can be `FileChannel` or `MappedByteBuffer` depending on implementation

- file delete
    - IODB needs to delete outdated files
    - files can not be deleted while it is opened for reading (race condition)
        - other threads could fail, while reading data from store
        - memory mapped file crashes (segfault) when accessing unmapped buffer
    - there is semaphore for each file
        - it protects file from deletion while it is read from
        - each read operation needs to lock files it will use
        - see `LogStore.fileSemaphore` for details

Memory mapped files
-------------------

- Current version (0.4) uses `FileChannel` to read files
    - that is very slow for binary search,
    - file caching does not work efficiently
    - `FileChannel` requires seeks and uses extra translation layer

- better way is to use memory mapped files
    - it maps file into memory address space, reads do not require translation
    - binary search is very efficient (10x) faster
    - caching works very well

- memory mapped (mmap) files have some downsides. Those are outlined [in this blog post](http://www.mapdb.org/blog/mmap_files_alloc_and_jvm_crash/)
    - JVM has no official way to close (unmap) mmap files
        - file handle is closed after GC, that can cause JVM process to run out of file handles
    - there is a hack to close mmap file forcibly
        - it is not supported in all JVMs (Java 9, Android etc are different)
        - after unmapping, mmap file can not be read from, it causes access to illegal address and JVM process crashes with SEGFAULT

- memory mapped files can be speedup even more by replacing `byte[]` with primitive `long` for binary search

Background Operations
--------------------------

- IODB performs various maintenance tasks on background

- background task are scheduled automatically in `Executor`
    - see `executor` constructor parameter at `ShardedStore` and `LogStore`
    - if you set `executor` param to null, background tasks are disabled
        - in that case user should perform compaction by calling background tasks directly
        - see `task*()` methods at `ShardedStore` and `LogStore`

IODB performs following tasks

### Distribute Task

- distribute task moves data from Journal into Shards
- it is triggered when number of updates in journal is over limit (see `ShardedStore#journalUpdateCounter`)
    - update count is since last merge, not total count
    - TODO it should be triggered if total space of undistributed updates is over limit, not update count
- can be triggered manually with `ShardedStore.taskDistribute()`

Distribute tasks works in following way:

- insert Distribute Placeholder Log Entry into journal
    - this is to prevent race conditions, journal might be updated while Distribute Task runs
- iterate over Journal Updates, produce list of all updates since previous Distribute Placeholder
- produce iterator over data in Journal Updates
- slice content of iterator into shards
    - only single pass is needed, lazy-N-way merge produces sorted result
    - each Shard is updated with single update
- Journal is updated with Distribute Log Entry
    - it contains pointers to latest updates in Shards
- Distribute Placeholder is replaced by final version of Distribute Log Entry
    - Offset Alias is inserted into journal

### Compact Task

- Distribute Task merges multiple Update Entries into single Merge Entry
- it makes Shards faster and releases unused version data
- it is triggered when number of unmerged updates in Shard is over limit (see `LogStore.umergedUpdatesCounter`)

- can be triggered manually by `LogStore.taskCompact()`
    - TODO expose this in `ShardedStore` as well

It works in following way:

- `ShardedStore` runs background task
    - every N seconds it selects most fragmented Shard
    - if number of unmerged Update Entries (since last merge) is over limit, it calls `LogStore.taskCompact()` on this Shard
- In LogStore it selects Most Recent Update
- It traverses Update Entries until it reaches previous Merge Entry (or start of log)
- It produces merge iterator over all updates in Shard
- It inserts merged content of Update Entries as new Merge Log Entry into Shard
- It replaces Most Recent Update with Merge Entry by using Offset Alias

### File Shrink Task

This tasks replaces old file with new one, and reclaims space in file. It is not present in 0.4 release

TODO current design (0.4) is not very effective to reclaim space released by rollback.

TODO FIG shrink file

Source code overview
------------------------

Some notes:

- key, values and VersionIds in IODB are represented as `byte[]`
    - for practical purposes (hash,comparable) `ByteArrayWrapper` is used

- Basic interface is `Store`
    - it is dictionary that maps key to value

### Quick Store
- it stores all data in memory
    -uses sorted map
- is used as a reference for writing unit tests (other stores should pass the same criteria as simple implementation does)
- uses simple `ReadWriteLock` for thread safety
- can store data on disk using log file
- rollback method replays updates in reverse order
- when opened it replays log file to reconstruct current state (all key-value pairs) into SortedMap

### Log Store

- `LogStore` stores all data in multiple log files
- it uses binary search to find keys
- this class is used for Journal and Shard in `ShardedStore`
- it has background operation that merges multiple Update Entries
    - merge operation uses lazy-N-way merge iterator, so it does not load all entries into heap

- `serialize*()` methods convert updates and other modifications into their binary representation
    - `byte[]` is used right now
    - TODO use temporary files, large updates can cause OOEM exceptions

- `LogStore#append` will take binary data and append it to end of the file

#### concurrency

- `LogStore` is thread safe, but does not have a global lock
- it maintains most state in file

- there is a race condition in file deletion
    - if file is deleted while other thread is reading it, it could cause JVM to crash (see memory mapped files)
    - to prevent that we use file semaphore
    - read operation must lock files it will use
        - delete operation is performed on background, and will not remove files that are locked for reading
    - see `fileSemaphore` and `fileToDelete` fields

- there is a race condition in file append (expand log file)
    - multiple writers should be allowed at the same time, but there is only single file
    - ideally the content of update should be written into temporary file, and latter transferred into log using `FileChannel.transferFrom()` (avoids CPU cache)
    - in 0.4 release append lock is used, so LogStore does not handle concurrent updates

- there is a race condition between `Store.get()` and `Store.update()`
    - `Store.get()` obtains last log file offset from atomic variable, and works with that
        - it can not see future updates

#### Background tasks

- `LogStore` has only one background task
- it merges multiple Updates into single Merge entry
- can be invoked manually with `LogStore.taskCompact()`

### Sharded Store

`ShardedStore` is combination of multiple logs (`LogStore`).
It is composed of Journal and Shards, all are represented by `LogStore`.

- newer data are in Journal
- data are moved into Shards by distribute task
- Sharded Store controls compaction of Shards

### Concurrency

- there is a race condition between rollback and distribute task
    - in 0.4 release rollback operation uses `distributeLock` so the distribute task and rollback can not run at the same time

- there is a race condition between update and distribute task
    - it is solved by using two Distribute Log Entries
        - Distribute Task inserts Distribute Placeholder Log Entry into Journal
        - that is latter replaced by Distribute Log Entry

File format overview
-----------------------

Binary format is described in `store_format.md`

In short log can contain following types of entries:

- **update log entry** - Is inserted on each update
- **merge log entry** - Is inserted by Compaction Task, merges content from multiple updates into single log entry
- **distribute placeholder log entry** - Inserted into journal when distribute task starts.
- **distribute log entry** - Inserted into journal after distribute task finishes. It contains pointers into shards.
- **offset alias log entry** - inserted by Rollback or Compaction Task. It replaces Update Entry with new one. For example after compaction the most recent update is replaced by Compacted Entry.

Unit tests
-------------------

- IODB uses JUnit 4 runner to run unit tests
    - test written in ScalaTest should extend `org.scalatest.junit.JUnitSuite`

#### io.iohk.iodb package
- contains normal unit tests
- there is abstract `StoreTest which tests general contract of `Store` interface
    - it has `QuickStoreRefTest`, `LogStoreTest` and `ShardedStoreTest` subclasses
- most tests extends `TestWithTempDir` class that creates and deletes temporary directory

### bench package
- contains benchmarks
- those are not run automatically, but using `main` method
    - from command line you can trigger it using following command line `sbt 'test:run-main io.iohk.iodb.bench.ClassName'`

### prop package
- properties testing
- generates random data and tries to break store

### smoke
- integration tests
- generates random data and tries to break store
- there is `M1Test` which is acceptance test for 0.4 release

### Long running tests
- by default unit tests run only short time (10 minutes) to make development easier
- however before release full test suite should run for several days to
    - find race condition
    - stress test store
    - test for memory and disk leaks
- Long running test are trigged by `longTest=1` system property
    - from command line use `sbt test -DlongTest=1` to run acceptance

repository: input-output-hk/iodb

extension: md

gha_language: Scala

repo_name: input-output-hk/iodb


text:
---
layout: article_post
title: "The SaaS Learning Curve"
date: "2018-10-27 11:00"
categories:
tags: [article, nonfiction, startups, business, advice]
author: The Mission
rating: 4
article_url: https://medium.com/the-mission/the-saas-learning-curve-a699afe19ac0
reading_time: 5
date_published: 2018-10-08
summary: "People > tech; SaaS = Service as a Software"
---

## Notes

* Innovators represent about 10% of the market and will provide you with great
  product feedback but rarely have a budget. As such, they are quite useful in
  the pre-revenue stage of a Service as a Software company.
* Early Adopters represent another 10–15% of the market and they have a budget.
  They often buy because they seek a competitive advantage. They see that being
  achieved by working with you to realize what should become a shared vision for
  your product.
* Start thinking of SaaS as Service as a Software, not Software as a Service.


text:
# truecaller-unofficial-api

 Unofficial API to the Truecaller phone number search.

## Installation

 ```
 pip install truecaller-unoffical-api
 ```

or if you like to live on the bleeding edge:

```
python setup.py install
```

## Usage Examples

```
import truecaller_unofficial_api as tr

human = tr.search('<phone number>')
print(human.name)

mobile = human.phone
print(mobile.phone)
print(mobile.countrycode)
print(mobile.carrier)

house = human.address
print(house.city)
print(house.timezone)
```

For more methods, check out the Documentation.

## Disclaimer

This method of accessing Truecaller's database may stop working any day. Use it at your own risk.

## License

`The MIT License`

repository: VishnuVandith/TrueCallerLookup

repo_name: VishnuVandith/TrueCallerLookup


text:
# vagrant-neo4j-miniconda-py2neo-scikit
Vagrant box based on ubuntu/trusty64 with Neo4j, Miniconda, and libs such as py2neo, scikit-learn and matplotlib

## Installation
```bash
vagrant up
```
To install toolkits such as py2neo, scikit-learn and matplotlib:
```sh
vagrant ssh
chmod +x /public/libs.sh
bash /public/libs.sh
```

### Neo4j
After the Neo4j installation http and https connectors are enabled on a default address.
You can access Neo4j via browser from your host via <IP_ADDRESS>:7474/browser/ or localhost:7474/browser/.
IP address is located in a Vagrant file (search for config.vm.network "private_network") and ports in setup/neo4j.sh.

### Miniconda
Miniconda is installed in /usr/local and path is added to ~/.bashrc in a setup/miniconda.sh

### Libs and toolkits
You can add/remove libs and toolkits in a public/libs.sh
To output matplotlib's charts as an image, use Agg [backend](http://matplotlib.org/faq/usage_faq.html#what-is-a-backend):
```python
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
```

### Other
Shared dir VM path: /public

### TODO
- install libs with vm provision
- do md5 checks


text:
# K8s Cluster management in EKS and AKS

Most common APIs of AKS and EKS have been implemented in Java are as below:

AWS Elastic Kubernetes Service (EKS) APIs:
 - Create cluster
 - Get cluster status
 - List cluster
 - Update cluster
 - Delete cluster
 - Add node group to cluster
 - Add fargated compute to cluster
 - Get status of fargated compute
 - Update node group
 - Delete node group
 - Create security group
 - Configure inbound rules for security group
 - Configure outbound rules for security groups
 - List security group
 - List roles
 - List VPC
 - List subnet from VPC
 - List kubernentes version

Azure Azure Kubernetes Service (AKS) APIs:
 - Create cluster
 - List cluster
 - Get cluster by id
 - Create node pool
 - Scale node pool
 - Delete node pool
 - Delete resource group
 - List vNet
 - List kubernentes version
 - List network security group
 - Add inbound rule to network security group
 - Add outbound rule to network security group
 - List regions
 - List subscriptions

How to run:
 - git clone https://github.com/falarica/cloudk8smgmt.git
 - To run AKS tests:
    - Go to AKSClusterTest.java and set tenentId,clientId and secret and run test method
 - To run EKS tests:
    - Go to EKSClusterTest.java and set accessKey,secretKey and run test method
 - Run 'mvn clean install' to build the code and run tests

branch_name: refs/heads/main


text:
---
layout: default
---

# 1994-01-10 - Twelve year flashback to Admiral Bobby Inman

## Header Data

From: gnu (John Gilmore)

To: cypherpunks, gnu

Message Hash: b590dc724c8fbfbe137066ff812b26c90839483e4b1a885ccbc36bffffdb6dd2

Message ID<EMAIL_ADDRESS>Reply To: _N/A_

UTC Datetime: 1994-01-10 10:51:20 UTC

Raw Date: Mon, 10 Jan 94 02:51:20 PST

## Raw message

```
{% raw  %}From: gnu (John Gilmore)
Date: Mon, 10 Jan 94 02:51:20 PST
To: cypherpunks, gnu
Subject: Twelve year flashback to Admiral Bobby Inman
Message-ID<EMAIL_ADDRESS>MIME-Version: 1.0
Content-Type: text/plain

San Jose Mercury, January 8, 1982 (EIGHTY-two), page F-1

	CIA boss assail high-tech leaks

[The Washington Post]

WASHINGTON -- Adm. Bobby R. Inman, deputy director of the CIA,
Thursday predicted a "tidal wave" of public outrage and laws
restricting scientists if scientists do not agree to voluntary
"review" of their work by intelligence agencies.

Scientists had better cooperate in making some of their papers secret
voluntarily, or they will face tough laws restricting them, Inman told
a panel session at the annual meeting of the American Association for
the Advancement of Science.

Scientists should beware that there are congressional investigations
now in progress that will point up the "thoroughly documented" fact
that in the buildup of Soviet defense capability "the bulk of new
technology which they have employed has been acquired from the United
States," Inman said.

When the details of this "hemorrhage of the country's technology" come
out in public, Inman said, there will be a "tidal wave" of public
outrage that will lead to laws restricting the publication of
scientific work that the government might consider "sensitive" on
national security grounds.

"The tides are moving, and moving fast, toward legislated solutions
that in fact are likely to be much more restrictive, not less
restrictive, than the voluntary" censorship system he has suggested,
Inman said.

When he was director of the National Security Agency, the codemaking
and breaking intelligence agency, Inman led an effort to get prominent
private researchers to submit their papers on the mathematical theory
of codes to his agency before publication.

The NSA also briefly put secrecy orders on some of the private code
research in recent years.

But in April 1981, cooperation among the National Science Foundation,
the American Council on Education and the NSA resulted in a voluntary
review system under which scientists can submit their papers to NSA
for review and receive a judgement on whether they possibly contain
information damaging to the national security.

Since then, about 25 papers have been reviewed and none had problems,
according to Daniel Schwartz, until recently chief counsel for the NSA.

"There are other fields where publication of certain information could
affect the national security in a harmful way," Inman said.  These
include the fields of "computer hardware and software, other
electronic gear and techniques, lasers, crop projections and
manufacturing procedures."

	-------

The above news article ran twelve years ago.  His tidal wave of
crypto censorship didn't appear; instead, a wave of support for free
expression ran through the scientific and library community.  About
half of the technical societies amended their by-laws to disallow
closed or censored meetings or conferences.

I wonder if Admiral Inman feels the same way today, as he faces Senate
confirmation hearings as Secretary of Defense.  Will we be seeing the
same sort of proposals?  How does he feel about export controls on
cryptographic software?  What should be done with the Skipjack
program?  If you wonder too, please ask your Senator to ask him about
it.  Seriously.

	John

{% endraw %}
```

## Thread

+ Return to [January 1994](/archive/1994/01)

+ Return to "[gnu (John Gilmore)](/authors/gnu_john_gilmore_)"
+ Return to<EMAIL_ADDRESS>(Fred Heutte)](/authors/phred_at_well_sf_ca_us_fred_heutte_)"

+ 1994-01-10 (Mon, 10 Jan 94 02:51:20 PST) - Twelve year flashback to Admiral Bobby Inman - _gnu (John Gilmore)_
  + 1994-01-11 (Tue, 11 Jan 94 02:06:42 PST) - [Re: Twelve year flashback to Admiral Bobby Inman](/archive/1994/01/c8f8242e8c6e5002ecb104dc67815f6593a2be4fca768c719fa053b44ce1f488) -<EMAIL_ADDRESS>(Fred Heutte)_

gha_license_id: MIT


text:
# **Currency Exchanger**
By _Olha Hizhytska_

Using this application user can type in an amount (in U.S. dollars) and then choose which currency it should be converted to (such as Ukrainian Hryvnia, Euro, Canadian Dollar, Armenian Dram, New Zealand Dollar). To determine the most recent exchange rate, this application will make an API call to the following [exchange rate API](https://www.exchangerate-api.com/). Then the user will be able to see the total amount they entered in the converted currency.

![Homepage](src/assets/currencyscreenshot.png)

## Description

This application demonstrates:
-  practicing making API calls using fetch method.
- creating modern development environments, using module bundling with Webpack, dependencies are managed using npm.
- using error handling and displaying errors in the DOM in case of server error, empty input fields and not available currency to convert.
- working with environment variable, storing API keys in it.
- correctly parsing data from an API response

## Setup

- Clone this repository to your desktop, using next command:

  git clone https://github.com/OlgaHi/Currency-Exchanger.git

- Navigate to the top level of the directory.

  cd Currency-Exchanger

- Open in text editor to view code:
  -  Run the command `npm run install` to install dev dependencies

  - Run the command `npm run start` to open in a local server

- To be able to use this application you will need to use your own API key.
  - Steps to follow:
     - go to [ExchangeRate API site](https://www.exchangerate-api.com/) and set up a developer account, by inputting your email address and clicking the "Get FREE Key" button.
     - please fill the form with your email, first name and password. Accept terms of use and click "Get Started!"
     - you will see in a dashboard your API key.
     - create file .env in the top level of the directory.
     - add your key into the .env file using variable: API_KEY=[your API key] (without spacing).
     - in a .gitignore file add .env file.

## Technologies Used:

* JavaScript
* HTML/CSS/Bootstrap
* jQuery
* Node
* Npm
* Webpack
* [ExchangeRate-API](https://www.exchangerate-api.com/)

## Known Bugs

- No known issues

## License

[MIT](https://en.wikipedia.org/wiki/MIT_License)
Copyright (c) 2021 _Olha Hizhytska_

## Contact Information

- Olha Hizhytska<EMAIL_ADDRESS>

repository: OlgaHi/Currency-Exchanger

repo_name: OlgaHi/Currency-Exchanger


text:
# android-nd-location-reminder

Location Reminder app for the Android Developer Nano-Degree course. This is the 4th project
for the Udacity Android Kotlin Developer Nanodegree Program.

Android LocationReminder is a TODO list app with location reminders that reminds the
user to do something when the user is at a specific location. The app will require the
user to create an account and login to set and access reminders.

The app consists of following screens:

- welcome screen which guides the user to the login screen
- login screen to login or register using email or Google account
- main screen displaying a list of reminders
- create reminder screen to create a new reminder and allows to set a location on a Google maps
- details screen which shows a reminder's details.

This app makes use of following Android features:

1. DI using Koin
2. Firebase authentication and FirebaseUI
3. Google maps
4. MVVM architecture
5. Android Architecture components such as ViewModels and LiveData
6. Room database for persistence
7. Runtime permission handling
7. User notifications
8. Use of Espresso and Mockito to test the app including DAO (Data Access Object) and Repository
   classes and End-To-End testing for the Fragments navigation.

## Firebase API and Google maps API

Firebase API and Google maps API require API keys for access and use.
For privacy reasons, these keys are not included in the public code repository.

In order to build the project, you need to

1. add your your own `google-services.json` file to `app/` directory
2. provide your own Google maps API key as described below.

The build of this app is configured in a way that it uses a build configuration field for
providing the necessary API key in the source code. It requires the definition of a string
with name `GOOGLE_MAPS_API_KEY` in a local `gradle.properties` file (e.g. in `~/.gradle/gradle.properties`).
Alternatively, you can override the value of the defined `resValue` directly in `app/build.gradle`
(not recommended).

## Screenshots

<img src="https://raw.githubusercontent.com/jploz/android-nd-location-reminder/main/screenshots/Screenshot_060852042.png" width="480"/>

repository: jploz/android-nd-location-reminder

repo_name: jploz/android-nd-location-reminder


text:
[Dockerfile][] | [requirements.txt][] |[myapp.py][] | [Authors][] | [License][]

This demo will read data from Kafka server that is running on Monasca machine and then it will push those data to "demo1" topic on Kafka machine. This way Policy Manager will listen for an event to met a given condition from Policy.

If this condition is met *Policy Manager* will trigger an action with fixed parameters.

# Warning

All ips shown in this example are not updated, they are example ones.
Each time CogNet infrastructure is deployed different servers get new dynamic ips. There is an automatically updated list of those ips/hosts combination in [hosts](https://github.com/CogNet-5GPPP/CogNet_services/blob/master/hosts) file from CogNet_services repository.

# Docker components
Following we are going to explain the content and working of the components of the Docker:

The components of this Docker will be 3 files:

  * Dockerfile
  * requirements.txt
  * myapp.py

# Dockerfile

Is the main configuration file from Dockers, there it will be all Docker configuration, in our case it will be:

```
FROM python:2-onbuild

USER root

################################################
#Your python application to be launched
COPY myapp.py /src/myapp.py
#################################################

#Run our application at startup
CMD ["/usr/bin/python","/src/myapp.py"]
```

This docker file will create a docker image based on python 2 and copy:

  * myapp.py

file to the Docker image.

In final part of this file we have set the starting comand to be launched when we run the Docker.

# requirements.txt

This file will contain python-pip dependencies. Here we can add all python dependencies that can be installed via *pip* that we want to use in our python application. We will run a kafka-python based aplication so content should be:

```
kafka-python
```

# myapp.py

This file will be our python application and the place where our source code must be.

This application will execute the following steps

  1. Prepare python Kafka dependencies
  2. Connect to **metrics** topic from kafka server to read metrics
  3. Connect to Kafka server in producer mode to push procesed data
  4. Read policy from json file
  6. Send policy to "newpolicy" topic in order to create a new policy
  5. Read metrics from Kafka
  6. Process metrics
    1. Read received metrics
    2. Get kind of metric that had been received
    3. Check if that is the kind of data we need
    4. Set data value to the policy definition
  7. Send policy with new value to **newtopic** defined in policy to Kafka

```
from kafka import KafkaConsumer
from kafka import KafkaProducer
from kafka.errors import KafkaError
import json

# To consume latest messages from metrics topic
consumer = KafkaConsumer('metrics',bootstrap_servers=["monasca:9092"])

# To produce new messages to kafka
producer = KafkaProducer(bootstrap_servers=["kafka:9092"])

# Read newpolicy.json
with open('newpolicy.json') as json_data:
    json_policy = json.load(json_data)

#Read topicName from policy
topicName=json_policy['supa-policy']['supa-policy-target']['topicName']

#Push new policy to Kafka
future = producer.send('newpolicy',json.dumps(json_policy))

# Receive messages from kafka metrics topic
for message in consumer:
                # do something with received messages

        #load each message as json data
        data = json.loads(message.value)

        #get type of metric of the message
        value_name=data['metric']['name']

        #check that that metric is the metric we need
        if value_name == "cpu.user_perc":

          #get metric value
          value_data=data['metric']['value']

          #append that value to the previous defined policy
          json_policy['supa-policy']['supa-policy-statement']['event']['event-value']=value_data

          #Send that policy as new measure to the listening topicName topic
          future = producer.send(topicName,json.dump(json_policy))
        #else:
          #print("Not valid data")
```

### Authors

- Angel Martin<EMAIL_ADDRESS>- Felipe Mogollon<EMAIL_ADDRESS>
### License

Copyright 2016 Vicomtech.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

[Dockerfile]: #dockerfile
[requirements.txt]: #requirementstxt
[myapp.py]: #myapppy
[start.sh]: #startsh
[Authors]: #authors
[License]: #license


text:
# Roadmap

## Overview

The initial roadmap of CRI-O was lightweight and followed the main
Kubernetes Container Runtime Interface (CRI) development lifecycle.
This is partially because additional features on top of that are either integrated
into a CRI-O release as soon as they’re ready, or are tracked through the
Milestone mechanism in GitHub.
Another reason is that feature availability is mostly tied to Kubernetes releases,
and thus most of its long-term goals are already tracked in [SIG-Node](https://github.com/kubernetes/community/blob/master/sig-node/README.md)
through the Kubernetes Enhancement Proposal (KEP) process.
Finally, CRI-O’s long-term roadmap outside of features being added by SIG-Node
is in part described by its mission:
to be a secure, performant and stable implementation of the Container Runtime Interface.
However, all of these together do construct a roadmap,
and this document will describe how.

## Milestones

CRI-O’s primary internal planning mechanism is the
Milestone feature in GitHub, along with Issues.
Since CRI-O releases in lock-step with Kubernetes minor releases,
where the CRI-O community aims to have a x.y.0 release released within three days
after the corresponding Kubernetes x.y.0 release, there is a well established
deadline that must be met.
For PRs and Issues that are targeted at a particular x.y.0 release can be added
to the x.y Milestone and they will be considered for priority in leading up
to the actual release.
However, since CRI-O’s releases are time bound and partially out of the CRI-O
communities’ control, tagging a PR or issue with a Milestone does not guarantee
it will be included.
Users or contributors who don’t have permissions to add the Milestone can
request an Approver to do so.
If there is disagreement, the standard [Conflict resolution and voting](https://github.com/cri-o/cri-o/blob/main/GOVERNANCE.md#conflict-resolution-and-voting)
mechanism will be used.
Pull requests targeting patch release-x.y branches are not part of any milestone.
The release branches are decoupled from the [Kubernetes patch release cycle](https://k8s.io/releases/patch-releases)
and fixes can be merged ad-hoc.
The support for patch release branches follows the yearly Kubernetes period and
can be longer based on contributor bandwidth.

## SIG-Node

CRI-O’s primary purpose is to be a CRI compliant runtime for Kubernetes, and thus
most of the features that CRI-O adds are added to remain conformant to the CRI.
Often, though not always, CRI-O will attempt to support new features in Kubernetes
while they’re in the Alpha stage, though sometimes this target is missed and
support is added while the feature is in Beta.
To track the features that may be added to CRI-O from upstream, one can watch
[SIG-Node’s KEPs](https://github.com/kubernetes/enhancements/pulls?q=is%3Apr+is%3Aopen+label%3Asig%2Fnode)
for a given release.
If a particular feature interests you, the CRI-O community recommends you open
an issue in CRI-O so it can be included in the Milestone for that given release.
CRI-O maintainers are involved in SIG-Node in driving various upstream initiatives
that can be tracked in the [SIG-Node planning document](https://docs.google.com/document/d/1U10J0WwgWXkdYrqWGGvO8iH2HKeerQAlygnqgDgWv4E/edit?usp=sharing).

## Long-term, CRI-O Specific Features

There still exist features that CRI-O will add that exist outside of the purview
of SIG-Node, and span multiple releases.
These features are usually implemented to fulfill CRI-O’s purpose of being secure,
performant, and stable.
As all of these are aspirational and seek to improve CRI-O structurally, as opposed
to fixing a bug or clearly adding a feature, it’s less appropriate to have
an issue for them.
As such, updates to this document will be made once per release cycle.
Finally, it is worth noting that the integration of these features will be deliberate,
slow, and strictly opted into.
CRI-O does aim to constantly improve, but also aims to never compromise its stability
in the process.
Some of these features can be seen below:

### Initiatives on CRI-O’s Roadmap

- Improve upstream documentation
- Automate release process
- Improved seccomp notification support
- Increase pod density on nodes:
  - Reduce overhead of relisting pods and containers (alongside [KEP 3386](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/3386-kubelet-evented-pleg/README.md))
  - Reduce overhead of metrics collection (alongside [KEP 2371](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/2371-cri-pod-container-stats/README.md))
  - Reduce process overhead of multiple conmons per pod (through [conmon-rs](github.com/containers/conmon-rs))
  - Improve maintainability by ensuring the code is easy to understand and follow
  - Improve observability and tracing internally
- Evaluate rust reimplementation of different pieces of the codebase.

## Known Risks

- Relying on different SIGs for CRI-O features:
  - We have a need to discuss our enhancements with different SIGs to get all
    required information and drive the change. This can lead into helpful, but maybe
    not expected input and delay the deliverable.

- Some features require initial research:
  - We're not completely aware of all technical aspects for the changes. This means
    that there is a risk of delaying because of investing more time in pre-research.

repository: cri-o/cri-o

gha_language: Go

repo_name: cri-o/cri-o


text:
# TEAS NS DT DESIGN TEAM CALL, MAY 14TH, 2020

Notes by Jari Arkko and Jie Dong.

## PARTICIPANTS

* Jari Arkko
* Dhruv Dhody
* Eric Gray
* Xufeng Liu
* Jeff Tantsura
* Bo Wu
* Kiran Makhijani
* Rakesh Gandhi
* Jie Dong
* Shunsuke Homma
* Luis M. Contreras
* Reza Rokui
* John Drake
* Sergio Belotti
* ...

## AGENDA

* Continuing discussion to resolve issues
* Any other business
* Closing

## DISCUSSION

### ACTN Picture and Framework document

The reference to mailing list discussion is https://mailarchive.ietf.org/arch/msg/teas-ns-dt/xCd_iH7H_xuHPA3rv7Z-L3rb5qk/

The discussion will continue on email.

### Isolation discussion from last time

Jari stated that the 4.1 changes to characteristics and changing them to use the "confidence level" language seemed to have broad support in the group. Kiran has alread implemented these in the newest version of the draft.

Jari also stated that the 4.1.1 changes were much more controversial. There was a discussion of whether the change of moving the text to appendix is sufficient or not. Jari said that the content matters more than the location. We need the group to have a buy-in for the document.

Reza asked if we can use the reliability term. Jeff believes that we mix things, from observable to non-observable things. Reliability isn't as easily measurable quantity.

Reza asks if could just delete the non-measurable parts.

Kiran has made a proposal to distinguish measureable and observable characteristics under Section 4.1.

Eric thinks there is no fundamental difference between measurable and observable.

Jie thinks that whether something is measurable depends on the tools we have, and not all the requirements are measurable (e.g. availability, security). He also mentions that the meanning of isolation is multi-dimension, for example: address space isolation (as one requirement in existing VPNs), security isolation, performance isolation, etc. Perhaps it could be described in each dimension separately, so that we could make it clearer step by step.

### Eric's annoted version of the definitions document

The reference to mailing list discussion and the annoted document are TBD and TBD

Eric's issues and suggested changes include

- Revision of the introduction for correctness

Characteristics issues:
- Wrong definitions for some characteristics, e.g., latency should be maximum latency
- Eric thinks resource redundancy and other items are off the mark, because customer who wants redundancy they will ask for two separate things.
- Same for security.
- Resolution of guarantee was objectionable, but Kiran's new version seemed workable.
- Take the whole section about isolation out, as a philosophical discussion.

Endpoints:
- Current text conflates link termination points and transports.
- Do we need the connectivity patterns section?
- Horizontal and vertical slicing discussion is on how rather than what.

Other:
- Other rewording

Jeff and Kiran disagreed with some of Eric's points (e.g., about security).

Eric suggests that authors take a look at his comments.

Discussion will need to continue on the mailing list.

Transport term was discussed but again the design team feels that the substance of the definition is important and we can use the term we want as long as the substance is there to define it.

### Kiran's 2-3 placeholders for modification

The reference to mailing list discussion thread is https://mailarchive.ietf.org/arch/msg/teas-ns-dt/OtRaQR9hlWdVOTtKj24ajWrzo3I/

## ANY OTHER BUSINESS

## CLOSING

The meeting was closed 17.58 CET.

repository: IETF-TEAS-WG/teas-ns-dt

repo_name: IETF-TEAS-WG/teas-ns-dt