text:
---
layout: default
title: 使用DDS來開發ROS
permalink: articles/ros_on_dds.html
abstract:
  這篇文章討論如何使用DDS來開發ROS 2.0，列出這種作法的優缺點，並討論使用者經驗和API帶來的影響。我們也會針對我們實作的prototype "ros_dds"給一些總結意見，並探討其中潛藏的問題。
author: '[William Woodall](https://github.com/wjwwood)'
published: true
translator: 賴柏任
categories: Middleware
---

{:toc}

# {{ page.title }}

This article was written at a time before decisions were made to use DDS and RTPS as the underlying communication standards for ROS 2.
For details on how ROS 2 has been implemented, see the [Core Documentation](http://docs.ros2.org/)

{{ page.abstract }}

Original Author: {{ page.author }}

Translator: {{ page.translator }}

專業術語:

- [Data Distribution Service (DDS)](http://en.wikipedia.org/wiki/Data_Distribution_Service)
- [Real-Time Publish Subscribe (RTPS)](https://en.wikipedia.org/wiki/Real-Time_Publish-Subscribe_(RTPS)_Protocol)
- The [Object Management Group (OMG)](http://www.omg.org/)
- OMG [Interface Description Language (IDL)](http://www.omg.org/gettingstarted/omg_idl.htm) | [Formal description](http://www.omg.org/cgi-bin/doc?formal/2014-03-01)

## 為什麼考慮DDS?

當我們在探索可以用於開發下一代ROS的通訊系統的選項時，最先考慮的是改進ROS 1.x的傳輸機制或是使用現有的函式庫來建構(例如使用ZeroMQ、Protocol Buffer和zeroconf)。但是，若採取上述這些選項，我們要不是得從頭打造整個中介軟體、不然就是要從各個部分開始組合，所以我們也考慮直接使用功能完整的其他中介軟體來打造ROS 2.0。經過我們的調查之後，其中一個脫穎而出的選項就是DDS。

### 一個完整的中介軟體

使用一個像DDS這種完整的中介軟體來開發的好處是，我們需要維護的程式碼少了很多，而且這種中介軟體的行為和詳細規格都已經被詳細地撰寫成文件。除了系統層級的說明文件之外，DDS也有推薦的使用個案和軟體API。有了這些具體的說明文件和規格，第三方的開發者就可以檢驗、審視，甚至可以根據自己的需求來實作具備不同程度的互動性的版本。這項特色是ROS從未有過的(ROS只有少量的基礎說明文件和一些可供參考的實作版本)，而且，就算使用現有的一些函式庫來建構起ROS，對於規格和軟體行為的詳細說明文件還是不可或缺的。

當然，這種開發方式也有缺點，就是ROS 2.0也會被DDS的設計所限制。如果DDS的設計不夠用於解決我們想要的使用情境或是不夠彈性，我們就必須去更動DDS的設計。某種程度上來說，採用完整的中介軟體來開發也包含了要採用這個中介軟體的開發哲學和社群文化，這件事也不應被輕忽。

## DDS是什麼?

DDS提供跟ROS 1.x很相似的一種訂閱-發佈傳輸機制。它使用[Object Management Group (OMG)](http://www.omg.org/)定義的“Interface Description Language (IDL)”來實作訊息的定義和序列化。雖然DDS還沒有提供request-response的傳輸機制(這可以被當作ROS的service系統)，但這種傳輸方式的實作已經有初步規劃，第一個Beta版也已經在2015年春天發佈(稱作 [DDS-RPC](http://www.omg.org/spec/DDS-RPC/))。

DDS提供的預設discovery系統是一種分散式的discovery系統(這有別於ROS 1.x必須由master來掌管所有node的方式)，其中也使用DDS的發佈-訂閱傳輸機制。這使得任意兩個DDS程式可以不透過ROS master這種機制就能相互溝通。這使整個系統的容錯程度更高、也更彈性。而且，我們不一定要使用動態的discovery機制，有幾種DDS的實作版本都提供了靜態discovery機制的選項。

### DDS是怎麼出現的?

DDS起源自一群擁有提供相似中介軟體的公司，因為這些公司的客戶希望這些公司提供的中介軟體可以彼此溝通，所以這些中介軟體漸漸地被整合起來形成一套標準。DDS的標準是由Object Management Group建立起來的，跟建立UML、CORBA和SysML和其他軟體標準的是同一群人。這可能是好消息也可能是壞消息，看你怎麼想。一方面，你擁有一個標準制定委員會，他們會不斷討論而且在軟體工程社群擁有極大的影響力；但另一方面，他們前進地相對緩慢、對於改變的適應力也較差，所以也很可能無法跟上軟體工程界的最新潮流。

DDS一開始是幾個相似的中介軟體，後來因為彼此變得太相近，制定一個統一的標準來整合這些中介軟體才變得合理。在這樣的脈絡之下，雖然DDS規格的制定是由委員會撰寫出來的，但他的演進原本就相當貼近使用者的需求。由於在標準制定之前就已經是藉由跟使用者的互動來演進，這多少可以消弭大家對於DDS的擔心，擔心它只是一個憑空想像出來的標準、但不符合實際應用的需求。歷史上也曾有一些標準委員會所制訂出的標準，雖然立意良善且制訂得很完整，但沒人想要使用或是不符合使用者需求。不過DDS看起來沒有這種問題。

除了上述的擔憂，也有人擔心DDS將會是一個不更新的標準，這種刻板印象來自於UML和CORBA給人的不良回憶，因為這兩者也都是OMG所制定的標準。不過DDS看起來不會有這種問題，除了最近仍持續在更新之外，現在也還在新增更多的規格，包含websockets、SSL安全協議、可擴充的型別、request-response傳輸機制以及C++11風格的API(用來取代現在的C++介面)。對我們來說，DDS的標準仍持續地演進是一件很正面的事，雖然跟現今軟體工程的潮流比起來仍相對緩慢，但它依然在往滿足其使用者需求的方向在前進。

### DDS技術上的可信度

有許多要求高精密度的系統都已經在使用DDS，它已經被用在:

- 軍艦
- 大型公共建設(例如水壩)
- 金融系統
- 航太系統
- 飛行系統
- 火車控制系統

除了這些使用情境之外，還有許多同等重要的不同系統都已經在使用DDS，這些成功的案例使得DDS設計的穩定度和彈性更加被信任。

DDS不僅滿足了這些使用情境下的需求，我們進一步訪問了政府及NASA的使用者(其中有些人也是ROS的使用者)，他們對於DDS的穩定性和彈性都讚譽有加。不過他們也提醒，DDS具備高彈性的代價就是高複雜度，它在API以及設定上的複雜度是我們應用DDS來開發ROS時需要注意的問題。

DDS的連接協議(DDSI-RTPS)具有極高的彈性，使得DDS可以被使用在需要高穩定性、高層次的的系統整合，也可以被real-time地執行在嵌入式裝置上。有幾個DDS函式庫的提供者都各自實作了用於嵌入式裝置的DDS，也都各自誇耀自己的實作在函式庫佔用的空間和記憶體使用量只需幾百KB。而由於DDS使用UDP傳輸協儀來實作，它不必依賴穩固的硬體或網路來傳輸。不過這也表示DDS需要自己重新建立確保資料傳輸不會遺漏的機制(基本上就是TCP機制，可能多幾個或少幾個功能而已)，好處是DDS較容易被移植到不同裝置上(因為較不穩定的硬體也能使用)、且對軟體行為的控制權增加。在DDS的實作中，可以藉由Quality of Service(QoS)來控制決定穩定性的參數，讓使用者有最高的彈性來控制傳輸時的行為。舉例來說，如果你希望程式執行滿足soft real-time的要求，那網路延遲(latency)就是一個需要考慮的問題，你可以將DDS設定成只使用UDP來傳輸。在另一種情形中，你可能希望DDS像TCP一樣可以可靠地傳送正確的資料，那就可以透過DDS的QoS參數來調整這些行為。

雖然DDS的實作預設是使用UDP傳輸協議，而且只要求這種等級的傳輸功能，OMG仍然在1.2版的DDS標準中新增了DDS使用TCP的部分。稍微survey一下，就會發現有兩個DDS實作版本的提供者(RTI跟PrismTech)都提供用TCP實作的DDS。

RTI的網站上有一段話([http://community.rti.com/kb/xml-qos-example-using-rti-connext-dds-tcp-transport](http://community.rti.com/kb/xml-qos-example-using-rti-connext-dds-tcp-transport)):

> RTI Connext DDS預設使用UDPv4和共享記憶體的方式來跟其他的DDS應用溝通。不過在某些情況下，discovery和資料傳輸時需要用到TCP的傳輸方式。如果你想知道更多關於RTI的TCP傳輸方式，可以去看RTI Core Libraries and Utilities User Manual裡面的 “RTI TCP Transport” 章節。

PrismTech的規格說明書上也寫了他們支援DDSI-RTPS1.2版裡的TCP部分:

[http://www.prismtech.com/opensplice/products/opensplice-enterprise/opensplice-dds-core](http://www.prismtech.com/opensplice/products/opensplice-enterprise/opensplice-dds-core) (search for TCP)
=======
From RTI's website ([http://community.rti.com/kb/xml-qos-example-using-rti-connext-dds-tcp-transport](http://community.rti.com/kb/xml-qos-example-using-rti-connext-dds-tcp-transport)):

> By default, RTI Connext DDS uses the UDPv4 and Shared Memory transport to communicate with other DDS applications.
> In some circumstances, the TCP protocol might be needed for discovery and data exchange.
> For more information on the RTI TCP Transport, please refer to the section in the RTI Core Libraries and Utilities User Manual titled "RTI TCP Transport".

From PrismTech's website, they support TCP as of OpenSplice v6.4:
>>>>>>> 42d8775846c51ae520960dd874923d294d5e96b4

[http://www.prismtech.com/vortex/vortex-opensplice/communication/ddsi2](http://www.prismtech.com/vortex/vortex-opensplice/communication/ddsi2)

### 發佈者跟授權條款

OMG跟幾間公司一起制定了DDS的標準規格，這幾間公司也同時是DDS實作版本的發佈者，幾間比較知名的有:

- RTI
- PrismTech
- Twin Oaks Software

這些公司提供的實作版本在實作策略和授權條款上都有些差異，OMG列出了一份DDS發佈者的[清單](http://dds-directory.omg.org/vendor/list.htm)。

RTI實作的Connext DDS是使用他們自訂的[Community Infrastructure License](https://www.rti.com/downloads/IC-license.html)來授權，這個授權條款符合ROS社群的需求，不過若要採用RTI的Connext DDS當作ROS主要使用的DDS實作版本，仍需要進一步的討論。我們這裡所說的"符合ROS社群的需求"的意思是，雖然Community Infrastructure License並不是[OSI](http://opensource.org/licenses)認可的授權條款，但是有研究指出這個授權條款還是可以允許ROS以[BSD](http://www.openfoundry.org/tw/legal-column-list/524--bsd)授權條款的形式被散佈，而且ROS的使用者可以任意地以原始碼或執行檔的形式散布自己重製的版本。RTI看起來也願意對授權條款進行溝通協調以滿足ROS社群的需求，不過這當然需要RTI和ROS社群間的來回討論來確保這件事情可行。跟其他發佈者相似的是，RTI的授權條款授權的部分只有函式庫的核心功能，也就是基本的DDS API，但其他像是開發工具或是introspection工具都是RTI私有的。此外，RTI看起來擁有最多的使用者。
=======
In addition to vendors providing implementations of the DDS specification's API, there are software vendors which provide an implementation with more direct access to the DDS wire protocol, RTPS.
For example:

- eProsima

These RTPS-centric implementations are also of interest because they can be smaller in scope and still provide the needed functionality for implementing the necessary ROS capabilities on top.

RTI's Connext DDS is available under a custom "Community Infrastructure" License, which is compatible with the ROS community's needs but requires further discussion with the community in order to determine its viability as the default DDS vendor for ROS.
By "compatible with the ROS community's needs," we mean that, though it is not an [OSI-approved license](https://opensource.org/licenses), research has shown it to be adequately permissive to allow ROS to keep a BSD style license and for anyone in the ROS community to redistribute it in source or binary form.
RTI also appears to be willing to negotiate on the license to meet the ROS community's needs, but it will take some iteration between the ROS community and RTI to make sure this would work.
Like the other vendors this license is available for the core set of functionality, basically the basic DDS API, whereas other parts of their product like development and introspection tools are proprietary.
RTI seems to have the largest on-line presence and installation base.
>>>>>>> 42d8775846c51ae520960dd874923d294d5e96b4

PrismTech的DDS實作版本OpenSplice是用LGPL授權條款來發佈，跟其他著名的開源函式庫是相同的，例如glibc、ZeroMQ跟Qt。可以直接在Github上找到:

[https://github.com/PrismTech/opensplice](https://github.com/PrismTech/opensplice)

PrismTech的實作版本還包含基本的[編譯功能](http://www.opensplice.org/dds-community/building)，而且相對容易被包成package。OpenSplice的使用者人數看起來是第二多的，不過很難判斷到底是不是真的如此。

TwinOaks實作的CoreDX DDS完全是私有的，而且他們專住在最精簡的實作以利讓CoreDX DDS可以在嵌入式裝置，甚至一塊開發板上就能運行。

<<<<<<< HEAD
既然擁有LGPL授權的OpenSplice以及RTI自訂的授權條款(可以進一步溝通協調)這些選項，使用現成的DDS實作版本或把它當作相依的函式庫重新發佈看起來都是相當可行的。我們在設計ROS 2.0的其中一個目標就是讓DDS的實作版本可以被替換，舉例來說，假設預設的DDS實作版本是RTI的Connext DDS，如果有人想要使用OpenSplice，那只要更改一個選項、重新編譯ROS原始碼，就可以使用OpenSplice。

![DDS and ROS API Layout](../img/ros_on_dds/api_levels.png "DDS and ROS API Layout")

這件事之所以可行是因為DDS的標準規格中已經制定了它的API，所以不同實作版本的API是一致的。有人做過較深入的研究確定這件事是可行的，雖然不同實作版本之間還是有一些細微差距，例如回傳的形別不同(pointer或是shared_ptr這種差別)跟標頭檔的規劃方式。
=======
eProsima's FastRTPS implementation is available on GitHub and is LGPL licensed:

[https://github.com/eProsima/Fast-RTPS](https://github.com/eProsima/Fast-RTPS)

eProsima Fast RTPS is a relatively new, lightweight, and open source implementation of RTPS.
It allows direct access to the RTPS protocol settings and features, which is not always possible with other DDS implementations.
eProsima's implementation also includes a minimum DDS API, IDL support, and automatic code generation and they are open to working with the ROS community to meet their needs.

Given the relatively strong LGPL option and the encouraging but custom license from RTI, it seems that depending on and even distributing DDS as a dependency should be straightforward.
One of the goals of this proposal would be to make ROS 2.0 DDS vendor agnostic.
So, just as an example, if the default implementation is Connext, but someone wants to use one of the LGPL options like OpenSplice or FastRTPS, they simply need to recompile the ROS source code with some options flipped and they can use the implementation of their choice.
>>>>>>> 42d8775846c51ae520960dd874923d294d5e96b4

This is made possible because of the fact that DDS defines an API in its specification.
Research has shown that making code which is vendor agnostic is possible if not a little painful since the APIs of the different vendors is almost identical, but there are minor differences like return types (pointer versus shared_ptr like thing) and header file organization.

### 社群及其文化

<<<<<<< HEAD
DDS從一群老公司的產品中產生出來，由OMG這個老派的軟體工程標準制定組織規劃，並大量被使用在政府跟軍事單位，所以DDS的使用者社群看起來跟ROS或是ZeroMQ這些較新的軟體社群相當不同，也就不令人感到意外。雖然RTI看起來有很多使用者，但社群裡的使用者所問的問題幾乎都是由RTI的員工回答的(而非其他使用者熱心參與回答)。而且雖然在技術上是開源的，但RTI的Connext或PrismTech的OpenSplice都沒有提供Ubuntu的apt或是Homebrew這些較流行的套件管理工具的package，也沒有由大量使用者撰寫的wiki或活躍的github repository。

DDS和ROS在社群文化上的巨大差異，對於採用DDS來開發ROS 2.0是最需要擔心的問題之一，跟繼續使用rostcp或使用ZeroMQ這兩種選項的重要差異在於，看起來並沒有很多使用者依賴DDS。但是DDS函式庫的發佈者在我們研究的過程中，一直很積極地回應我們的問題。不過如果是由廣大的ROS社群來問問題，這種積極回應的態度能否持續就很難說了。

雖然社群文化這個議題在決定是否採用DDS的過程中應該要被考慮，但也不能把這個議題的重要程度看得比DDS在技術層面的優缺點還要高。

## 建立在DDS之上的ROS

我們的目標是希望DDS變成ROS 2.0的實作細節，換句話說，DDS相關的API跟訊息格式的定義都需要被隱藏起來(如果你是ROS 2.0的使用者，完全不需要了解DDS的API)。DDS提供了Discovery、訊息格式的定義、訊息序列化和發佈-訂閱的傳輸機制，因此，我們至少會利用DDS的discovery、發佈-訂閱的傳輸機制和訊息序列化來開發ROS。ROS 2.0會提供跟ROS 1.x類似的介面，這個介面建立在DDS之上，所以對於大部分的ROS使用者來說，並不需要接觸到DDS複雜的部分。但同時，我們也會另外提供方法來更換底層的DDS函式庫，讓有特殊需求或需要使用其他DDS實作版本的使用者擁有高彈性的發揮空間。要更換底層的DDS函式庫，需要額外的package來處理dependency的問題，如此一來，使用者只需要看這個額外的package的dependency，就可以知道現在是用哪個DDS的實作版本。在設計建構於DDS函式庫之上的ROS API時，應該要將目標放在滿足ROS使用者社群的需求，因為當使用者開始觸碰到某個特定的DDS實作版本之後，就喪失了使用不同DDS實作版本的可移植性(如果某個package需要更動到底層的DDS函式庫，那要安裝到其他使用者的ROS環境中，就會引起麻煩)。我們之所以讓使用者可以在多種不同的DDS實作版本中切換，並不是為了鼓勵使用者常常更換實作版本，而是要讓使用者可以在有特殊需求的時候，利用不同實作版本的切換彈性來滿足自己的需求，同時也避免掉ROS 2.0在未來會抗拒除了預設的DDS實作版本之外的可能性。所以，ROS 2.0還是會有一個推薦使用、支援最完整的預設DDS實作版本。
=======
DDS comes out of a set of companies which are decades old, was laid out by the OMG which is an old-school software engineering organization, and is used largely by government and military users.
So it comes as no surprise that the community for DDS looks very different from the ROS community and that of similar modern software projects like ZeroMQ.
Though RTI has a respectable on-line presence, the questions asked by community members are almost always answered by an employee of RTI and though technically open source, neither RTI nor OpenSplice have spent time to provide packages for Ubuntu or Homebrew or any other modern package manager.
They do not have extensive user-contributed wikis or an active Github repository.

This staunch difference in ethos between the communities is one of the most concerning issues with depending on DDS.
Unlike options like keeping TCPROS or using ZeroMQ, there isn't the feeling that there is a large community to fall back on with DDS.
However, the DDS vendors have been very responsive to our inquiries during our research and it is hard to say if that will continue when it is the ROS community which brings the questions.

Even though this is something which should be taken into consideration when making a decision about using DDS, it should not disproportionately outweigh the technical pros and cons of the DDS proposal.

## ROS Built on DDS

The goal is to make DDS an implementation detail of ROS 2.0.
This means that all DDS specific APIs and message definitions would need to be hidden.
DDS provides discovery, message definition, message serialization, and publish-subscribe transport.
Therefore, DDS would provide discovery, publish-subscribe transport, and at least the underlying message serialization for ROS.
ROS 2.0 would provide a ROS 1.x like interface on top of DDS which hides much of the complexity of DDS for the majority of ROS users, but then separately provides access to the underlying DDS implementation for users that have extreme use cases or need to integrate with other, existing DDS systems.

![DDS and ROS API Layout](/img/ros_on_dds/api_levels.png)
*DDS and ROS API Layout*

Accessing the DDS implementation would require depending on an additional package which is not normally used.
In this way you can tell if a package has tied itself to a particular DDS vendor by just looking at the package dependencies.
The goal of the ROS API, which is on top of DDS, should be to meet all the common needs for the ROS community, because once a user taps into the underlying DDS system, they will lose portability between DDS vendors.
Portability among DDS vendors is not intended to encourage people to frequently choose different vendors, but rather to enable power users to select the DDS implementation that meets their specific requirements, as well as to future-proof ROS against changes in the DDS vendor options.
There will be one recommended and best-supported default DDS implementation for ROS.
>>>>>>> 42d8775846c51ae520960dd874923d294d5e96b4

### Discovery

DDS可以完全取代以往由master為基礎的discovery系統。取代之後，ROS 2.0可以透過DDS API來取得node的列表、topic的列表，以及他們之間的連接關係。換句話說，使用者不需直接呼叫DDS的API，而是可以呼叫把這些細節都隱藏起來的ROS 2.0 API。

<<<<<<>>>>>> 42d8775846c51ae520960dd874923d294d5e96b4
### Publish-Subscribe Transport

The DDSI-RTPS (DDS-Interoperability Real Time Publish Subscribe) protocol would replace ROS's TCPROS and UDPROS wire protocols for publish/subscribe.
The DDS API provides a few more actors to the typical publish-subscribe pattern of ROS 1.x.
In ROS the concept of a node is most clearly paralleled to a graph participant in DDS.
A graph participant can have zero to many topics, which are very similar to the concept of topics in ROS, but are represented as separate code objects in DDS, and is neither a subscriber nor a publisher.
Then, from a DDS topic, DDS subscribers and publishers can be created, but again these are used to represent the subscriber and publisher concepts in DDS, and not to directly read data from or write data to the topic.
DDS has, in addition to the topics, subscribers, and publishers, the concept of DataReaders and DataWriters which are created with a subscriber or publisher and then specialized to a particular message type before being used to read and write data for a topic.
These additional layers of abstraction allow DDS to have a high level of configuration, because you can set QoS settings at each level of the publish-subscribe stack, providing the highest granularity of configuration possible.
Most of these levels of abstractions are not necessary to meet the current needs of ROS.
Therefore, packaging common workflows under the simpler ROS-like interface (Node, Publisher, and Subscriber) will be one way ROS 2.0 can hide the complexity of DDS, while exposing some of its features.

### Efficient Transport Alternatives

In ROS 1.x there was never a standard shared-memory transport because it is negligibly faster than localhost TCP loop-back connections.
It is possible to get non-trivial performance improvements from carefully doing zero-copy style shared-memory between processes, but anytime a task required faster than localhost TCP in ROS 1.x, nodelets were used.
Nodelets allow publishers and subscribers to share data by passing around `boost::shared_ptr`s to messages.
This intraprocess communication is almost certainly faster than any interprocess communication options and is orthogonal to the discussion of the network publish-subscribe implementation.

In the context of DDS, most vendors will optimize message traffic (even between processes) using shared-memory in a transparent way, only using the wire protocol and UDP sockets when leaving the localhost.
This provides a considerable performance increase for DDS, whereas it did not for ROS 1.x, because the localhost networking optimization happens at the call to `send`.
For ROS 1.x the process was: serialize the message into one large buffer, call TCP's `send` on the buffer once.
For DDS the process would be more like: serialize the message, break the message into potentially many UDP packets, call UDP's `send` many times.
In this way sending many UDP datagrams does not benefit from the same speed up as one large TCP `send`.
Therefore, many DDS vendors will short circuit this process for localhost messages and use a blackboard style shared-memory mechanism to communicate efficiently between processes.

However, not all DDS vendors are the same in this respect, so ROS would not rely on this "intelligent" behavior for efficient **intra**process communication.
Additionally, if the ROS message format is kept, which is discussed in the next section, it would not be possible to prevent a conversion to the DDS message type for intraprocess topics.
Therefore a custom intraprocess communication system would need to be developed for ROS which would never serialize nor convert messages, but instead would pass pointers (to shared in-process memory) between publishers and subscribers using DDS topics.
This same intraprocess communication mechanism would be needed for a custom middleware built on ZeroMQ, for example.

The point to take away here is that efficient **intra**process communication will be addressed regardless of the network/interprocess implementation of the middleware.

### Messages

There is a great deal of value in the current ROS message definitions.
The format is simple, and the messages themselves have evolved over years of use by the robotics community.
Much of the semantic contents of current ROS code is driven by the structure and contents of these messages, so preserving the format and in-memory representation of the messages has a great deal of value.
In order to meet this goal, and in order to make DDS an implementation detail, ROS 2.0 should preserve the ROS 1.x like message definitions and in-memory representation.

Therefore, the ROS 1.x `.msg` files would continue to be used and the `.msg` files would be converted into `.idl` files so that they could be used with the DDS transport.
Language specific files would be generated for both the `.msg` files and the `.idl` files as well as conversion functions for converting between ROS and DDS in-memory instances.
The ROS 2.0 API would work exclusively with the `.msg` style message objects in memory and would convert them to `.idl` objects before publishing.

![Message Generation Diagram](/img/ros_on_dds/message_generation.png "Message Generation Diagram")

At first, the idea of converting a message field-by-field into another object type for each call to publish seems like a huge performance problem, but experimentation has shown that the cost of this copy is insignificant when compared to the cost of serialization.
This ratio between the cost of converting types and the cost of serialization, which was found to be at least one order of magnitude, holds true with every serialization library that we tried, except [Cap'n Proto](http://kentonv.github.io/capnproto/) which doesn't have a serialization step.
Therefore, if a field-by-field copy will not work for your use case, neither will serializing and transporting over the network, at which point you will have to utilize an intraprocess or zero-copy interprocess communication.
The intraprocess communication in ROS would not use the DDS in-memory representation so this field-by-field copy would not be used unless the data is going to the wire.
Because this conversion is only invoked in conjunction with a more expensive serialization step, the field-by-field copy seems to be a reasonable trade-off for the portability and abstraction provided by preserving the ROS `.msg` files and in-memory representation.

This does not preclude the option to improve the `.msg` file format with things like default values and optional fields.
But this is a different trade-off which can be decided later.

### Services and Actions

DDS currently does not have a ratified or implemented standard for request-response style RPC which could be used to implement the concept of services in ROS.
There is currently an RPC specification being considered for ratification in the OMG DDS working group, and several of the DDS vendors have a draft implementation of the RPC API.
It is not clear, however, whether this standard will work for actions, but it could at least support non-preemptable version of ROS services.
ROS 2.0 could either implement services and actions on top of publish-subscribe (this is more feasible in DDS because of their reliable publish-subscribe QoS setting) or it could use the DDS RPC specification once it is finished for services and then build actions on top, again like it is in ROS 1.x.
Either way actions will be a first class citizen in the ROS 2.0 API and it may be the case that services just become a degenerate case of actions.

### Language Support

DDS vendors typically provide at least C, C++, and Java implementations since APIs for those languages are explicitly defined by the DDS specification.
There are not any well established versions of DDS for Python that research has uncovered.
Therefore, one goal of the ROS 2.0 system will be to provide a first-class, feature complete C API.
This will allow bindings for other languages to be made more easily and to enable more consistent behavior between client libraries, since they will use the same implementation.
Languages like Python, Ruby, and Lisp can wrap the C API in a thin, language idiomatic implementation.

The actual implementation of ROS can either be in C, using the C DDS API, or in C++ using the DDS C++ API and then wrapping the C++ implementation in a C API for other languages.
Implementing in C++ and wrapping in C is a common pattern, for example [ZeroMQ](http://zeromq.org/) does exactly this.
The author of [ZeroMQ](http://zeromq.org/), however, did not do this in his new library, [nanomsg](http://nanomsg.org/), citing increased complexity and the bloat of the C++ stdlib as a dependency.
Since the C implementation of DDS is typically pure C, it would be possible to have a pure C implementation for the ROS C API all the way down through the DDS implementation.
However, writing the entire system in C might not be the first goal, and in the interest of getting a minimal viable product working, the implementation might be in C++ and wrapped in C to begin with and later the C++ can be replaced with C if it seems necessary.

### DDS as a Dependency

One of the goals of ROS 2.0 is to reuse as much code as possible ("do not reinvent the wheel") but also minimize the number of dependencies to improve portability and to keep the build dependency list lean.
These two goals are sometimes at odds, since it is often the choice between implementing something internally or relying on an outside source (dependency) for the implementation.

This is a point where the DDS implementations shine, because two of the three DDS vendors under evaluation build on Linux, OS X, Windows, and other more exotic systems with no external dependencies.
The C implementation relies only on the system libraries, the C++ implementations only rely on a C++03 compiler, and the Java implementation only needs a JVM and the Java standard library.
Bundled as a binary (during prototyping) on both Ubuntu and OS X, the C, C++, Java, and C# implementations of OpenSplice (LGPL) is less than three megabytes in size and has no other dependencies.
As far as dependencies go, this makes DDS very attractive because it significantly simplifies the build and run dependencies for ROS.
Additionally, since the goal is to make DDS an implementation detail, it can probably be removed as a transitive run dependency, meaning that it will not even need to be installed on a deployed system.

## The ROS on DDS Prototype

Following the research into the feasibility of ROS on DDS, several questions were left, including but not limited to:

- Can the ROS 1.x API and behavior be implemented on top of DDS?
- Is it practical to generate IDL messages from ROS MSG messages and use them with DDS?
- How hard is it to package (as a dependency) DDS implementations?
- Does the DDS API specification actually make DDS vendor portability a reality?
- How difficult is it to configure DDS?

In order to answer some of these questions a prototype and several experiments were created in this repository:

[https://github.com/osrf/ros_dds](https://github.com/osrf/ros_dds)

More questions and some of the results were captured as issues:

[https://github.com/osrf/ros_dds/issues?labels=task&page=1&state=closed](https://github.com/osrf/ros_dds/issues?labels=task&page=1&state=closed)

The major piece of work in this repository is in the `prototype` folder and is a ROS 1.x like implementation of the Node, Publisher, and Subscriber API using DDS:

[https://github.com/osrf/ros_dds/tree/master/prototype](https://github.com/osrf/ros_dds/tree/master/prototype)

Specifically this prototype includes these packages:

- Generation of DDS IDLs from `.msg` files: [https://github.com/osrf/ros_dds/tree/master/prototype/src/genidl](https://github.com/osrf/ros_dds/tree/master/prototype/src/genidl)

- Generation of DDS specific C++ code for each generated IDL file: [https://github.com/osrf/ros_dds/tree/master/prototype/src/genidlcpp](https://github.com/osrf/ros_dds/tree/master/prototype/src/genidlcpp)

- Minimal ROS Client Library for C++ (rclcpp): [https://github.com/osrf/ros_dds/tree/master/prototype/src/rclcpp](https://github.com/osrf/ros_dds/tree/master/prototype/src/rclcpp)

- Talker and listener for pub-sub and service calls: [https://github.com/osrf/ros_dds/tree/master/prototype/src/rclcpp_examples](https://github.com/osrf/ros_dds/tree/master/prototype/src/rclcpp_examples)

- A branch of `ros_tutorials` in which `turtlesim` has been modified to build against the `rclcpp` library: [https://github.com/ros/ros_tutorials/tree/ros_dds/turtlesim](https://github.com/ros/ros_tutorials/tree/ros_dds/turtlesim).
  This branch of `turtlesim` is not feature-complete (e.g., services and parameters are not supported), but the basics work, and it demonstrates that the changes required to transition from ROS 1.x `roscpp` to the prototype of ROS 2.0 `rclcpp` are not dramatic.

This is a rapid prototype which was used to answer questions, so it is not representative of the final product or polished at all.
Work on certain features was stopped cold once key questions had been answered.

The examples in the `rclcpp_example` package showed that it was possible to implement the basic ROS like API on top of DDS and get familiar behavior.
This is by no means a complete implementation and doesn't cover all of the features, but instead it was for educational purposes and addressed most of the doubts which were held with respect to using DDS.

Generation of IDL files proved to have some sticking points, but could ultimately be addressed, and implementing basic things like services proved to be tractable problems.

In addition to the above basic pieces, a pull request was drafted which managed to completely hide the DDS symbols from any publicly installed headers for `rclcpp` and `std_msgs`:

[https://github.com/osrf/ros_dds/pull/17](https://github.com/osrf/ros_dds/pull/17)

This pull request was ultimately not merged because it was a major refactoring of the structure of the code and other progress had been made in the meantime.
However, it served its purpose in that it showed that the DDS implementation could be hidden, though there is room for discussion on how to actually achieve that goal.

## Conclusion

After working with DDS and having a healthy amount of skepticism about the ethos, community, and licensing, it is hard to come up with any real technical criticisms.
While it is true that the community surrounding DDS is very different from the ROS community or the ZeroMQ community, it appears that DDS is just solid technology on which ROS could safely depend.
There are still many questions about exactly how ROS would utilize DDS, but they all seem like engineering exercises at this point and not potential deal breakers for ROS.

repository: Po-Jen/design

branch_name: refs/heads/gh-pages

repo_name: Po-Jen/design


text:
A hands-off Anomaly Detection Library

[![PyPI version](https://badge.fury.io/py/luminaire.svg)](https://badge.fury.io/py/luminaire)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/luminaire.svg)](https://pypi.org/project/luminaire/)
[![License](http://img.shields.io/:license-Apache%202-blue.svg)](https://github.com/zillow/luminaire/blob/master/LICENSE.txt)
[![build](https://github.com/zillow/luminaire/workflows/Luminaire%20CI/badge.svg)](https://github.com/zillow/luminaire/actions?query=workflow%3A%22Luminaire+CI%22)
[![publish](https://github.com/zillow/luminaire/workflows/Luminaire%20CD/badge.svg)](https://github.com/zillow/luminaire/actions?query=workflow%3A%22Luminaire+CD%22)
[![docs](https://github.com/zillow/luminaire/workflows/Luminaire%20Docs/badge.svg)](https://github.com/zillow/luminaire/actions?query=workflow%3A%22Luminaire+Docs%22)

---
**Table of contents**

- [What is Luminaire](#what-is-luminaire)
- [Quick Start](#quick-start)
- [Time Series Outlier Detection Workflow](#time-series-outlier-detection-workflow)
- [Anomaly Detection for High Frequency Time Series](#anomaly-detection-for-high-frequency-time-series)
- [Contributing](#contributing)
- [Acknowledgements](#acknowledgements)
- [Development Team](#development-team)

## What is Luminaire

[Luminaire](https://zillow.github.io/luminaire/) is a python package that provides ML-driven solutions for monitoring time series data. Luminaire provides several anomaly detection and forecasting capabilities that incorporate correlational and seasonal patterns as well as uncontrollable variations in the data over time.

## Quick Start

Install Luminaire from [PyPI](https://pypi.org/project/luminaire/) using ``pip``

```bash
pip install luminaire
```

Import ``luminaire`` module in python
```python
import luminaire
```

Check out [Luminaire documentation](https://zillow.github.io/luminaire) for detailed description of methods and usage.

## Time Series Outlier Detection Workflow
![Luminaire Flow](docs/assets/luminaire_flow.png)

Luminaire outlier detection workflow can be divided into 3 major components:

### Data Preprocessing and Profiling Component

This component can be called to prepare a time series prior to training an anomaly detection model on it. This step applies a number of methods that make anomaly detection more accurate and reliable, including missing data imputation, identifying and removing recent outliers from training data, necessary mathematical transformations, and data truncation based on recent change points. It also generates profiling information (historical change points, trend changes, etc.) that are considered in the training process.

Profiling information for time series data can be used to monitor data drift and irregular long-term swings.

### Modeling Component

This component performs time series model training based on the user-specified configuration OR optimized configuration (see Luminaire hyperparameter optimization). Luminaire model training is integrated with different structural time series models as well as filtering based models. See [Luminaire outlier detection](https://zillow.github.io/luminaire/tutorial/outlier_batch.html) for more information.

The Luminaire modeling step can be called after the data preprocessing and profiling step to perform necessary data preparation before training.

### Configuration Optimization Component

Luminaire's integration with configuration optimization enables a hands-off anomaly detection process where
the user needs to provide very minimal configuration for monitoring any type of time series data. This step can be combined with the preprocessing and modeling for any auto-configured anomaly detection use case. See [fully automatic outlier detection](https://zillow.github.io/luminaire/tutorial/optimization.html#fully-automatic-outlier-detection) for a detailed walkthrough.

## Anomaly Detection for High Frequency Time Series

Luminaire can also monitor a set of data points over windows of time instead of tracking individual data points. This approach is well-suited for streaming use cases where sustained fluctuations are of greater concern than individual fluctuations. See [anomaly detection for streaming data](https://zillow.github.io/luminaire/tutorial/streaming.html) for detailed information.

## Contributing

Want to help improve Luminaire? Check out our [contributing documentation](CONTRIBUTING.rst).

## Citing

Please cite the following article if Luminaire is used for any research purpose or scientific publication:

*Chakraborty, S., Shah, S., Soltani, K., Swigart, A., Yang, L., & Buckingham, K. (2020, December). Building an
Automated and Self-Aware Anomaly Detection System. In 2020 IEEE International Conference on Big Data (Big Data)
(pp. 1465-1475). IEEE.* ([arxiv link](https://arxiv.org/abs/2011.05047))

## Other Useful Resources

- *Chakraborty, S., Shah, S., Soltani, K., & Swigart, A. (2019, December). Root Cause Detection Among Anomalous Time
Series Using Temporal State Alignment. In 2019 18th IEEE International Conference On Machine Learning And Applications
(ICMLA) (pp. 523-528). IEEE.* ([arxiv link](https://arxiv.org/abs/2001.01056))

## Blogs

- Zillow Tech Hub: [*Automatic and Self-aware Anomaly Detection at Zillow Using Luminaire*](https://medium.com/zillow-tech-hub/automatic-and-self-aware-anomaly-detection-at-zillow-using-luminaire-7addfdae4ca9)

## Development Team

Luminaire is developed and maintained by [Sayan Chakraborty](https://github.com/sayanchk), [Smit Shah](https://github.com/shahsmit14), [Kiumars Soltani](https://github.com/kiumarss), [Luyao Yang]( https://github.com/snazzyfox), [Anna Swigart](https://github.com/annaswigart), [Kyle Buckingham](https://github.com/kylebuckingham) and many other contributors from the Zillow Group A.I. team.

repository: MyAdminC/luminaire

branch_name: refs/heads/master

extension: md

license_type: permissive

repo_name: MyAdminC/luminaire


text:
---
title: FL008 Invalid datestamps in file names (space instead of a zero)
---
# Invalid datestamps in file names (space instead of a zero)

Problem ID | Manufacturer | Affected Firmware | Affected Hardware | Status
-----------|--------------|-------------------|-------------------|---------
FL008 | Frontier Labs | 3.08 | BAR-LT | New/unconfirmed |

It appears that some files produced by the BAR-LTs are emitted with a filename that has a faulty time stamp.

Such files appear to sometimes have a `0` leading digit in the _day_ field of the datestamp replaced with a ` ` (a space).
The problem does not seem to apply when the day of the month is greater or equal to `10` (i.e. when a non-zero is required in the datestamp)
and does not affect the daily sub-directories created by the sensors.

Excerpts from the log files confirm a space is used instead of a 0 in the sensors's view of the world - this is not a bug with the sensor.

Some files from our filesystem:

- ./255/445/20190324_AAO/20190324T060000+1000_REC.flac
- ./255/445/20190508_AAO/201905 8T020000+1000_REC.flac
- ./255/445/20190508_AAO/201905 8T200000+1000_REC.flac
- ./255/445/20190415_AAO/20190415T200000+1000_REC.flac
- ./255/445/20190408_AAO/201904 8T060000+1000_REC.flac
- ./255/445/20190405_AAO/201904 5T080000+1000_REC.flac

A count of affected files finds the problem in 1499 files out of 5572 (26%).

## Status of the problem

Minor issue, the correct data is in the **header** for the files.

## Status with vendor

FL is not aware of the problem yet. We will reach out for comment soon.

## Effects of the problem in common tools

Most tools we are aware of do not cater for date stamps with spaces in place of 0 digits.

- EMU does not handle this (issue: )
- AP.exe does not handle this date format
- The acoustic workbench does not handle this date format

## Example files

<https://cloudstor.aarnet.edu.au/plus/s/hrmQPSrkqV0Evvv?path=%2Ffrontier_labs%2FFL008>

repository: ecoacoustics/known-problems

branch_name: refs/heads/main

repo_name: ecoacoustics/known-problems


text:
# Humio Exporter
Exports data to Humio using JSON over the HTTP [Ingest API](https://docs.humio.com/reference/api/ingest/).

Supported pipeline types: traces (with metrics and logs to follow soon)

> :construction: This exporter is currently intended for evaluation purposes only!

## Getting Started
This exporter requires the following configuration options:

- `ingest_token` (no default): The token that has been issued in relation to the Humio repository to export data into. This token grants write-only access to a single, specific Humio repository. See [Ingest Tokens](https://docs.humio.com/docs/ingesting-data/ingest-tokens/) for more details.
- `endpoint` (no default): The base URL on which the Humio backend can be reached, in the form `host:port`. For testing this locally with the Humio Docker image, the endpoint could be `http://localhost:8080/`. For use with the Humio cloud, the URLs are as follows, both of which use port `80`:
    - EU: `https://cloud.humio.com/`
    - US: `https://cloud.us.humio.com/`

As defined in the [TLS Configuration Settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md#tls-configuration-settings), TLS is enabled by default. This can be disabled by overriding the following configuration options:

- `insecure` (default: `false`): Whether to enable client transport security for the exporter's HTTP connection. Not recommended for production deployments.
- `insecure_skip_verify` (default: `false`): Whether to skip verifying the server's certificate chain or not. Not recommended for production deployments.

In addition, the following global configuration options can be overridden:

- `disable_compression` (default: `false`): Whether to stop compressing payloads with gzip before sending them to Humio. This should only be disabled if compression can be shown to have a negative impact on performance in your specific deployment.
- `tags` (no default): A series of key-value pairs used to target specific Data Sources for storage inside a Humio repository. Refer to [Humio Tagging](https://docs.humio.com/docs/parsers/tagging/) for more details.
- `disable_service_tag` (default: `false`): By default, the service name will be used to tag all exported events in addition to user-provided tags. If disabled, only the user-provided tags will be used. However, at least one tag _must_ be specified.

### Traces
For exporting structured data (traces), the following configuration options are available:

- `unix_timestamps` (default: `false`): Whether to use Unix or ISO 8601 formatted timestamps when exporting data to Humio. If this is set to `true`, timestamps will be represented in milliseconds (Unix time) in UTC, and the time zone of the event is stored separately in the payload sent to Humio.

## Advaced Configuration
This exporter, like many others, includes shared configuration helpers for the following advanced settings:

- [HTTP Client Configuration](https://github.com/open-telemetry/opentelemetry-collector/tree/main/config/confighttp#client-configuration)
- [TLS Configuration](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md#tls-configuration-settings)
- [Queueing, Retry, and Timeout Configuration](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md#configuration)

## Example Configuration
Below are two examples of configurations specific to this exporter. For a more advanced example with all available configuration options, see [This Example](testdata/config.yaml).

```yaml
exporters:
    humio:
        ingest_token: "00000000-0000-0000-0000-0000000000000"
        endpoint: "https://my-humio-host:8080"
    humio/advanced:
        ingest_token: "00000000-0000-0000-0000-0000000000000"
        endpoint: "http://localhost:8080"
        timeout: 10s
        disable_compression: true
        disable_service_tag: true
        tags:
            host: "web_server"
            environment: "production"
        traces:
            unix_timestamps: true
```


text:
# Feathers sync

[![Greenkeeper badge](https://badges.greenkeeper.io/feathersjs-ecosystem/feathers-sync.svg)](https://greenkeeper.io/)

[![Build Status](https://travis-ci.org/feathersjs-ecosystem/feathers-sync.png?branch=master)](https://travis-ci.org/feathersjs-ecosystem/feathers-sync)
[![Code Climate](https://codeclimate.com/github/feathersjs-ecosystem/feathers-sync.png)](https://codeclimate.com/github/feathersjs-ecosystem/feathers-sync)
[![Test Coverage](https://codeclimate.com/github/feathersjs-ecosystem/feathers-sync/badges/coverage.svg)](https://codeclimate.com/github/feathersjs-ecosystem/feathers-sync/coverage)
[![Dependency Status](https://img.shields.io/david/feathersjs-ecosystem/feathers-sync.svg?style=flat-square)](https://david-dm.org/feathersjs-ecosystem/feathers-sync)
[![Download Status](https://img.shields.io/npm/dm/feathers-sync.svg?style=flat-square)](https://www.npmjs.com/package/feathers-sync)
[![Slack Status](http://slack.feathersjs.com/badge.svg)](http://slack.feathersjs.com)

> Synchronize service events between application instances using Redis or MongoDB publish/subscribe

## About

When running multiple instances of your Feathers application (e.g. on several Heroku Dynos), service events (`created`, `updated`, `patched`, `removed`) do not get propagated to other instances. feathers-sync uses MongoDB publish/subscribe via [mubsub](https://github.com/scttnlsn/mubsub) or Redis via [redis](https://github.com/NodeRedis/node_redis) to propagate all events to all application instances.

This allows to scale real-time websocket connections to any number of clients.

The application initialized in the following example will use the local `feathers-sync` database and `sync` collection and share service events with every other instance connected to the same database:

```js
var feathers = require('feathers');
var sync = require('feathers-sync');

var app = feathers();
app.configure(feathers.rest())
  .configure(feathers.socketio())
  .configure(sync({
    db: 'mongodb://localhost:27017/sync',
    collection: 'events'
  }))
  .use('/todos', todoService);

app.listen(3000);
```

### Use with MongoDB:

- __db__ - The MongoDB connection string (e.g. `mongodb://localhost:27017/events`) or database object
- __collection__ - The name of the capped event collection (default is `events`)
- __connect__ - A callback for when the MongoDB connection has been established
- __mubsub__ - Settings to be passed to mubsub (e.g. `{authSource:'admin'}`)

Additionally you can pass the original sync options:

- __size__ - Max size of the collection in bytes, default is 5mb
- __max__ - Max amount of documents in the collection
- __retryInterval__ - Time in ms to wait if no docs are found, default is 200ms
- __recreate__ - Recreate the tailable cursor when an error occurs (default is `true`)

### Use with Redis:

- __db__ - The Redis connection string (e.g. `redis://localhost:6379`) or database object
- __connect__ - A callback for when the Redis connection has been established

### Use with AMQP

- __uri__ - The AMQP connection string (e.g. `amqp://guest:guest@localhost:5672`)
- __prefix__ - A prefix that will be applied to all queues, exchanges and messages created by sync
- __amqpConnectionOptions__ - AMQP connection options

## How it works

![alt tag](https://raw.githubusercontent.com/PedroMD/feathers-sync/master/feathers-sync%20and%20real-time%20events-60.png)

## Caveats

When listening to service events with this, all events are going to get propagated to all clients. This means, that your event listeners should not perform any actions that change the global state (e.g. write something into the database) because every client will perform the same action.

Instead, event listeners should only be used to update the local state (e.g. a local cache) and send real-time updates to all connected clients, e.g. all browsers listening to websocket events.

If you need to perform actions, for example setting up a first blog post after a new user has been created add it to the service method itself (which will only run on its own instance) or use [feather-hooks](https://github.com/feathersjs/feathers-hooks) after hooks.

## Changelog

__0.1.0__

- Can now use Redis or Mongo
- Initial release

## Author

- [David Luecke](https://github.com/daffl)

## License

Copyright (c) 2015 David Luecke

Licensed under the [MIT license](LICENSE).

license: MIT


text:
Syscoin Core integration/staging tree
=====================================

[![Build Status](https://travis-ci.org/syscoin/syscoin.svg?branch=master)](https://travis-ci.org/syscoin/syscoin)

For an immediately usable, binary version of the Syscoin Core software, see
https://syscoincore.org/en/download/.

Further information about Syscoin Core is available in the [doc folder](/doc).

What is Syscoin Core?
----------------

Syscoin Core is an experimental digital currency that enables instant payments to anyone, anywhere in the world. Syscoin uses peer-to-peer technology to operate with no central authority: managing transactions and issuing money are carried out collectively by the network. Syscoin Core is the name of open source software which enables the use of this currency.

For more information, as well as an immediately useable, binary version of the Syscoin Core software, see https://syscoin.org/, or read the [original whitepaper](https://syscoin.org/zdag_syscoin_whitepaper.pdf).

Syscoin is a merge-minable SHA256 coin which provides an array of useful services which leverage the bitcoin protocol and blockchain technology.

Hybrid layer 2 PoW/PoS consensus with bonded validator system (masternodes) ZDAG technology for point-of-sale speeds and probabilistic confirmations useful for microtransactions Trustless sidechain access to NEVM and back through a custom permissionless/trustless sidechain technology (SYSX bridge) https://github.com/syscoin/sysethereum Decentralized governance (blockchain pays for work via proposals and masternode votes) Digital asset creation and management. ZDAG is open-source and developed over the course of 12 months by [Blockchain Foundry Inc](https://www.blockchainfoundry.co/), with external audit done over 6 months by [Whiteblock](https://whiteblock.io). [Performance report can be found here](https://syscoin.org/tps_whiteblock_syscoin_report.pdf).

Governance formula: We have implemented a % decline schedule — starting the first month with 1500000 SYS and ending on the 24th month with 196708 SYS. After that period, the superblock starts with 151767 SYS a month and will decline 5% every year. Read more about it here: https://medium.com/@syscoin/syscoin-4-0-rewards-specifications-a3dc01d85adf.

Interoptibility between UTXO assets and ERC20 NEVM account model through a trust-less, zero custodian, zero counterparty internal bridge. Documentation can be found [here](https://github.com/syscoin/sysethereum-docs).

For more information, as well as an immediately useable, binary version of
the Syscoin Core software, see https://syscoin.org/, or read the
[original whitepaper](https://syscoin.org/zdag_syscoin_whitepaper.pdf).

Syscoin is a merge-minable SHA256 coin which provides an array of useful services which leverage the bitcoin protocol and blockchain technology. It enables turing complete smart contracts running in an NEVM (Network-enhanced Virtual Machine) to leverage bitcoin security through merged-mining. Scaling the technology will happen on layer 2 (zkRollups for NEVM and Lightning Networks for UTXO assets).

- Block time: 150 seconds target
- Halving interval: 210240 (~1 year)
- Rewards: 96.25 Syscoins per block deflated 5 percent per year
  - 10 percent to governance proposals
  - 90 percent split with miner/masternode of which:
    - 25 percent to miner
    - 75 percent to masternode
- 50 percent of the transaction fees paid to masternode
- Masternode minimum subsidy(before seniority): 5.275 Syscoins (can not go below this amount even accounting for deflation)
- NEVM subsidy (EIP1559): 10.55 Syscoins (static, not deflating)
- SHA256 Proof of Work
- Mineable either exclusively or via merge-mining any SHA256 PoW coin
- Masternode collateral requirement: 100000 Syscoins
- Masternode seniority: 35 percent increase after 210240 blocks (~1 year), 100 percent increase after 525600 blocks (~2.5 years)
- Governance proposals payout schedule: every 17520 blocks (~1 month)
- Governance funding per round (Approx. 2m Syscoins per month to start)
- Governance funding gets 5% deflation per round (superblock). See formula below
- Codebase based off of latest Bitcoin Core (https://github.com/bitcoin/bitcoin)

For more information read the Syscoin whitepaper.

License
-------

Syscoin Core is released under the terms of the MIT license. See [COPYING](COPYING) for more
information or see https://opensource.org/licenses/MIT.

Development Process
-------------------

The `master` branch is regularly built (see `doc/build-*.md` for instructions) and tested, but it is not guaranteed to be
completely stable. [Tags](https://github.com/syscoin/syscoin/tags) are created
regularly from release branches to indicate new official, stable release versions of Syscoin Core.

The https://github.com/syscoin-core/gui repository is used exclusively for the
development of the GUI. Its master branch is identical in all monotree
repositories. Release branches and tags do not exist, so please do not fork
that repository unless it is for development reasons.

The contribution workflow is described in [CONTRIBUTING.md](CONTRIBUTING.md)
and useful hints for developers can be found in [doc/developer-notes.md](doc/developer-notes.md).

Testing
-------

Testing and code review is the bottleneck for development; we get more pull
requests than we can review and test on short notice. Please be patient and help out by testing
other people's pull requests, and remember this is a security-critical project where any mistake might cost people
lots of money.

### Automated Testing

Developers are strongly encouraged to write [unit tests](src/test/README.md) for new code, and to
submit new unit tests for old code. Unit tests can be compiled and run
(assuming they weren't disabled in configure) with: `make check`. Further details on running
and extending unit tests can be found in [/src/test/README.md](/src/test/README.md).

There are also [regression and integration tests](/test), written
in Python.
These tests can be run (if the [test dependencies](/test) are installed) with: `test/functional/test_runner.py`

The CI (Continuous Integration) systems make sure that every pull request is built for Windows, Linux, and macOS,
and that unit/sanity tests are run automatically.

### Manual Quality Assurance (QA) Testing

Changes should be tested by somebody other than the developer who wrote the
code. This is especially important for large or high-risk changes. It is useful
to add a test plan to the pull request description if testing the changes is
not straightforward.

Translations
------------

Changes to translations as well as new translations can be submitted to
[Syscoin Core's Transifex page](https://www.transifex.com/syscoin/syscoin/).

Translations are periodically pulled from Transifex and merged into the git repository. See the
[translation process](doc/translation_process.md) for details on how this works.

**Important**: We do not accept translation changes as GitHub pull requests because the next
pull from Transifex would automatically overwrite them again.

repository: syscoin/syscoin

gha_license_id: MIT

repo_name: syscoin/syscoin


text:
## rosa verify quota

Verify AWS quota is ok for cluster install

### Synopsis

Verify AWS quota needed to create a cluster is configured as expected

```
rosa verify quota [flags]
```

### Examples

```
  # Verify AWS quotas are configured correctly
  rosa verify quota

  # Verify AWS quotas in a different region
  rosa verify quota --region=us-west-2
```

### Options

```
  -h, --help   help for quota
```

### Options inherited from parent commands

```
      --debug            Enable debug mode.
      --profile string   Use a specific AWS profile from your credential file.
  -r, --region string    AWS region in which to run (overrides the AWS_REGION environment variable)
  -v, --v Level          log level for V logs
```

### SEE ALSO

* [rosa verify](rosa_verify.md)	 - Verify resources are configured correctly for cluster install

repository: mikenairn/rosa

repo_name: mikenairn/rosa


text:
# Evil Front
Modern, modular, component-based approach to presentation logic in Rails

Based on the [post](https://evilmartians.com/chronicles/evil-front-part-1) by [Evil Martians](https://evilmartians.com/)


text:
# dev-utils
Try to give a comprehensive list of utilities during Go development. Keep small and elegant.

[![GitHub license](https://img.shields.io/github/license/csimplestring/dev-utils)](https://github.com/csimplestring/dev-utils/blob/main/LICENSE)[![made-with-Go](https://img.shields.io/badge/Made%20with-Go-1f425f.svg)](http://golang.org)![gopherbadger-tag-do-not-edit](https://img.shields.io/badge/Go%20Coverage-80%25-brightgreen.svg?longCache=true&style=flat)[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square)](http://makeapullrequest.com)[![Maintenance](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://GitHub.com/Naereen/StrapDown.js/graphs/commit-activity)

## install
```
go get github.com/csimplestring/dev-utils
```

# features
- file watcher
- stop watch
- ring buffer matcher

# examples
- *file watcher*: The file watch will detects if the file is changed or not, and execute a callback if it is changed.
```
    // In this example, the last-modified-time determines if a file is modified or not.
    // if you want to use the md5 sum of a file content as the *state*, you can create your own Monitor to detect if the file state is changed by implementing the UpdateModifer interface.

    w, err := NewWatcher(tmpfile.Name(), &LastModifiedMonitor{})
	assert.NoError(t, err)

	cnt := 0
	cb := func(path string) {
		cnt++
	}

    // file is not changed, cnt == 0
	err = w.Check(cb)
	assert.NoError(t, err)
	assert.Equal(t, 0, cnt)

	_, err = tmpfile.WriteString("add")
	assert.NoError(t, err)

    // file is changed, last modified time is updated, cnt == 1
	err = w.Check(cb)
	assert.NoError(t, err)
	assert.Equal(t, 1, cnt)

```
- *stop watch*: the stop watch sometimes can help developers to log the running time of functions, code blocks etc. See [Example](https://github.com/csimplestring/dev-utils/blob/main/pkg/debug/stopwatch_test.go).

- *ring buffer matcher*: A RingBuffer that can be used to scan byte sequences for subsequences. This class implements an efficient naive search algorithm, which allows the user of the library to identify byte sequences in a stream on-the-fly so that the stream can be segmented without having to buffer the data.

repository: csimplestring/dev-utils

repo_name: csimplestring/dev-utils


text:
#Apache Shiro Articles

Here are some articles written by and for members of the Apache Shiro community. Please post any errata to the user or dev [mailing lists](mailing-lists.html "Mailing Lists").

##Introductory Articles

New to Shiro? Here are some great introductory articles:

*   **[Application Security with Apache Shiro](https://www.infoq.com/articles/apache-shiro)** InfoQ article by Les Hazlewood, Apache Shiro PMC Chair.

*   **[Apache Shiro Beginner's Webapp Tutorial](webapp-tutorial.html "Apache Shiro Beginner's Webapp Tutorial"): a step-by-step tutorial to enable Shiro in a web application** on 19 November 2013 by Les Hazlewood

*   **[What's new in Apache Shiro 1.2](https://stormpath.com/blog/whats-new-apache-shiro-12)** on 13 March 2012 by Les Hazlewood.

*   **[Introducing Apache Shiro](http://www.ibm.com/developerworks/web/library/wa-apacheshiro/)** by Nathan Good on IBM DeveloperWorks, 14 September 2010.

*   **An Introduction to Shiro (formerly JSecurity/Ki) - A Beginner's Tutorial** by [Bruce Phillips](http://www.brucephillips.name):
    *   [Part 1](http://www.brucephillips.name/blog/index.cfm/2009/4/5/An-Introduction-to-Ki-formerly-JSecurity--A-Beginners--Tutorial-Part-1)
    *   [Part 2](http://www.brucephillips.name/blog/index.cfm/2009/4/5/An-Introduction-to-Ki-formerly-JSecurity--A-Beginners--Tutorial-Part-2)
    *   [Part 3](http://www.brucephillips.name/blog/index.cfm/2009/4/5/An-Introduction-to-Ki-formerly-JSecurity--A-Beginners--Tutorial-Part-3)
    *   [Part 4](http://www.brucephillips.name/blog/index.cfm/2009/4/5/An-Introduction-to-Ki-formerly-JSecurity--A-Beginners--Tutorial-Part-4)
    *   [Part 5](http://www.brucephillips.name/blog/index.cfm/2009/5/1/An-Introduction-to-Ki-formerly-JSecurity--A-Beginners-Tutorial-Part-5)

##Additional Articles

Once you've gotten your feet wet, you might find these useful too:

*   **[How to Integrate Apache Shiro with JavaEE6](http://czetsuya-tech.blogspot.com/2012/10/how-to-integrate-apache-shiro-with.html?spref=tw)** by czetsuya on 11 October 2012.

*   **[Custom Apache Shiro JDBC Realm](https://mehmetceliksoy.wordpress.com/2015/06/28/shiro-jdbc-realm/)** by Mehmet Celiksoy

*   **[Securing ZK Applications with Apache Shiro](https://www.zkoss.org/wiki/Small_Talks/2012/March/Securing_ZK_Applications_With_Apache_Shiro)** by Ashish Dasnurkar on 6 March 2012.

*   **Facebook Login with Apache Shiro** by Mike Warren on 28 November 2011
    *   [Part 1](https://mrdwnotes.wordpress.com/2011/11/28/using-apache-shiro-security-to-allow-login-via-facebook-part-1/)
    *   [Part 2](https://mrdwnotes.wordpress.com/2011/11/28/using-apache-shiro-security-to-allow-login-via-facebook-part-2/)

*   **Apache Shiro - a blog series by Meri**
    *   [Part 1 - Basics](http://meri-stuff.blogspot.com/2011/03/apache-shiro-part-1-basics.html) on 27 March 2011
    *   [Part 2 - Realms, Database and PGP Certificates](http://meri-stuff.blogspot.com/2011/04/apache-shiro-part-2-realms-database-and.html) on 18 April 2011
    *   [Part 3 - Cryptography](http://meri-stuff.blogspot.com/2011/12/apache-shiro-part-3-cryptography.html) on 4 December 2011

*   **[The New RBAC: Resource-Based Access Control](https://stormpath.com/blog/new-rbac-resource-based-access-control)** by Les Hazlewood on 9 May 2011

*   **[HTTP Authentication and Security with Apache Shiro](http://blog.xebia.com/author/yamsellem/)** blog article by yamsellem on 18 April 2011.

*   **[Using Shiro for Authorization via CDI Interceptors then Easily Test with Arquillian](http://spring-java-ee.blogspot.com/2011/04/using-shiro-for-authorization-via-cdi.html)** blog article by Hendy Irawan on 16 April 2011.

*   **[Apache Shiro Support for Mule](http://blogs.mulesoft.com/dev/mule-dev/apache-shiro-support-for-mule/)** by Dan Diephouse on 10 January 2011.

*   **[Apache Shiro tags for JSF - Securing Your JSF Pages](http://techbeats.deluan.com/apache-shiro-tags-for-jsffacelets)** by Deluan Quintão on 1 November 2010.

*   **Shiro DevNexus 2009 Presentation** by Jeremy Haile: ([PDF](assets/images/articles/Ki-DevNexus-2009.pdf?version=1&modificationDate=1246602947000)) ([Keynote](assets/images/articles/Ki-DevNexus-2009.key.zip?version=1&modificationDate=1246602947000)) ([Powerpoint](assets/images/articles/Ki-DevNexus-2009.ppt.zip?version=1&modificationDate=1246602947000))
<input type="hidden" id="ghEditPage" value="articles.md"></input>

repository: evfool/shiro-site

repo_name: evfool/shiro-site


text:
# Masonite
Core files for the python framework

[![Build Status](https://travis-ci.org/MasoniteFramework/core.svg?branch=master)](https://travis-ci.org/MasoniteFramework/core) <img src="https://img.shields.io/badge/coverage-79%25-green.svg">

To get started with this repo, [read the CONTRIBUTING.md file on the masonite repo](https://github.com/josephmancuso/masonite/blob/master/CONTRIBUTING.md)

repository: mapeveri/core

repo_name: mapeveri/core


text:
## Gesture Recognition - CNN2D + RNN Modelling Technique

### Cross Validation Results

repository: anugrahasinha/neuralnetwork

repo_name: anugrahasinha/neuralnetwork


text:
+++
# Date this page was created.

# Project title.
title = "Brain-inspired recurrent neural network"
date = "2021-07-29"

# Project summary to display on homepage.
summary = "Master's thesis proposal with prof. Luca Mazzucato"

# Optional image to display on homepage (relative to `static/img/` folder).
image_preview = ""

# Tags: can be used for filtering projects.
# Example: `tags = ["bachelor-thesis-proposal", "master-thesis-proposal"]`
tags = ["master-thesis-proposal"]

# Optional external URL for project (replaces project detail page).
external_link = ""

# Does the project detail page use math formatting?
math = false

# Optional featured image (relative to `static/img/` folder).
[header]
image = ""
caption = ""
+++

# Abstract
Brain-inspired recurrent neural networks are remarkably efficient at performing tasks, however, their performance is challenged by tasks involving temporal multiplexing, defined as the ability to manipulate information over multiple timescales simultaneously. Here, we will test the hypothesis that recurrent circuits endowed with multiple timescales, comprising both fast and slow neural populations, can robustly perform temporal multiplexing.

* In collaboration with prof. Luca Mazzucato, University of Oregon, Eugene

repository: liphlab/liph-website

repo_name: liphlab/liph-website


text:
# Getting started with Tablesaw

Java is a great language, but it wasn't designed for data analysis. Tablesaw makes it easy to do data analysis in Java.

This tutorial will help you get up and running, and introduce some of Tablesaw's basic functionality.

## Setup

First, Tableasaw requires Java 8 or newer.

Second, you need to add the dependency to your pom file. It's available on Maven Central.

````
<dependency>
    <groupId>tech.tablesaw</groupId>
    <artifactId>tablesaw-core</artifactId>
    <version>LATEST</version>
</dependency>
````

That's it for setup. On to design

## Tables and Columns

As you would expect, Tablesaw is all about tables, and tables are made of columns. We'll start with columns.

### Columns

A column is a named, one-dimensional collection of data. It may or may not be part of a table. All data in a column must be of the same type.

Tablesaw supports columns for Strings, floats, doubles, ints, shorts, longs, booleans, LocalDates, LocalTimes, Instants, and LocalDateTimes. The date and time columns are comparable with the java.time classes introduced in Java 8.

To create a column you can use one of its static *create()* methods:

```java
@@snip [create_column](./src/main/java/tech/tablesaw/docs/GettingStarted.java)
```

which produces:

```java
@@snip [create_column](./output/tech/tablesaw/docs/GettingStarted.txt)
```

Each column has an associated 0-based index. To get an individual value call *get()* with the index.

```java
@@snip [column_index](./src/main/java/tech/tablesaw/docs/GettingStarted.java)
```
which returns 3.0.

#### Array Operations

Tablesaw makes columns easy to work with. Operations that work on numbers in standard Java, for example, often work on *columns* of numbers in Tablesaw. To multiply every value in a column by 4, we use the *multiply()* method, which returns a new column like the original.

```java
@@snip [column_multiply](./src/main/java/tech/tablesaw/docs/GettingStarted.java)
```
producing:

```java
@@snip [column_multiply](./output/tech/tablesaw/docs/GettingStarted.txt)
```
As you can see, the values are 4x the values in the original. The new column's name is made by combining the original
"Test" and the operation (* 4). You can change it if you like using `setName(aString)`.

There are so many columnar operations in Tablesaw that, as a general rule, if you find yourself writing a for
loop to process a column or table, you may be missing something.

#### Objects and Primitives

Many Java programs and programmers work exclusively with Objects, rather than primitives. In Tablesaw, we often use
primitives because they use *much* less memory than their boxed alternatives.  A Byte *object*, for example, uses as
much memory as a primitive double, even though bytes have a range of only 256 values.

There is a price for this frugality. When you work with primitives, you forgo some common java capabilities, like the
use of standard Java 8 predicates. While Java thoughtfully provides some specialized predicate interfaces
(e.g. *IntPredicate*), they don't provide any primitive *BiPredicate* implementations, nor do their primitive
interfaces cover all primitive types. Without an IntBiPredicate, we can't implement operations like a < b.
So we were left to roll our own. You can find them in the package *tech.tablesaw.filtering.predicates*.
They work like the standard objects.

This just covers the most basic information about columns. You can find more in the section on
[Columns](https://jtablesaw.github.io/tablesaw/userguide/columns), or in the Javadocs for the
[api package](http://www.javadoc.io/page/tech.tablesaw/tablesaw-core/latest/tech/tablesaw/api/package-summary.html)
and the [columns package](http://www.javadoc.io/page/tech.tablesaw/tablesaw-core/latest/tech/tablesaw/columns/package-summary.html).

### Selections

Before going on to tables, we should talk about *selections*. *Selections* are used to filter both tables and columns.
Often they work behind the scenes, but you can use them directly. For example, consider our `DoubleColumn` containing
the values {1, 2, 3, 4}. You can filter that column by sending it a message. For example:

```java
@@snip [selection_one](./src/main/java/tech/tablesaw/docs/GettingStarted.java)
```

This operation returns a *Selection*. Logically, it's a bitmap of the same size as the original column. The method above, effectively, returns 1, 1, 0, 0, since the first two values in the column are less than three, and the last two are not.

What you probably wanted was not a `Selection` object, but a new `DoubleColumn` that contains only the values that passed the filter. To get this, you use the *where(aSelection)* method to apply the selection:

```java
@@snip [selection_two](./src/main/java/tech/tablesaw/docs/GettingStarted.java)

@@snip [selection_two](./output/tech/tablesaw/docs/GettingStarted.txt)
```
Doing this in two steps provides many benefits. For one, it lets us combine filters. For example:

```java
@@snip [selection_three](./src/main/java/tech/tablesaw/docs/GettingStarted.java)
```

If the methods returned columns directly, they couldn't be combined this way.  It also lets us use the same method for
filtering both tables and columns, as you'll see below.

##### Selecting by index

These examples show how to select using predicates. You can also use a selection to retrieve the value at a specific
index, or indexes. Both of the following are supported:

```java
@@snip [selection_by_index](./src/main/java/tech/tablesaw/docs/GettingStarted.java)
```

If you have several columns of the same length as you would in a table of data, you can make a selection with one column
and use it to filter another:

```java
@@snip [selection_other_col](./src/main/java/tech/tablesaw/docs/GettingStarted.java)
```

> **Key point:** Note the methods *startsWith(aString)*, *isLessThan(aNumber)*, and *isPositive()*. These were predefined
>for your use. There are many such methods that can be used in building queries. For StringColumn, they're defined in the
>[tech.tablesaw.columns.strings.StringFilters interface](http://www.javadoc.io/page/tech.tablesaw/tablesaw-core/latest/tech/tablesaw/columns/strings/StringFilters.html).
>It also includes *endsWith()*, *isEmpty()*, *isAlpha()*, *containsString()*[^1], etc. Each column has a similar set of
>filter operations. They can all be found in the filter interfaces located in sub-folders of tech.tablesaw.columns
> (e.g. tech.tablesaw.columns.dates.DateFilters).

#### Map functions

Map functions are methods defined on columns that return new Columns as their result. You've already seen one:
The column *multiply(aNumber)* method above is a map function with a scalar argument. To multiple the values in two
columns, use *multiply(aNumberColumn)*:

```java
DoubleColumn newColumn = nc1.multiply(nc2);
```

```java
@@snip [multiply_two_columns](./src/main/java/tech/tablesaw/docs/GettingStarted.java)

@@snip [multiply_two_columns](./output/tech/tablesaw/docs/GettingStarted.txt)
```

Each value in column nc1 is multiplied by the corresponding value in nc2, rather than by a scalar value in the earlier example.

There are many map functions built-in for the various column types. Here are some examples for StringColumn:

```java
@@snip [string_map_examples](./src/main/java/tech/tablesaw/docs/GettingStarted.java)

```

As you can see, for many String methods that return a new String. StringColumn provides an equivalent map method that
returns a new StringColumn. It also includes other helpful methods found in Guava's String library and in the
Apache Commons String library.

> **Key point:** Every column type has a set of map operations like *multiply(aNumber)*. For StringColumn,
>these methods are defined in the [*tech.tablesaw.columns.strings.StringMapFunctions*](http://www.javadoc.io/page/tech.tablesaw/tablesaw-core/latest/tech/tablesaw/columns/strings/StringFilters.html) interface. It includes many methods beyond those shown above. Methods for all column types can all be found in their filter interfaces located in the sub-folders of tech.tablesaw.columns (e.g. [*tech.tablesaw.columns.dates.DateMapFunctions*](http://www.javadoc.io/page/tech.tablesaw/tablesaw-core/latest/tech/tablesaw/columns/strings/StringFilters.html), which provides date methods like *plusDays(anInt)*, *year()*, and *month()*).
>

#### Reduce (aggregate) functions: Summarizing a column

Sometimes you want to derive a singly value that summarizes in some sense the data in a column. Aggregate functions do
just that. Each such function scan all the values in a column and returns a single scalar value as a result.
All columns support some aggregate functions: *min*() and *max*(), for example, plus *count()*, *countUnique()*, and *countMissing()*.
Some also support type-specific functions. BooleanColumn, for example, supports *all()*, which returns *true* if all
of the values in the column are *true*. The functions *any()*, and *none()*,  return true if any or none the values in
the column are true, respectively. The functions *countTrue()*, and *countFalse()* are also available.

NumberColumn has many more aggregate functions. For example, to calculate the standard deviation of the values in a column, you would call:

```java
@@snip [stdev](./src/main/java/tech/tablesaw/docs/GettingStarted.java)
```

> **Key point:** NumberColumn supports many aggregation functions, including many of the most useful.
>Among those available are *sum*, *count*, *mean*, *median*, *percentile(n)*, *range*, *variance*, *sumOfLogs*, and so on.
>These are defined in the [NumericColumn](http://www.javadoc.io/page/tech.tablesaw/tablesaw-core/latest/tech/tablesaw/api/NumericColumn.html) class.
>

When we discuss tables below, we'll show how to calculate sub-totals in one or more numeric columns by the values in one or more grouping columns.

### Tables
A table is a named collection of columns. All columns in the table must have the same number of elements, although
missing values are allowed. A table can contain any combination of column types.

#### Creating Tables

You can create a table in code. Here we create a table and add two new columns to it:

```java
@@snip [tables](./src/main/java/tech/tablesaw/docs/GettingStarted.java)
```

#### Importing data

More frequently, you will load a table from a CSV or other delimited text file.

```java
@@snip [read_bush_table](./src/main/java/tech/tablesaw/docs/GettingStarted.java)
```

Tablesaw does a pretty good job at guessing the column types for many data sets, but you can specify them
if it guesses wrong, or to improve performance. Numerous other options are available, such as specifying whether or
not there's a header, using a non-standard delimiter, supplying a custom missing value indicator, and so on.

***Note:*** Getting data loaded is sometimes the hardest part of data analysis. Advanced options for loading data are
described in the documentation on [Importing Data](https://jtablesaw.github.io/tablesaw/userguide/importing_data).
That section also shows how you can read data from a database, a stream, or an HTML table. The stream interfaces
lets you read data from a Web site or an S3 bucket.

#### Exploring Tables

Because Tablesaw excels at manipulating tables, we use them whenever we can.  When you ask tablesaw for the structure
of a table, the answer comes back in the form of another table where one column contains the column names, etc.
The methods ` structure()`, `shape()`, `first(n)`, and `last(n)` can help you get to know a new data set. Here are some examples.

```java
@@snip [table_structure](./src/main/java/tech/tablesaw/docs/GettingStarted.java)

@@snip [table_structure](./output/tech/tablesaw/docs/GettingStarted.txt)
```

```java
@@snip [table_shape](./src/main/java/tech/tablesaw/docs/GettingStarted.java)

@@snip [table_shape](./output/tech/tablesaw/docs/GettingStarted.txt)
```

```java
@@snip [first_three](./src/main/java/tech/tablesaw/docs/GettingStarted.java)

@@snip [first_three](./output/tech/tablesaw/docs/GettingStarted.txt)
```

```java
@@snip [last_three](./src/main/java/tech/tablesaw/docs/GettingStarted.java)

@@snip [last_three](./output/tech/tablesaw/docs/GettingStarted.txt)
```

Table's *toString()* method returns a String representation like those shown above. It returns a limited number of
rows by default, but you can also use *table.printAll()*, or *table.print(n)* to get the output you want.

Of course, this is just the beginning of exploratory data analysis. You can also use numeric and visual tools to
explore your data. These facilities are described in the documentation on statistics and
[plotting](userguide/Introduction_to_plotting.md), respectively.

#### Working with a table's columns

Often you'll work with specific columns in a table. Here are some useful methods:

```java
@@snip [working_with_columns_examples](./src/main/java/tech/tablesaw/docs/GettingStarted.java)
```
In tablesaw, column names are case-insensitive. You get the same column if you ask for any of these:

```java
@@snip [get_column_case_insensitive](./src/main/java/tech/tablesaw/docs/GettingStarted.java)
```

remembering column names is enough of a burden without having to remember exactly which characters are capitalized.

##### Getting specific column types from a table

Columns can be retrieved from tables by name or position. The simplest method *column()* returns a object of type Column.
This may be good enough, but often you want to get a column of a specific type. For example, you would need to cast the
value returned to a NumberColumn to use its values in a scatter plot.

```java
@@snip [get_column](./src/main/java/tech/tablesaw/docs/GettingStarted.java)
```

When a variable type is "Column" it only provides methods that are available on *all* columns. You can't perform math
or do a string replace directly on a Column type. If you need a StringColumn you could cast the column, for example:

```java
@@snip [get_column_cast](./src/main/java/tech/tablesaw/docs/GettingStarted.java)
```

Table also supports methods that return columns of the desired type directly:

```java
@@snip [get_typed_columns](./src/main/java/tech/tablesaw/docs/GettingStarted.java)
```

> **Key point:** You may want a specific kind of column to work with. Either use the standard *column()* method and
>cast the result or use one of the type specific methods (like *numberColumn()*) that handle the cast for you.
>There are also methods or getting columns of a specific type.

####  Working with rows

As with columns, many options exist for working with tables in row-wise fashion. Here are some useful ones:

```java
@@snip [row_wise_examples](./src/main/java/tech/tablesaw/docs/GettingStarted.java)
```

You can also perform arbitrary operations on each row in the table.  One way is to just iterate over the rows and
work with each column individually.

```java
@@snip [for_loop](./src/main/java/tech/tablesaw/docs/GettingStarted.java)
```

Another approach lets you skip the iteration and just provide a Consumer for each row.

```java
@@snip [stream](./src/main/java/tech/tablesaw/docs/GettingStarted.java)
```

If you need to process more than one row at a time, there are several methods to help.

```java
@@snip [stepping_rolling_stream](./src/main/java/tech/tablesaw/docs/GettingStarted.java)
```

See [Rows](https://jtablesaw.github.io/tablesaw/userguide/rows) for more information and other options.

#### Sorting

To sort a table, you can just use the `sortOn()` method and give it one or more column name:

```java
@@snip [sort](./src/main/java/tech/tablesaw/docs/GettingStarted.java)
```

See [Sorting](https://jtablesaw.github.io/tablesaw/userguide/sorting) for more information and other options.

#### Filtering

Query filters can be combined using the logical operations *and*, *or*, and *not*.
These are implemented on the `QuerySupport` class.

```java
@@snip [static_boolean_operators_import](./src/main/java/tech/tablesaw/docs/GettingStarted.java)
```

Each method accepts a function with the following signature `Function<Table, Selection>`. Lambadas work nicely.
```java
@@snip [filtering](./src/main/java/tech/tablesaw/docs/GettingStarted.java)
```

#### Summarizing

```java
@@snip [static_aggregate_function_import](./src/main/java/tech/tablesaw/docs/GettingStarted.java)
```

The usual way to calculate values is  to use the *summarize()* method:

```java
@@snip [summarize_basic](./src/main/java/tech/tablesaw/docs/GettingStarted.java)
```

It's important to recognize, that the column need not exist when summarize is invoked. Any map function can be used
in the *by()* statement to group on calculated values. A common use case is in handling dates. You can summarize sales
by day-of-week, as follows:

```java
@@snip [summarize_calculated_column](./src/main/java/tech/tablesaw/docs/GettingStarted.java)
```

which says "return the mean and median sales by day of week."

> **Key point**: Tables are usually split based on columns, but the columns can be calculated on the fly

See the documentation on [Summarizing](https://jtablesaw.github.io/tablesaw/userguide/reducing) data, and the classes in
the [aggregate package](http://www.javadoc.io/page/tech.tablesaw/tablesaw-core/latest/tech/tablesaw/aggregate/package-summary.html)
for more detail.

##### Cross-Tabulations (AKA contingency tables)

If you're only interested in how the frequently observations appear in different categories, you can use cross-tabulations.
In the example below we show the table percents, but you can also get row and column percents and raw counts.

```java
@@snip [crosstab](./src/main/java/tech/tablesaw/docs/GettingStarted.java)
```

The formatted output is shown below.

```java
@@snip [crosstab](./output/tech/tablesaw/docs/GettingStarted.txt)
```

See the section on [Cross Tabs](https://jtablesaw.github.io/tablesaw/userguide/crosstabs), and the JavaDocs for the
[CrossTab](http://www.javadoc.io/page/tech.tablesaw/tablesaw-core/latest/tech/tablesaw/aggregate/CrossTab.html) class.

## Conclusion

We've covered a lot of ground. To learn more, please take a look at the
[User Guide](https://jtablesaw.github.io/tablesaw/userguide/toc) or API documentation
([Java Docs](http://www.javadoc.io/page/tech.tablesaw/tablesaw-core/latest/index.html)).

[^1]: Note that containsString(String subString) is different from contains(). The first method looks at each
string in the column to see if it conains the substring. The second method looks at every row in the column and returns
true if any matches the entire string. In other words, contains is like contains as defined on List<String>. , etc.

repository: jtablesaw/tablesaw

gha_language: Java

repo_name: jtablesaw/tablesaw


text:
TodoList - Firebase
===
> It's a pratice for using firebase to build a todolist

## Getting Started

Download and add your firebase config in `js/todolist.js` then you can enjoy it!

## Introduction

Just a todo list, you can add anything you want to do on it.By the way, it's chinese.

"新增" means `add`, "全部清除" means `clear all`.

- Loading before data loaded
![](https://i.imgur.com/5xqc5Aj.png)

- After first todo added
![](https://i.imgur.com/UNyCkPn.png)

## What's used in this project?
- Html
- CSS (also used sass, but it's not important at all here) with Bootstrap 4
- JavaScript with jQuery
- Firebase

That's all!

repository: TMineCola/Todolist-firebase

repo_name: TMineCola/Todolist-firebase


text:
# Letendraea fuscospora (Plowr.) Petch, 1947 SPECIES

#### Status
SYNONYM

#### According to
The Catalogue of Life, 3rd January 2011

#### Published in
Naturalist, Hull 822: 94 (1947)

#### Original name
Nectria fuscospora Plowr., 1888

### Remarks
null

repository: mdoering/backbone

path: /life/Fungi/Ascomycota/Sordariomycetes/Hypocreales/Nectriaceae/Cosmospora/Cosmospora vilior/ Syn. Letendraea fuscospora/README.md

repo_name: mdoering/backbone