text:
---
layout:     post
title:      A point cloud based approach for plant height estimation from aerial platforms
date:       2019-10-10
author:     Mary Li
header-img: img/2019-10-10-bg.png
catalog: true
tags:
    - 3D mapping and reconstruction
    - point cloud processing
    - Agriculture-Plant Phenotyping
    - image processing
---
_In 2018-2019,I led a project to develop an aerial data processing pipeline that aims to extract various plant traits by minimum number of flights.
The software should be able to process data collected by both manned (helicopter) and unmanned (UAV) vehicle and segment single plant/plot from large scale
agriculture field for trait analysis. I include here the published part of the research. Please cite the paper at bottom for any information from this article._

## Introduction

With the development of computer vision technologies, using images acquired by aerial platforms to measure large scale agricultural fields has been increasingly studied. In order to provide a more time efficient, light weight and low cost solution, in this paper we present a highly automated processing pipeline that performs plant height estimation based on a dense point cloud generated from aerial RGB images, requiring only a single flight. A previously acquired terrain model is not required as input. The process extracts a segmented plant
layer and bare ground layer. Ground height estimation achieves sub 10cm accuracy. High throughput plant height estimation has been performed and results are compared with LiDAR based measurements.
The key innovation is to estimate
ground height underneath the target plant area using an inverse
distance weighting from visible points of the neighbouring
ground in form of gridded ground controids. It not only
enables plant height calculation at plot level, but can also
be applied to various shapes of canopies for different types
of plants. The only manual input required is the marking of
Ground Control Points (GCP) and contour labeling(annotation) of the
Regions of Interest (ROI) in the plot detection process.

## METHODOLOGY

The proposed processing pipeline consists of multiple steps,
as shown in Figure 1: 1) image acquisition and filtering
; 2) plot automated detection from the ortho-mosaics 3) generation of dense 3D point cloud; 4) point cloud
segmentation; 5) ground height estimation; 6) plant height
estimation. In this paper we focused on 3-6 and deep learning based plot detection will discussed separately in another paper.

### Image Acquisition and 3D reconstruction

An integrated unit that can be mounted on both manned or unmaned aerial platforms was designed for this research. A high resolution camera (CanonEOS6D) was integrated in the unit for 3D reconstruction. In this experiment images were taken from a helicopter with 90% frontal overlap and 70% side overlap on average. The flight height was approximately 90 meters above ground with a speed of travel at 25-30 knots. A total of 9 evenly distributed GCPs at centimeter level accuracy were used together with 539 images, covering
an area of 0.168km^2. A 1.89cm Ground Sampling Distance (GSD) has been achieved.

Full 3D reconstruction was done by Pix4Dmapper Pro, which is a commercial software package that uses structure from motion ([SFM](http://marysfishingpool.tk///2019/10/12/SFM/)) techniques. It corrects camera distortion, refines location using ground control points (GCPs), accounts for camera poses and generates 3D maps in the form of ortho-mosaic and 3D point cloud. We mainly use the 3D point cloud for further analysis:
![field.gif](https://i.loli.net/2019/10/11/KtXDkHJ4Ijl9orb.gif)

### Point cloud segmentation

![Kqkj1O.gif](https://s2.ax1x.com/2019/11/02/Kqkj1O.gif)
 Plant Layer Segmentation
The point cloud segmentation process consists of three parts: plant layer segmentation, ground layer segmentation and ground surface smoothing. More specifically, we first segment the plant layer based on colour indices.

[![KboZOH.md.png](https://s2.ax1x.com/2019/11/02/KboZOH.md.png)](https://imgchr.com/i/KboZOH)
Plant Layer
The remaining points are the soil ground as well as other objects in the field, such as buildings, vehicle and dark or less green vegetation. In order to extract the bare ground, a Progressive Morphological Filter
[1] was used to filter out these objects. The filter uses classical morphological opening with a gradually increasing window size and elevation differences to remove non-ground
objects while retaining the terrain slope changes. This method has previously been used in
filtering LiDAR point clouds and DEM generation, and it is increasingly being adopted in Photogrammetric research.

[![KboMkt.md.png](https://s2.ax1x.com/2019/11/02/KboMkt.md.png)](https://imgchr.com/i/KboMkt)
 Ground Layer
[![Kbodkq.md.png](https://s2.ax1x.com/2019/11/02/Kbodkq.md.png)](https://imgchr.com/i/Kbodkq)
 Object Layer
The resulting point cloud, the bare ground, is still noisy. Moving least squares (MLS) surface reconstruction [2] is used to remove noise and build a smooth ground surface from the point cloud. Compared with other noise filtering and surface reconstruction methods, MLS is more powerful in dealing with intrinsic errors such as data irregularities due to small object measurements. The idea is to use an analytical function to approximate surfaces based on weighted local data points. It is being increasingly adopted as the standard definition of point set surfaces, and various algorithms in this class have been proposed [3]. Afterwards, colour information is further removed to reduce data size.
![KbTI5q.png](https://s2.ax1x.com/2019/11/02/KbTI5q.png)

### Grid based ground height estimation

Once a ground surface has been obtained, the next step is to calculate the ground height. At this stage, the plants inside each plot have been removed, leaving only holes. We then grid the ground surface and calculate ground heights based on point clouds within each grid.
A 2D bounding box in the XY plane of the ground surface is used to divide the ground into regular rectangle grids. Within each grid, a sub-point cloud is vertically extracted and a centroid of the point cloud is calculated for all axes (X,Y,Z). This centroid is called the ground centroid (GC). The centroid Z value is used as the height for this grid. For this
particular dataset, a 50×50 rectangular grid was used, resulting in 11.2m x 7.45m size single grid unit for height calculation (see Figure 9). Therefore, ground height has been calculated
at 50 × 50 resolution with each grid covering approximately 80 m^2 . If a grid has no data points, no centroid is generated.
![Screenshot from 2019-10-11 16-03-18.png](https://i.loli.net/2019/10/11/aZOtIMqhcGEFXkP.png)

### Plant height estimation

Plant height estimation consists of two steps: 1) region of interest (ROI) extraction; 2) height estimation for each ROI.

####  region of interest (ROI) extraction
The ROI is extracted by employ state of the art FasterRCNN, where contours/bounding boxes of plots and trees are detected and located on top of the ortho-mosaic (this part hasn't been published).
Their 2D bounding boxes are then used for 3D point cloud segmentation.
![polygon_roi.gif](https://i.loli.net/2019/10/11/MlwS6yVoriHL7vI.gif)

#### Plant height estimation using inverse distance weighting
After each ROI point cloud has been extracted, its height is
calculated by subtracting the local ground height from each
point height. Then a histogram of height values is generated
for each ROI and the average value is used as the overall
height for each ROI. The key element is to calculate the local
ground height underneath each ROI.

By using the ground centroids calculated as shown in
section "Grid based ground height estimation", we are able to obtain the ground height at various locations. A 2D centroid for each ROI is calculated based on its 2D contour and used as its position. Then k-
nearest neighbour (k-NN) search is used to find its k nearest ground centroids within 2D space. Based on the 2D Euclidean
distances between the ground centroids and the ROI centroid,
an inverse distance weighting (IDW) algorithm is used to
estimate the height at the ROI centroid location. That height
is used as the local ground height for the ROI. The process is shown in Algorithm 1 and illustrated in the following figure:
![Screenshot from 2019-10-11 16-09-42.png](https://i.loli.net/2019/10/11/GHi3pfBThUw1OMK.png)
![Screenshot from 2019-10-11 23-06-20.png](https://i.loli.net/2019/10/11/T1VvFOdQMkHf5i7.png)

## Experiments and Discussion

We first validate our ground height estimation accuracy by comparing it with 31 evenly distributed check points.The root mean square error (RMSE) between the ground
height measured by RTK GPS and by our method is 8.2cm.
The agreement between our method and the reference data is
shown in Figure 13.
![Screenshot from 2019-10-11 21-47-13.png](https://i.loli.net/2019/10/11/HqCs6prljiORQMK.png)

Data from four experiments of total 784 plots have been
used to test the height estimation pipeline. All 784 ROI have been extracted from the plant layer so as generating 784 point clouds. The height for each ROI has
been estimated and compared with Lidar data collected 2 days before the flight.
![Screenshot from 2019-10-11 22-38-34.png](https://i.loli.net/2019/10/11/lquMDEVf35UtnCS.png)

A consistent offset between aerial measurement and Lidar can be observed. Interestingly, the same phenomenon has been reported in other research too. In [4], compared with
ground LiDAR, a systematic under-estimation is observed for structure from motion techniques (aerial point cloud derived plant height) . The authors believe their result agrees with previous studies [5] where it is found that structure from motion from aerial imagery lacks the ability to reconstruct accurately the top of the canopy, due to the reason that
aerial RGB imagery has a coarser spatial resolution and limited penetration capacity compared with ground LiDAR. Future research will be focused mitigating these two
shortcomings with other sensors and technologies.

## Development
I personally contributed to over 80% of system design and development (the whole backend pipeline), including the main innovation of ground height estimation of invisible areas and various point cloud processing procedures for layer segmentation and height estimation.
Programming aspect: C++, Python, PCL library, OpenCV, Pytorch

## Reference
[1] K. Zhang, S.-C. Chen, D. Whitman, M.-L. Shyu, J. Yan, and C. Zhang,
“A progressive morphological filter for removing nonground measure-
ments from airborne LIDAR data,” IEEE Transactions on Geoscience
and Remote Sensing, vol. 41, no. 4, pp. 872–882, apr 2003.

[2] M. Alexa, J. Behr, D. Cohen-Or, S. Fleishman, D. Levin, and C. Silva,
“Point set surfaces,” in Proceedings Visualization, 2001. VIS’01. IEEE,
2001.

[3] Z.-Q. Cheng, Y.-Z. Wang, B. Li, K. Xu, G. Dang, and S.-Y. Jin, “A
survey of methods for moving least squares surfaces,” 2008.

[4] S. Madec, F. Baret, B. de Solan, S. Thomas, D. Dutartre, S. Jezequel,
M. Hemmerlé, G. Colombeau, and A. Comar, “High-throughput pheno-
typing of plant height: Comparing unmanned aerial vehicles and ground
LiDAR estimates,” Frontiers in Plant Science, vol. 8, nov 2017.

[5] G. J. Grenzdörffer, “Crop height determination with UAS point clouds,”
ISPRS - International Archives of the Photogrammetry, Remote Sensing
and Spatial Information Sciences, vol. XL-1, pp. 135–140, nov 2014.

## Related Publication
Li X., Bull G., Coe R., Eamkulworapong S., Scarrow J., Sirault X., Salim M., Schaefer M., 2019.
High-Throughput Plant Height Estimation from RGB Images Acquired with Aerial Platforms: a 3D
Point Cloud based Approach. Accepted by the International Conference on Digital Image Computing:Techniques and Applications (DICTA 2019). Perth, Australia, 02 Dec - 04 Dec 201

repository: Mary-xl/MarysBlog

repo_name: Mary-xl/MarysBlog


text:
# Campaign Script 2

Used to pull fb insights data into a csv, including custom conversions.

## Requirements

* Python 3
* Requests library (pip3 install requests)

### Required Files

 * accounts.txt - file containing a list of ad account id's separated by new lines
    ```
      12345
      67890
      1233
      12424
    ```
 * token.txt - file containing only the access token that can access your ad accounts
    ```
      EAABsbCS.....
    ```

## Usage:

```
python3 campaign_data_2.py --version

python3 campaign_data_2.py --ad_accounts_file accounts.txt --access_token_file token.txt --output_file output.csv
```

### Output Files

 * output.csv - file containing a list of ad account id's separated by new lines
    ```
      "account_id","account_name","campaign_id","campaign_name","account_currency","spend","reach","frequency","impressions","cpm","clicks","cpc","ctr","inline_link_clicks","inline_link_click_ctr","cost_per_inline_link_click","date_start","date_stop"
      "12345","account 12345","23457","campaign name","USD","129.12","45074","5.398804","252803","26.580516","346","6.038966","0.357005","266","0.269705","2.661129","2019-07-01","2019-07-31"
    ```

license: MIT

repository: dosz-fb/SE-projects

branch_name: refs/heads/master

extension: md

license_type: permissive

repo_name: dosz-fb/SE-projects


text:
# Derived PHP images from original repo
Dockerized version of PHP, based on original PHP images, but with some additional module

## There are various versions depending on needs
- 7.4-apache : many modules (phpredis, mailparse, mysqli, bz2, gd, zip, pdo_mysql)
- 7.2-epe : very custom image
- ~~7.4-fpm : phpredis, pdo_mysql ~~ DON'T USE IT ANYMORE, use 7.4-fpm instead
- 7.4-fpm-redis : (phpredis, pdo_mysql)
- ~~7.4-apache-redis : (phpredis, pdo_mysql) and apache ~~ DON'T USE IT ANYMORE, use 7.4-apache instead
- 7.3-fpm : like 7.2-epe but with PHP 7.3
- 5.6-apache: many modules (mysqli, bz2, gd, zip, pdo_mysql). NO Redis support, NO mailparse support

repository: Neomediatech/php

repo_name: Neomediatech/php


text:
# 18650C HAT

<el-tag effect="plain">SKU:U080</el-tag>

## Description

**18650C HAT** is a rechargeable battery base designed for M5SticKC with built-in 18650 large capacity rechargeable lithium battery.It can provide long-term endurance for the equipment. The base is designed with the plug-in structure of the HAT series, which can be used to maintain reliable connection with the M5StickC. The bottom is equipped with a USB interface charging interface, which is convenient and quick for the battery. The USB port of the battery base is only used as a charging interface, and does not have UART function.There are many types of mounting holes on the back of the base,which can be easily installed by the user into all types of planes.

Note: The M5StickC produced after October 2019 has a new fixing hole design. If your M5StickC is an old device (no fixing hole), you cannot use screws to fix it to the 18650CHAT, but this does not affect Functional use of modules

## Product Features

- Chargable
- Battery replaceable
- 2200mAh rechargeable battery

## Include

- 1x 18650 HAT（include 2200mAh battery)

## Specification

      Resources
      Parameter

      Net weight
      90g

      Gross weight
      94g

      Product Size
      150*24*24mm

      Package Size
      170*20*24mm

gha_language: JavaScript

gha_license_id: MIT


text:
# Nim language {#nim}

* https://nim-lang.org/
    * https://forum.nim-lang.org/

## Links

* [picheta] Dominik Picheta

    [Nim in Action](https://www.manning.com/books/nim-in-action)

    Manning, August 2017  ISBN<PHONE_NUMBER>436  320 pages
    * https://livebook.manning.com/book/nim-in-action/about-this-book/
* [salewski] Dr. Stefan Salewski

    [Computer Programming with the Nim Programming Language -- A gentle Introduction](https://github.com/StefanSalewski/NimProgrammingBook/blob/master/nimprogramming.adoc)

## Web

### Karax
### Jester

## GUI

### `ui`

* https://nicedoc.io/nim-lang/ui

### GTK

* Stefan Salewski

    [GTK4 for Graphical User Interfaces (in Nim)](https://github.com/StefanSalewski/GtkProgrammingBook/blob/master/gtkprogramming.adoc)

## IDE/Editors

* https://nim-lang.org/nimedit/
* https://github.com/nim-lang/Aporia
    * https://wiki.gnome.org/Projects/GtkSourceView
        * https://gitlab.gnome.org/GNOME/gtksourceview

repository: ponyatov/cp

branch_name: refs/heads/main

repo_name: ponyatov/cp


text:
# basekit
The commonly used go language development kit

[![License](https://img.shields.io/:license-apache-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Go Report Card](https://goreportcard.com/badge/github.com/alex023/basekit)](https://goreportcard.com/report/github.com/alex023/basekit)
[![GoDoc](https://godoc.org/github.com/alex023/basekit?status.svg)](https://godoc.org/github.com/alex023/basekit)
[![Build Status](https://travis-ci.org/alex023/basekit.svg?branch=master)](https://travis-ci.org/alex023/basekit)

Some usual toolkit，include：
## pub-sub
Just mediate implemention in memory(publish：Topic，subscribe：channel）
## singleflight
only duplicate of singleflight in `groupcache`
## svc
program init,start,safe exit
## hash
- consistent:
- hrw:provides an implementation of Highest Random Weight hashing, an alternative to consistent hashing which is both simple and fast
## other
- counter:multithreading counter
- waitwraper:a wrapper for simplify  calling for  waitgroup

repository: zenling/basekit

repo_name: zenling/basekit


text:
# Creating an Eclipse installation for Spoofax development

If you are developing a project that is included in Spoofax it is recommended to set up Eclipse for Spoofax development. This section describes how to create an Eclipse installation for developing Spoofax components.

## Requirements

1. A working **Spoofax and StrategoXT build** is required before being able to develop. Follow the [Building Spoofax and StrategoXT](/dev/build/#building-spoofax-and-strategoxt) section for instructions on how to build Spoofax and StrategoXT.

## Generating an Eclipse instance

The `releng` script in the spoofax-releng repository can generate an Eclipse installation for you. Change directory into the spoofax-releng repository and run:

{% highlight bash %}
./releng gen-dev-spoofax -d ~/eclipse/spoofax-dev
{% endhighlight %}

This will download and install Eclipse into `~/eclipse/spoofax-dev` with the right plugins and eclipse.ini for Spoofax development. The latest nightly version of the Spoofax plugin will be installed into that Eclipse. If you would like to install your locally built Spoofax plugin instead, pass the `-l` flag:

{% highlight bash %}
./releng gen-dev-spoofax -l -d ~/eclipse/spoofax-dev
{% endhighlight %}

Generating an Eclipse installation can take several minutes. After it's done generating, open the Eclipse installation and confirm that it works by creating a Spoofax entity project.

## Fixing Eclipse settings

Some Eclipse settings unfortunately have sub-optimal defaults. Go to the Eclipse preferences and set these options:

* General
	* Enable: Keep next/previous editor, view and perspectives dialog open
* General → Startup and Shutdown
	* Enable: Refresh workspace on startup
* General → Workspace
	* Enable: Refresh using native hooks or polling
* Maven
	* Enable: Do not automatically update dependencies from remote repositories
	* Enable: Download Artifact Sources
	* Enable: Download Artifact JavaDoc
* Maven → User Interface
  * Enable: Open XML page in the POM editor by default
* Run/Debug → Launching
	* Disable: Build (if required) before launching

## Developing

Now check out any projects you want to work on, import them into Eclipse, and start developing!

To test changes, a new Eclipse instance needs to be started.
Clone the `https://github.com/metaborg/spoofax.git` repository and import the `org.strategoxt.imp.runtime` project into Eclipse, which provides launch configurations for starting new Eclipse instances.

Press the little down arrow next to the bug icon (next to the play icon) and choose `Spoofax (no assertions)` to start a new Eclipse instance that contains your changes. To run with assertions enabled, use the `Spoofax (with assertions)` item, this will also provide more debugging output in the console.

## Advanced: developing from scratch

In some cases it can be beneficial to have full control over all plugin projects, instead of relying on an installed Spoofax plugin. Only follow this approach if you know what you are doing!

To develop from scratch, uninstall Spoofax from Eclipse, and import the following projects into Eclipse:

* imp-patched
	* org.eclipse.imp.cheatsheets
	* org.eclipse.imp.java.hosted
	* org.eclipse.imp.lpg
	* org.eclipse.imp.lpg.ide
	* org.eclipse.imp.lpg.metatooling
	* org.eclipse.imp.metatooling
	* org.eclipse.imp.preferences
	* org.eclipse.imp.prefspecs
	* org.eclipse.imp.presentation
	* org.eclipse.imp.runtime
	* org.eclipse.imp.smapi
	* org.eclipse.imp.smapifier
	* org.eclipse.imp.xform
* mb-rep
	* org.spoofax.aterm
	* org.spoofax.interpreter.library.index
	* org.spoofax.terms
	* org.strategoxt.imp.editors.aterm
* mb-exec
	* org.spoofax.interpreter
	* org.spoofax.interpreter.adapter.ecj
	* org.spoofax.interpreter.core
	* org.spoofax.interpreter.library.eclipse
	* org.spoofax.interpreter.library.interpreter
	* org.spoofax.interpreter.library.java
	* org.spoofax.interpreter.library.jline
	* org.spoofax.interpreter.library.xml
	* org.spoofax.interpreter.ui
* mb-exec-deps
	* org.spoofax.interpreter.externaldeps
* runtime-libraries
	* org.metaborg.runtime.task
	* org.metaborg.meta.lang.analysis
	* org.spoofax.meta.runtime.libraries
* spoofax
	* org.strategoxt.imp.generator
	* org.strategoxt.imp.metatooling
	* org.strategoxt.imp.nativebundle
	* org.strategoxt.imp.runtime
	* org.strategoxt.imp.runtime.sidebyside.latest
	* org.strategoxt.imp.runtime.sidebyside.main
	* make-permissive-jar-installer (inside ```org.strategoxt.imp.generator/morepoms```)
* spoofax-sunshine
	* org.metaborg.sunshine
* meta languages
	* stratego
		* org.strategoxt.imp.editors.stratego
	* spt
		* org.strategoxt.imp.testing
		* org.strategoxt.imp.testing.ui
	* esv
		* org.strategoxt.imp.editors.editorservice
	* sdf
		* org.strategoxt.imp.editors.sdf
		* org.strategoxt.imp.editors.template
	* aster
		* org.strategoxt.imp.editors.aster
	* box
		* org.strategoxt.imp.editors.pp
	* nabl
		* org.metaborg.meta.lang.nabl
	* rtg
		* org.strategoxt.imp.editors.rtg
	* ts
		* org.metaborg.meta.lang.ts
	* dynsem
		* org.metaborg.meta.interpreter.framework
* jsglr
	* org.spoofax.interpreter.library.jsglr
	* org.spoofax.jsglr
* spoofax-debug
	* org.strategoxt.imp.debug.core
	* org.strategoxt.imp.debug.stratego.core
	* org.strategoxt.imp.debug.stratego.runtime
	* org.strategoxt.imp.debug.stratego.test
	* org.strategoxt.imp.debug.stratego.transformer
	* org.strategoxt.imp.debug.ui
* modelware
	* org.spoofax.modelware.emf
	* org.spoofax.modelware.gmf
* strategoxt
	* org.strategoxt.jarprovider
	* org.strategoxt.strj
* deployment
	* org.metaborg.maven.build.java
	* org.metaborg.maven.build.spoofax.eclipse
	* org.metaborg.maven.build.spoofax.libs
	* org.metaborg.maven.build.spoofax.sunshine
	* org.metaborg.maven.build.strategoxt
	* org.metaborg.maven.parent
	* org.metaborg.maven.parent.java
	* org.metaborg.maven.parent.language
	* org.metaborg.maven.parent	.plugin
	* org.spoofax.modelware.emf.feature
	* org.spoofax.modelware.gmf.feature
	* org.strategoxt.imp.feature
	* org.strategoxt.imp.updatesite
* other dependencies
	* shrike
		* com.ibm.wala.shrike
	* lpg-runtime
		* lpg.runtime.java

If you change a language project, run the Spoofax build on the command-line, because those languages cannot be built inside Eclipse without the Spoofax plugin.

You **must turn off 'Project → Build Automatically'** in Eclipse, otherwise the Maven and Eclipse compilers will interfere and possibly fail the build. After the Maven build is finished, enable 'Build Automatically' again.
{: .notice .notice-warning}


text:
---
layout: post
title: How to Add A Comment System to Your Blog
description: "The comment system is easy."

I planned to read several papers this afternoon, but once heard from [Patrick](http://patrick-peng.me/) that the [github student pack](https://education.github.com/pack) can provide a free domain name for one year, I could wait to try it. However, as soon as I set up my redirection, the comment system I previously built was down. I tried another one, but with the same failure. What was really interesting was that using Safiri, everything was fine, but testing on my own computer, the comment system was reluctant to come out. Well, in the end, I switched back to the previous system and gave up the free domain name.

Recently, I have been working on my blog for its improvements. Here, I would introduce two comment systems that can be added into your blog, if you are interested in having one. A blog would not be a real blog if it has no comment system, right?

OK, here we go！

## Facebook Comment Plugin
I personally would not recommend this comment system due to that it can only use Facebook account to comment. However, having a Facebook comment system in your blog is much more awesome, maybe, at least to some of us. Seriously, through my own experience, this comment system is not as good as the second one.

Fine, cut the useless talk. Let us begin.

First, visit the [Facebook developer website](https://developers.facebook.com/) and log in. Look to your top right and find the button shown in the figure below.

Click it and click 'Add a New App'. Then choose 'Website' and name your app. You will find you have already got a segment of codes. Put them in the file which will be included in any page file that you want to show the system, traditionally in the  tag, or in the 'head.html' as I do. What you get is something like this:

``` html
<script>
  window.fbAsyncInit = function() {
    FB.init({
      appId      : 'your app id',
      xfbml      : true,
      version    : 'v2.7'
    });
  };

  (function(d, s, id){
     var js, fjs = d.getElementsByTagName(s)[0];
     if (d.getElementById(id)) {return;}
     js = d.createElement(s); js.id = id;
     js.src = "//connect.facebook.net/en_US/sdk.js";
     fjs.parentNode.insertBefore(js, fjs);
   }(document, 'script', 'facebook-jssdk'));
</script>
```

Then type in your website link and click 'next'. Scroll down to 'Next Steps' and click 'Social Plugins' which opens a new page. Scroll down a little to find this:

Click 'Read the Docs' and you should see the Comments Plugin's document. You do not have to configure any of the blanks shown, just click 'Get Code' and two segments of codes will appear. Copy the first segment in your  tag like this:

``` html
<body id="post" {% if page.image.feature %}class="feature"{% endif %}>

<div id="fb-root"></div>
<script>(function(d, s, id) {
  var js, fjs = d.getElementsByTagName(s)[0];
  if (d.getElementById(id)) return;
  js = d.createElement(s); js.id = id;
  js.src = "//connect.facebook.net/en_US/sdk.js#xfbml=1&version=v2.7&appId=your app id";
  fjs.parentNode.insertBefore(js, fjs);
}(document, 'script', 'facebook-jssdk'));</script>

<!-- contents below -->
```

And copy the second segment to the place you want the system to show. Now you can read the introduction to the attributes of the system and configure them, but you can also add regardless of them like me, where all the values for the configuration will be default:

``` html
<center><div class="fb-comments"></div></center>
```

And you are done! I have given you all the facts that you need to know about how to insert Facebook comment system into your blog. Push and try it!

## Disqus
This one will be easier than the previous one and it has more ways for the commentor to log in and comment on your blog, through Facebook, Twitter, and Google. The appearance will be more satisfying, to be honest.

Visit the [official website of Disqus](https://disqus.com/) and get an account. Click 'GET STARTED':

Choose 'I want to install Disqus on my site'. Then type in the information and click 'Create Site'.
     *  LEARN WHY DEFINING THESE VARIABLES IS IMPORTANT: https://disqus.com/admin/universalcode/#configuration-variables */
    /*
    var disqus_config = function () {
        this.page.url = PAGE_URL;  // Replace PAGE_URL with your page's canonical URL variable
        this.page.identifier = PAGE_IDENTIFIER; // Replace PAGE_IDENTIFIER with your page's unique identifier variable
    };
    */
    (function() { // DON'T EDIT BELOW THIS LINE
        var d = document, s = d.createElement('script');
        s.src = '//shortname.disqus.com/embed.js';
        s.setAttribute('data-timestamp', +new Date());
        (d.head || d.body).appendChild(s);
    })();
    </script>
    Please enable JavaScript to view the comments powered by Disqus.
```

Copy them to where the comment system would show and, you are done here! Push and drink a cup of coffee.

It will be like this:

You can futher configure relative settings and use other functions provided by Disqus.

Well, the second part is a little short, mainly because Disqus is much more easier compared to Facebook Comment Plugin. Pfff. I spent the whole afternoon fighting against why the change of the domain name would cause the comment system to fail and I had not found the answer. I guess the problem result from the redirection. When setting the url in the comment system, it is a little bit tricky between the real url and the redirected url.

At least, the comment system works. Cheers.


text:
# Ardusat SDK
The Ardusat Sensor SDK is a software package designed to make interacting with the sensors found in
the [Ardusat Space Kit](http://www.ardusat.com/products) as easy as possible, while also providing
a powerful unified interface to use the same code to interact with ground-based sensors as well as
satellite systems. It builds on top of popular open source libraries provided by
[Adafruit](https://github.com/adafruit) and others.

## Installing the SDK
The first step to getting going is to install the Arduino Integrated Development Environment (IDE).
Downloads for all major OS versions can be found at
[http://arduino.cc/en/Main/Software](http://arduino.cc/en/Main/Software).

Once you have the IDE, installing the SDK is easy - it works like any other third party Arduino
library. Just [download the SDK](https://www.ardusat.com/downloads/ArdusatSDK.zip) or
clone this repository to your hard drive, then open the Arduino IDE, go to 'Sketch -> Import Library
-> Add Library' and navigate to your download (zip file or the directory cloned with Git). You now
should be able to use the SDK in your sketches.

If you're interested in using the SDK for logging data to an SD card, you'll need to do this same
thing with our other [Ardusat Logging SDK Library](http://github.com/ardusat/ardusatsdk-logging)

## Downloaded and Used the SDK before June 20, 2016?
If you want to use the latest SDK and you have existing Arduino sketches (from *before June 20,
2016*) that you would like to keep using, take a look at the
[Transition Documentation](https://github.com/ArduSat/ArdusatSDK/wiki/SDK-Transition-Information)
for how to update your sketches to work with the latest SDK.

## Using the SDK
The first step to using the SDK is to include it into your sketch. This can be done with a simple
include statement:

```cpp
#include <ArdusatSDK.h>
```

After the SDK is included, the basic I/O functions and sensor drivers should be available.

## Sensor Overview and Reference
Every Sensor in the SDK is accessible with a Sensor class that helps wrap up internal functionality.
Each Sensor Class supports reading data from one or more physical sensors.

#### Overview
Sensor Class | Sensors | Required Constructor Arguments
--- | --- | ---
Acceleration   | LSM303 (9DOF breakout)   | `None`
Gyro           | L3GD20 (9DOF breakout)   | `None`
Luminosity     | TCS34725                 | `None`
Magnetic       | LSM303 (9DOF breakout)   | `None`
Orientation    | Derived from Acceleration and Magnetic | `Acceleration & accel, Magnetic & mag` (Existing Accel and Mag objects)
Pressure       | BMP180                   | `None`
RGBLight       | TCS34725 (Default)       | `None`
RGBLightTCS    | TCS34725                 | `None`
RGBLightISL    | ISL29125                 | `None`
Temperature    | TMP102 (Default)         | `None`
TemperatureTMP | TMP102                   | `None`
TemperatureMLX | MLX90614 (IR)            | `None`
UVLight        | ML8511 (Default)         | `None`
UVLightML      | ML8511                   | `None`
UVLightSI      | SI1132                   | `None`

```cpp
UVLight uv;   // --> uv is now a sensor object that reads from the ML8511 Sensor
UVLightML uv; // --> uv is now a sensor object that reads from the ML8511 Sensor
UVLightSI uv; // --> uv is now a sensor object that reads from the SI1132 Sensor
```

Note that the Orientation object is not technically a sensor, but rather a value
derived from readings from the Acceleration and Magnetic Sensors. Therefore, you must
provide existing Acceleration and Magnetic objects when constructing the Orientation
object. It is used in the following way:

```cpp
Acceleration accel;
Magnetic mag;
Orientation orient(accel, mag); // --> orient uses existing Acceleration and Magnetic objects
orient.read();                  // --> orient calls accel's and mag's read and then derives a value
```

#### Common Functions
Every Sensor in the SDK has the following functions that can be used to initialize, read,
and print data:

Available Functions | Required Parameters | Description
--- | --- | --- | ---
`begin()`      | None                      | Initializes sensors with any optional advanced configurations
`read()`       | None                      | Takes a reading from the sensor
`readToCSV()`  | `const char * sensorName` | Takes a reading and prints it in a CSV format
`readToJSON()` | `const char * sensorName` | Takes a reading and prints it in a JSON format
`toCSV()`      | `const char * sensorName` | Prints the last reading taken in a CSV format
`toJSON()`     | `const char * sensorName` | Prints the last reading taken in a JSON format

* CSV  format: `timestamp,sensorName,values,...,checksum`
* JSON format: `~{"sensorName": "NAME", "unit": "UNIT", "value": 123.4, "cs": 123}|`

NOTE: The `toCSV` functions will not guarantee that you are outputting data in a format that is acceptable for
importing CSV logs into the [Ardusat Experiment Platform](http://experiments.ardusat.com). These functions
are for live STREAMING data into the Experiment Platform.
If you are calling the `toCSV` functions on exactly one type of Sensor Class per sketch, this is fine for
STREAMING data and LOGGING data to be imported.
If you are calling the `toCSV` functions on more than one type of Sensor Class per sketch, this is fine for
only STREAMING data, LOGGING data to be important will not be formatted correctly. If you want to LOG more
than one type of Sensor Class, see the CSV Logging Example with `valuesToCSV`.

There is a full example in the CSV Examples of how to use `valuesToCSV`, but this function is available to make
it easier to format data properly:
```cpp
const char * valuesToCSV(const char *sensorName, unsigned long timestamp, int numValues, float values...);
const char * valueToCSV(const char *sensorName, float value, unsigned long timestamp); // if timestamp == 0, it will just calculate a timestamp using millis()
```

There is a similar function for formatting arbitrary values in the JSON format called `valuesToJSON`:
```cpp
const char * valuesToJSON(const char *sensorName, unsigned char unit, int numValues, float values...);
const char * valueToJSON(const char *sensorName, unsigned char unit, float value);
```

#### Sensor Specifics
This is an overview of the sensor specific fields and advanced configuration parameters

Sensor Class | Data Fields | Optional Constructor Parameters (for advanced configuration)
--- | --- | ---
Acceleration   | `float x`, `float y`, `float z`              | `lsm303_accel_gain_e gGain`
Gyro           | `float x`, `float y`, `float z`              | `uint8_t range`
Luminosity     | `float lux`                                  | `tcs34725IntegrationTime_t intTime`, `tcs34725Gain_t gain`
Magnetic       | `float x`, `float y`, `float z`              | `lsm303_mag_scale_e gaussScale`
Orientation    | `float roll`, `float pitch`, `float heading` | `None`
Pressure       | `float pressure`                             | `bmp085_mode_t mode`
RGBLight       | `float red`, `float green`, `float blue`     | `None`
RGBLightTCS    | `float red`, `float green`, `float blue`     | `tcs34725IntegrationTime_t it`, `tcs34725Gain_t gain`
RGBLightISL    | `float red`, `float green`, `float blue`     | `uint8_t islIntensity`
Temperature    | `float t`                                    | `None`
TemperatureTMP | `float t`                                    | `None`
TemperatureMLX | `float t`                                    | `None`
UVLight        | `float uvindex`                              | `None`
UVLightML      | `float uvindex`                              | `int pin`
UVLightSI      | `float uvindex`                              | `None`

These are the specific values avaliable for each of the advanced configurations

Constructor Configuration Parameter Type | Acceptable Values | Sensor Class
--- | --- | ---
`lsm303_accel_gain_e gGain` | `LSM303_ACCEL_GAIN2G`, `LSM303_ACCEL_GAIN4G`, `LSM303_ACCEL_GAIN6G`, `LSM303_ACCEL_GAIN8G` (Default), `LSM303_ACCEL_GAIN16G` | Acceleration
`uint8_t range` | `0x00` (SENSITIVITY_250DPS), `0x10` (SENSITIVITY_500DPS), `0x20` (SENSITIVITY_2000DPS) (Default) | Gyro
`tcs34725IntegrationTime_t intTime` | `TCS34725_INTEGRATIONTIME_2_4MS`, `TCS34725_INTEGRATIONTIME_24MS` (Default), `TCS34725_INTEGRATIONTIME_50MS`, `TCS34725_INTEGRATIONTIME_101MS`, `TCS34725_INTEGRATIONTIME_154MS`, `TCS34725_INTEGRATIONTIME_700MS` | Luminosity
`tcs34725Gain_t gain` | `TCS34725_GAIN_1X`, `TCS34725_GAIN_4X`, `TCS34725_GAIN_16X` (Default), `TCS34725_GAIN_60X` | Luminosity
`lsm303_mag_scale_e gaussScale` | `LSM303_MAG_SCALE1_3GAUSS`, `LSM303_MAG_SCALE2GAUSS`, `LSM303_MAG_SCALE2_5GAUSS`, `LSM303_MAG_SCALE4GAUSS` (Default), `LSM303_MAG_SCALE4_7GAUSS`, `LSM303_MAG_SCALE5_6GAUSS`, `LSM303_MAG_SCALE8GAUSS`, `LSM303_MAG_SCALE12GAUSS` | Magnetic
`bmp085_mode_t mode` | `BMP085_MODE_ULTRALOWPOWER`, `BMP085_MODE_STANDARD`, `BMP085_MODE_HIGHRES`, `BMP085_MODE_ULTRAHIGHRES` (Default) | Pressure
`tcs34725IntegrationTime_t it` | `TCS34725_INTEGRATIONTIME_2_4MS`, `TCS34725_INTEGRATIONTIME_24MS`, `TCS34725_INTEGRATIONTIME_50MS`, `TCS34725_INTEGRATIONTIME_101MS`, `TCS34725_INTEGRATIONTIME_154MS` (Default), `TCS34725_INTEGRATIONTIME_700MS` | RGBLightTCS
`tcs34725Gain_t gain` | `TCS34725_GAIN_1X` (Default), `TCS34725_GAIN_4X`, `TCS34725_GAIN_16X`, `TCS34725_GAIN_60X` | RGBLightTCS
`uint8_t islIntensity` | `CFG1_375LUX`, `CFG1_10KLUX` (Default) | RGBLightISL
`int pin` | `DRIVER_ML8511_UV_PIN` (A0) (Default), or any other Arduino Pin | UVLightML

#### Units
Sensor Class | Unit Type | Unit String
--- | --- | ---
Acceleration | DATA_UNIT_METER_PER_SECONDSQUARED | "m/s^2"
Gyro         | DATA_UNIT_RADIAN_PER_SECOND       | "rad/s"
Luminosity   | DATA_UNIT_LUX                     | "lux"
Magnetic     | DATA_UNIT_MICROTESLA              | "uT"
Orientation  | DATA_UNIT_DEGREES                 | "deg"
Pressure     | DATA_UNIT_HECTOPASCAL             | "hPa"
RGBLight     | DATA_UNIT_LUX                     | "lux"
Temperature  | DATA_UNIT_DEGREES_CELSIUS         | "C"
UVLight      | DATA_UNIT_MILLIWATT_PER_CMSQUARED | "mW/cm^2"
None         | DATA_UNIT_NONE                    | ""

#### Sensor Notes
The Barometric Pressure sensor has two additional convenience functions to calculate altitude (which
requires knowing the current sea level barometric pressure, a value that's easily available from
weather observation data), or current sea level barometric pressure (which requires knowing the
current altitude). For these convenience functions, altitude should be provided in meters, and
pressure values should be provided in hPa.

```cpp
float Pressure::altitudeFromSeaLevelPressure(float seaLevelPressure);
float Pressure::seaLevelPressureFromAltitude(float altitude);
```

To translate meters to feet, multiply the meter value by `3.28084`. To translate feet to meters,
multiply the feet value by `0.3084`.

#### Global Variables
Allows the user to manually decide in an Arduino sketch if the SDK should
dynamically check if the SpaceBoard is being used or not.
Defaults to false
```cpp
boolean MANUAL_CONFIG;
```

Used when dynamically checking if the SpaceBoard is being used or not. If
the SpaceBoard is being used, different addresses might be used for each
sensor.
```cpp
boolean ARDUSAT_SPACEBOARD;
```

Example Usage
```cpp
...

void setup(void) {
  MANUAL_CONFIG = true;
  ARDUSAT_SPACEBOARD = false;
  ...
}

...
```

You will likely never need to use these, unless you are interested in building
your own hardware setup with new sensors, or are curious to see what will happen.

## Sensor Usage Examples
```cpp
#include <ArdusatSDK.h>

Luminosity lum;

void setup(void) {
  Serial.begin(9600);
  lum.begin();
}

void loop(void) {
  Serial.println(lum.readToJSON("Luminosity")); // >> ~{"sensorName": "Luminosity", "unit": "lux", "value": 123.5, "cs": 43}|
  Serial.println(lum.readToJSON("lum2"));       // >> ~{"sensorName": "lum2", "unit": "lux", "value": 121.9, "cs": 30}|
  Serial.println(lum.readToCSV("Luminosity"));  // >> 243,Luminosity,128.2,49
  Serial.println(lum.readToCSV("lum4"));        // >> 311,lum4,124.6,29
  Serial.println(lum.lux);                      // >> 124.6
  Serial.println(lum.lux);                      // >> 124.6
  lum.read();
  Serial.println(lum.lux);                      // >> 127.5
}
```

```cpp
#include <ArdusatSDK.h>

RGBLightISL rgb(CFG1_375LUX);

void setup(void) {
  Serial.begin(9600);
  rgb.begin();
}

void loop(void) {
  Serial.println(rgb.readToJSON("RGBLight")); // >> ~{"sensorName": "RGBLight", "unit": "lux", "red": 89.2, "green": 177.9, "blue": 183.2, "cs": 83}|
}
```

## CSV Examples
As mentioned in the *NOTE* in "Sensor Overview and Reference", the Experiment Platform reads reads two different types of CSV
formats. The `toCSV` family of functions work fine on multiple types of Sensor Classes at a time when you want to STREAM data,
but if you want to LOG data to be imported, you need to have properly formatted data.

####CSV STREAMING example:
```cpp
#include <ArdusatSDK.h>

Magnetic mag;
Luminosity lum;

void setup(void) {
  Serial.begin(9600);
  mag.begin();
  lum.begin();
}

void loop(void) {
  Serial.println(mag.readToCSV("Magnetic"));
  Serial.println(lum.readToCSV("Luminosity"));
}
```
Output: (not proper CSV format) Will fail to be imported to Experiment Platform
        However, this is fine if you're streaming CSV data to the Experiment PLatform.
```cpp
89,Magnetic,0.78,9.11,0.02,123
123,Luminosity,128.2,234
145,Magnetic,0.74,9.01,0.03,234
168,Luminosity,126.7,423
...
```

####CSV LOGGING example:
```cpp
#include <ArdusatSDK.h>

Acceleration accel;
Temperature temp;

void setup(void) {
  Serial.begin(9600);
  accel.begin();
  temp.begin();

  // CSV Headers to print once at the top of the data
  Serial.print("timestamp (millis), name, ");
  Serial.print("temperature (C), x acceleration (m/s^2), y acceleration (m/s^2), z acceleration (m/s^2), ");
  Serial.println("checksum");
}

void loop(void) {
  accel.read();
  temp.read();

                            //Log Name, Time Stamp, Number of Values, Temp, Accel X, Accel Y, Accel Z
  Serial.println(valuesToCSV("reading", accel.header.timestamp, 4, temp.t, accel.x, accel.y, accel.z));
}
```
Output: Proper CSV format. This will be successfully imported into the Experiment Platform
        However, this will not work if you're streaming CSV data to the Experiment PLatform.
```cpp
timestamp (millis), name, temperature (C), x acceleration (m/s^2), y acceleration (m/s^2), z acceleration (m/s^2), checksum
89,reading,27.3,9.11,0.02,0.78,123
123,reading,27.2,9.10,0.08,0.73,120
145,reading,27.3,9.02,0.11,0.88,121
198,reading,27.2,9.08,0.03,0.80,128
211,reading,27.3,9.13,0.08,0.75,119
229,reading,27.5,9.19,0.01,0.78,127
...
```

## Different Output Formats
The Ardusat SDK can output sensor data in both JSON and CSV format to allow interfacing with
external systems such as the [Ardusat Experiment Platform](http://experiments.ardusat.com). To use these functions,
call the `toJSON` or `toCSV` family of functions:

### Checksum
Both JSON and CSV output formats optionally include checksum values to verify that the data remains
uncorrupted through transmission. This is an integer value that is calculated when the data packet
is first written in a known way. The client receiving the data packet can then re-calculate the
checksum using the same algorithm, then verify that the calculated value matches the transmitted
checksum. The algorithm used to calculate the checksum sums the characters in the `sensorName` field
as integers, then rounds the value (s) and adds this to the sum of the name characters.

Example checksum calculation:

```cpp
int calculate_checksum(const char *sensorName, float value) {
  int checksum = 0;
  int i, len;

  len = strlen(sensorName);
  for (i = 0; i < len; ++i) {
    checksum += sensorName[i];
  }
  checksum += lroundf(value);
  return checksum;
}
```

### Serial Library
The regular Arduino `Serial` library works fine with the SDK. However, sometimes when connecting to
XBee or other wireless modules, it is useful to be able to wire serial connections that leave the
main Arduino USB Serial free for programming. To do this, the SDK leverages the `SoftwareSerial`
library in a new class called `ArdusatSerial`. This class has the same interface as both `Serial`
and `SoftwareSerial`, but can be configured to output on hardware serial (`Serial`), software
serial, or both.

If a software serial mode is selected, the `ArdusatSerial` constructor must be supplied with
arguments for `softwareReceivePin` and `softwareTransmitPin`:

```cpp
ArdusatSerial(serialMode mode, unsigned char softwareReceivePin, unsigned char softwareTransmitPin);
```

Usage:
```cpp
ArdusatSerial serialConnection(SERIAL_MODE_HARDWARE_AND_SOFTWARE, 10, 11);
Temperature temp;

void setup(void) {
  serialConnection.begin(9600);
  temp.begin();

  serialConnection.println("This message will go out on hardware serial and software serial!");
}

void loop(void) {
  temp.read();
  serialConnection.println("The recorded temperature is ");
  serialConnection.println(temp.t);
}

```

#### Serial Modes
Name | Description
--- | ---
**SERIAL_MODE_HARDWARE** | Output serial data on built-in USB hardware serial
**SERIAL_MODE_SOFTWARE** | Output serial data on software serial (must specify transmit and receive pins in the constructor, see arguments in example above)
**SERIAL_MODE_HARDWARE_AND_SOFTWARE** | Output to both hardware and software serial interfaces

#### Limitations
SoftwareSerial does not appear to work reliably above 57600 baud.

# Getting Help
If you're having trouble running the examples, chances are something is messed up with the external
library locations in your Arduino IDE. Double check that the ArdusatSDK library is included into
your Arduino libraries (Sketch -> Import Libraries -> Contributed). If the sketches are compiling
and uploading but not behaving as expected, make sure you double check your wiring, it's always easy
to accidentally plug something in wrong!

If you get really stuck, feel free to reach out at<EMAIL_ADDRESS>or the "Issues" section of
this repository.

# Attributions
This SDK leverages some really excellent code already written by
[Adafruit](https://www.adafruit.com), [SparkFun](https://www.sparkfun.com),
and [Pololu](https://www.pololu.com). They have all provided really great sensor
drivers and Arduino Libraries, so please support them in thanks for their efforts!


text:
# Odin-Securities 1.0

A master database of securities data for use with the Odin algorithmic trading platform. The database includes information on historical stock prices, on corporate actions, and includes dividend and stock-split adjusted prices and volumes.

This software is provided under the MIT license.

## Setup & Installation

It is recommended that you read all of these instructions before beginning to install the Odin Securities database. Please ensure that the Odin's `requirements.txt` file has been installed.

### Postgres

You will first need to install Postgres. Instructions on installing Postgres can be found on the [official site](https://www.postgresql.org/). From there, you will need to create a database called `securities_master` and a user called `securities`. This can be achieved by the commands:

```
CREATE DATABASE securities_master;
CREATE USER securities;
```

To allow Python to interface with Postgres, the execute the following three commands to install packages.

```
sudo apt-get install postgresql
sudo apt-get install python-psycopg2
sudo apt-get install libpq-dev
```

To initialize the tables in the Odin Securities database, navigate to the `odin_securities` directory and execute `make`. This will instruct Postgres to construct a database in its default configuration. In the event that you need to start over from scratch, simply execute `make` again (if the command appears to hang, execute `make restart` and try again).

### Data Vendors

Odin leverages both Yahoo! Finance and Quandl to provide price and volume data. To leverage Quandl as a data vendor, you will require a Quandl API key. This can be obtained by signing up at their website: [Quandl.com](https://www.quandl.com/). With this in hand, edit the file `./odin_securities/vendors/quandl/quandl_init.py` to specify your particular API key. Please ensure that this is done before you install Odin Securities itself; if you already installed Odin Securities, you will have to reinstall it.

### Installing Odin Securities and Building the Database

To install Odin Securities itself, execute the command:

```
python3 setup.py install
```

Installation, however, is not as important as constructing the database itself. For this purpose, refer to the file `./constructor/constructor.py`, which will both attempt to build the database from scratch and keep it updated with the latest price data. To build or update Odin Securities, navigate into the `constructor` directory and run:

```
python3 constructor.py
```

**Though this command takes a very long time to execute.** Therefore, it is recommended instead to build a smaller version of the database with Alphabet's, Facebook, and Amazon's stock prices with the command:

```
python3 constructor.py -i 'GOOG FB AMZN'
```

To update the data only for symbols that are already present in the database, supply the optional local `-l` argument:

```
python3 constructor.py -l
```

repository: parrondo/Odin-Securities

repo_name: parrondo/Odin-Securities


text:
# trending

## Features

RSS Feed for trending hashtags and words in Amsterdam area.

## Usage

Create a `trending.ini` configuration file
```
[default]
tweets_file = tweets.txt
baseurl = http://localhost

[twitter]
consumer_key = <key here>
consumer_secret = <secret here>
access_token = <access token here>
access_token_secret = <access token secret here>

```

Start by
`PYTHONPATH=src: python src/trending/main.py`

Following RSS Feeds are exposed
- `<host>:<port>/hashtags`
- `<host>:<port>/words`

Other URLS will return a 404 error.
PS: analysis is not real time as of yet.

## CI/CD pipeline
Included in the setup is some groundwork to create an AWS pipeline for running this application using Terraform.

See the `pipeline` folder.

## Development / testing

### Run unittests / coverage report
```
python setup.py test

```

repository: ffung/trending

gha_language: Python

repo_name: ffung/trending


text:
---
layout: post
title:  "Programming (Chess) Puzzles with a Tweet"
date:   2020-04-22 011:54:29 +0200
tags: [programming, puzzles, chess, minification, recreative, python]
---

A friend posed the following puzzle/problem on social media: *"Given a 8x8 chessboard, your goal is to place 4 queens  and 1 bishop so that all squares of the board are controlled (through diagonales/lines; a piece controls the square where it is located)."* My usual reaction is to either promptly ignore this kind of fake problem or to try the resolution for real on a concrete chessboard or mentally, [sans voir](https://en.wikipedia.org/wiki/Blindfold_chess). But in this >quarantine period, I wanted to find a solution with a *program* (in next blog posts, I may explain how this attitude becomes a pattern beyond chess puzzles). Here is a short story about the process that lead to a Python solution in less than 280 characters that fits in a Tweet.

## The chess puzzle

An animation is worth than thousands of words for explaining the problem/puzzle[^1]. The animation below almost shows a solution. 4 queens and 1 bishop are placed into a regular chess board. Circles in red represent squares that are not controlled (and yet to be controlled). The number at the top left indicates the number of squares not controlled (the goal is to reach 0). At the end of the animation, there is still missing one square ;)

![Chess Puzzle](../assets/chesspuzzle.gif)

In fact, with a physical chessboard, I quickly found a solution with 63 squares (basically with some tries and errors). But the 64 graal was quite hard to achieve. Maybe there is no solution and it was a joke? I had the feeling I could succeed with a bit of time, but it would be like "déjà vu": yet another puzzle solved, what's next? Don't get me wrong, there is lot of fun and satisfaction to resolve a puzzle, but I wanted to have another kind of fun. So I decided to program a solution!

## Programming

[As most (?) of the programmers](https://en.wikipedia.org/wiki/Software_development_effort_estimation#Psychological_issues), I even had the impression I would be able to quickly find a solution like in a few minutes (1 hour maximum yea). After all the algorithm is quite simple: try (all) possible placements of 4 queens and 1 bishop, assess each try by counting the number of controlled squares, and stop at 64. So I launched a [Python notebook and started an implementation](https://github.com/acherm/progpuzzles/blob/master/ChessPuzzle.ipynb).

For trying possible placements, I consider that the placement of a piece (whether it is a queen or a bishop) is a number between 0 and 63 (see [for a visual representation](https://www.chessprogramming.org/Efficient_Generation_of_Sliding_Piece_Attacks#Squares_and_Bitindex) of the board). A possible placement is a 5-tuple and the last element is the placement of the bishop. There are many possible placements and I try to generate *all*: like this, I'm sure to explore the whole space, I can compute some statistics of all solutions or eventually have a solid proof that there is no solution. There are different variants of the generation: from very naive but too computationally expensive (all possible permutations of length 5) to the exploitation of the symmetry of the chess board.

The second task is to program a procedure that assesses whether a placement controls 64 squares. You need to compute all squares controlled by a given queen or bishop, and then make the union of all squares. So a bit of maths/algorithms and ways to represent the board/squares/pieces for computing diagonals, horizontals, verticals.

So I got some solutions (btw you can find more [statistics about all solutions of the puzzle](https://github.com/acherm/progpuzzles/blob/master/ChessPuzzle.ipynb)):

```python
for cmb in gen_combinations():
    n = nbcases_covered(cmb)
    if (n > 63): # the solution(s)
        print(cmb, "=>", n, ntries)
        break
```

Looking back at my experience, I was a bit... puzzled ;) I was thinking it should have taken a few minutes but I was wrong, more something like 4 hours. I under-estimated the time of some tasks:

* scaling issues with combinations
* I have to remember how to properly represent a board and movements of pieces. It's not that easy and I forget also the case of squares a queen don't control because the bishop blocks the horizontal/vertical movements
* the amount of code for programming `nbcases_covered` is an indicator: not *that* straightforward

There are obviously some threats to my experience (I didn't perform the task in one shot and the quarantine at home is not a long quiet river...), but I guess similar issues arise when you program from scratch: there are many (accidental) details to deal with.

## Opportunistic Reuse

Don't you hear the voice of *"Don't reinvent the wheel, reuse!"*? This principle applies to many cases, but reuse what? it's a chess puzzle with 4 queens and 1 bishop... Is there a chess library that supports it?

I knew [python-chess](http://python-chess.readthedocs.io/) quite well, I even implemented a chess game player on top of it, within a notebook (see my [blog post](http://blog.mathieuacher.com/JupyterChess/)). But I was thinking the library was not *designed* for tackling my problem, like: it cannot handle illegal chess positions (eg without kings!), it won't help me much (eg it's mainly a library for reading PGN game), or I will hardly generalize my solution to strange cases (eg like puzzle with 6x6 board) and I don't want to stuck on a third party library. I was wrong and missed an opportunity.

So I decided to apply what I called *opportunistic reuse*, i.e., reusing a library that is not supposed to resolve your exact problem (who says "hack" a library). After further exploring the library, all the assumptions above collapsed:

```python
# we reuse built-in facilities of python-chess for generating moves
def nbcases_covered_reuse(cmb):
    board = chess.Board()
    board.clear_board()

    for q in range(0, 4):
        board.set_piece_at(cmb[q], chess.Piece(chess.QUEEN, color=True))
    board.set_piece_at(cmb[4], chess.Piece(chess.BISHOP, color=True))

    squares = []
    for q in range(0, 5):
        squares.append(cmb[q])
        squares = squares + list(board.attacks(cmb[q]))

    return len(set(squares))
```

The library does have facilities to compute the controlled squares (with the `attacks` method) and you can well give positions without kings. The method for computing the number of cases is dramatically short: I don't have to design about what is a board, a piece, a square, etc.

Thanks to opportunistic reuse, I have now an elegant and concise solution (bonus: it's certainly better in terms of performance and testing maturity). For me, it's a lesson learned: think deeply before not considering (opportunistic) reuse. Retrospectively it is now obvious.

## Tweetification of the solution

Now we are here, why not going further and produce an even more concise solution? The first motivation is that opportunistic reuse has already drastically reduced the amount of code, but we can do better! Secondly, chess has some nice stories with short programs (e.g., chess engine with only [760 characters C](https://en.wikipedia.org/wiki/International_Obfuscated_C_Code_Contest#Toledo_Nanochess) or that can work with [only a few bytes in memory](https://en.wikipedia.org/wiki/1K_ZX_Chess))... and I like the art of obfuscation! Thirdly, I am a big fan of incredible piece of art generated out of programs that fit in a tweet (examples: [e1](https://twitter.com/Hau_kun/status/1254709350813143040) [e2](https://twitter.com/yuruyurau/status/1254810004428750853) [e3](https://twitter.com/Hau_kun/status/1246082151017144320)); here the library [Processing](https://processing.org/) plays the role of [python-chess](http://python-chess.readthedocs.io/) and provides the facilities.

Minification or obfuscation is well-known in programming, but here it's a bit different: you cannot fully automate the size code reduction, you need to find some tricks to simplify your code and produce the essence of it. You can also play with the specificities of programming language (here Python) to write concise code (I've learned many things about Python by doing this exercise).

Here is the final solution:

```python
import chess as c,random as r
w=True
pk=[5,5,5,5,3]
while(w):
    s=[r.randrange(0,64) for i in range(0,5)]
    b=c.Board(fen=None)
    for q in range(0,5):b.set_piece_at(s[q],c.Piece(pk[q],w))
    if len(set()|(*[list(b.attacks(s[q]))for q in range(0,5)],s))>63:break
```

I can briefly comment:

* some shortcuts for import in such a way you can use only one letter then in the code
* `>63` takes less characters than `==64`
* `w=True` sounds weird (`True` is soooo fundamental, why using a variable?) but is only 6 characters... the reason is that `True` is used two times: in the loop and for setting the color (the library has not default color for piece, which is something I understand but not here ;)). So it will cost eight characters... Mine uses 6+2=8. Same, so it's okay ;-)
* for the generation of combinations, I'm using a random strategy since it's shorter in number of characters... and it might be the case that this strategy is faster to find *one* solution
* the hack I'm the most proud is the use of an array (see `pk`), each representing a kind of piece (5 is for queen, 3 is for bishop)... it allows to call `set_piece_at` one time, thus drastically saving some characters (btw you can find older versions of my attempt in a notebook or on Twitter). This kind of trick can hardly be find by a machine (I would be *very* surprised) and, above all, it's in fact a way to *generalize* my solution: I have created an abstraction that allows one to easily modify the kind of piece we want to place

## Generalization and beyond chess puzzles

I have generalized my solution to resolve other (who says *any*?) chess puzzles:

* placing 3 queens in a 6x6 board in such a way all squares are controlled (you have to hack a bit to manage a 6x6 board, since the library only manages 8x8 board)
* kings that respect social distancing ![Social distancing](../assets/kings-socialdistancing.png)
* or the famous n-queens problem.

Again with opportunistic reuse and in a way the program fits in a tweet ;)

There are so many ideas to explore: the use of constraint programming, genetic algorithms, etc. If you're interested, you can [explore online the notebook and programs](https://github.com/acherm/progpuzzles/blob/master/minimality.ipynb). I highly recommend [the notebook of Luc Lesoil](https://github.com/acherm/progpuzzles/blob/master/Chess%20-%204%20queens%201%20bishop.ipynb) who nicely resolves the first chess puzzle in a different and elegant way (eg with numpy and matplotlib to visualize the solution).

Maybe the easiest is to watch my talk [Towards programming puzzle spaces available on Youtube](https://youtu.be/oHEJJv7vUDw), given at the famous [DiverSE coffees](http://www.diverse-team.fr). In addition to what I've written in this blog post, I'm also developing and discussing the idea of *generating* interesting and enough hard puzzles (not necessary chess ones, see [examples](https://github.com/acherm/progpuzzles/blob/master/puzzlecoco.ipynb)). I'm basically arguing that whenever you see a puzzle, launch your notebook, resolve, tweetify, generalize, generate, and move to another puzzle space!

[^1]: A chess puzzle is sometimes used to refer to a real-world chess position in which you have to find the good series of moves (e.g., checkmate in 3). It's good for training your chess skills. Here "puzzle" refers to the general term that challenges the person's ingenuity and ability to solve some problems.


text:
---
title: Matomo OpenShift
description: Documentation and resources required to deploy a Matomo web analytics server instance into an OpenShift environment.  Matomo is a fully featured web analytics server and is a great alternative to Google Analytics when data ownership and privacy compliance are a concern.
author: esune
resourceType: Components
personas:
  - Developer
  - Product Owner
  - Designer
labels:
  - matomo
  - google
  - analytics
  - web
---
# Matomo OpenShift
Matomo is a fully featured web analytics server and is a great alternative to Google Analytics when data ownership and privacy compliance are a concern.

[Matomo OpenShift](https://github.com/BCDevOps/matomo-openshift) provides a set of OpenShift configurations to set up an instance of the Matomo web analytics server. See: [matomo.org](https://matomo.org/) for additional details regarding Matomo.

## Architecture
The service is composed by the following components:
- *matomo*: includes two containers within a single pod, the matomo analytics instance and matomo-proxy, which is the nginx used to proxy http requests.
- *matomo-db*: a [mariadb](https://mariadb.org) instance that will be used to store the analytics data.

## Deployment / Configuration
The templates provided in the `openshift` folder include everything that is necessary to create the required builds and deployments.

Since there are interdependencies between deployment configurations, please make sure to follow this order when creating them for the first time:
1) build and deploy the database
2) build and deploy the Matomo analytics server and proxy

Note: The TAG_NAME in the matomo-deploy is used for both the proxy and matomo which is annoying. That's why there is the rename from 3.11.0-fpm to latest

```
oc new-app -p NAME=mariadb -p OUTPUT_IMAGE_TAG=latest -f mariadb/mariadb-build.json
oc new-app -p NAME=matomo -p OUTPUT_IMAGE_TAG=latest -p SOURCE_IMAGE_TAG=3.11.0-fpm -f matomo/matomo-build.json

oc new-app -p NAME=matomo-proxy OUTPUT_IMAGE_TAG=latest -f matomo-proxy/matomo-proxy-build.json

oc new-app -p IMAGE_NAMESPACE=matomo -p TAG_NAME=latest -p NAME=matomo-db -p PERSISTENT_VOLUME_CLASS=glusterfs-storage -f matomo-db/matomo-db-deploy.json

oc new-app -p NAME=matomo -p IMAGE_NAMESPACE=matomo -p TAG_NAME=latest -p PERSISTENT_VOLUME_CLASS=glusterfs-storage -f matomo/matomo-deploy.json
```

## First Run
Once everything is up and running in OpenShift, follow the [instructions](https://matomo.org/docs/installation/#the-5-minute-matomo-installation) to create your superuser, set-up the connection to the database and initialize the Matomo dashboard.

To start tracking, copy the snippet for the appropriate website in the Matomo dashboard and place it in your website.


text:
# cbtree 0.9.3-4 (Nov 2013)
### Enhancements:
* The HTML tree node elements with class **_dijitTreeRow_** now have an additional attribute
**_expandable_** making it easy to distinguish between tree branches and tree leafs.
The **_expandable_** attribute value is either "true" or "false"

```html
<div class="dijitTreeRow" expandable="true" role="presentation" ... >
```

As a result you can now use css selectors like:

```html
<style type="text/css">
	.dijitTreeRow[expandable="true"] .dijitTreeLabel {
		background-color: yellow;
	}
	.dijitTreeRow[expandable="false"] .dijitTreeIcon {
		border-style: solid;
		border-width: 2px;
	}
</style>
```

### New Features:
None

### Bug Fixes:
None

# cbtree 0.9.3-3
### New Features:
* A new property, **_branchCheckBox_**, has been added to the tree. The **_branchCheckBox_**
property controls if checkboxes will be displayed for tree branches. The default is `true`.
if set to `false` the branch checkboxes will be hidden but still available for checking their
state.

* Tree Nodes can now be sorted using the new **_options_** property of the store models:
```javascript
var mySortOptions = {sort: [
    {attribute: "name", descending: true, ignoreCase: true},
    {attribute: "hair", ignoreCase: true},
    {attribute: "age"}
]};
var model = new TreeStoreModel( { store: store,
                                  query: {name: "Root"},
                                  options: {sort: mySortOptions}
                                });
```
For a detailed description of this new feature and the ABNF definition of the **_options_**
property please refer to the
[cbtree Wiki](https://github.com/pjekel/cbtree/wiki/CheckBox-Tree-Usage#sorting-tree-nodes)

### Bug Fixes:
* When the **_checkAttr_** property on the store model is set to anything other than the
default 'checked', both the expandChecked() and collapseUnChecked() method did not work
as documented. Issue #36: [expandChecked()](https://github.com/pjekel/cbtree/issues/36)

# cbtree 0.9.3-2

### Enhancements:
* The name of tree checkboxes will now default to the dijit generated checkbox id. This
will guarantee that any tree checkbox visible in a form will be included in a form submission.
See also the new properties **_openOnChecked_** and **_closeOnUnchecked_**.
* The TreeStyling extension now allows the wildcard character (*) to be used in icon object
properties when mapping values to icons.

### New Features:
* Two new properties have been added to the tree:
	1. openOnChecked
	2. closeOnUnchecked
* Two new functions have been added to the tree:
	1. expandChecked()
	2. collapseUnchecked()

Please refer to the wiki [documentation](https://github.com/pjekel/cbtree/wiki/CheckBox-Tree-API)
for a detailed description of the new features.

### Bug Fixes:
* When using an Observable store the tree/model/_base/BaseStoreModel class did not remove
obsolete Observers. Issue #28: [Observable vs Eventable store](https://github.com/pjekel/cbtree/issues/28)
* Icon styling provided with a mapped icon object was not applied. Issue #30:
[Mapping a property value to a specific icon object](https://github.com/pjekel/cbtree/issues/30)
* Root children always appeared as the last item when dragged and dropped. Issue: #31:
[DnD of root children](https://github.com/pjekel/cbtree/issues/31)

### Known issues:
Please see the Github project page for any [open](https://github.com/pjekel/cbtree/issues?page=1&state=open)
issues.

repository: bridgji/cbtree

repo_name: bridgji/cbtree


text:
---
layout: post
title: "DebCamp18: Reporting my experience and technical work of my first DebCamp"
#feature-img: "assets/img/pexels/taipei-taiwan.jpeg"
thumbnail: "assets/img/thumbnails/taiwan-wall.png"
lang-ref: debonf-2018-2
locale: en
categories:
  - Hacking
  - FLOSS
tags:
  - gsoc
  - debconf
  - trip
  - debian
  - floss
  - distro-tracker
excerpt_separator:
---

![DebConf18]({{ "/assets/img/others/debconf18.png" }}){: .center }

Last week I was attending my first DebCamp which was an amazing and enjoyable
experience. In a nutshell,
I was in the right environment to leverage my
[GSoC project]({% post_url en/2018-05-09-google-summer-of-code-2018 %})
since I was able to develop new features for
[Distro Tracker](https://tracker.debian.org) while interacting with other
experienced Debian contributors who provided me highly valuable feedback and
ideas. However, in this post, I will focus on reporting my main activities during
DebCamp and presenting my early impressions.

Thanks to Debian project sponsorship, I was able to attend DebConf18
in Hsinchu, Taiwan. DebCamp occurs the week before DebConf which aims at
gathering together Debian contributors and teams to focus on their tasks.
Despite its technical nature, DebCamp seemed to be more about the interaction
between people since it enables people who work remotely around the world
to work together
in person. In this sense, they not only have the chance to quickly resolve open
bugs and to keep doing their great packaging/translating/reviewing/(.\*?)
work, but also to discover
other projects with which community members are involved, to meet
new people, to encourage, discuss, and bootstrap new ideas,
to teach and learn about [FLOSS](https://www.gnu.org/philosophy/floss-and-foss.html),
and to have fun with their friends by doing what they like and believe.

In fact, people are the most important part of free software projects and I
was really happy to see that the Debian community shares this thought. For
this reason, DebCamp was very fruitful for me. Most people are very
receptive and open to conversation. They do care about your experience during
the event and are happy to meet new people as well.
Meal times are great for sitting down with different people and talking about
the most varied things, such as curiosities about their home countries.
A positive surprise is the number of people who are involved with
non-technical aspects of the project, such as translation, teaching,
bug triage, events organization, and user support, although there is a need
for many more people getting involved in these areas.
It is awesome to have the opportunity of talking with people
from different countries and with such varied background.

I was able to talk to many
experienced Debian contributors who spoke warmly about the Debian history,
their activities and experience, and who are also interested in what I am
doing. Some of them had already seen my work on Distro Tracker before
and they provided several good ideas for improvements and new features. I am
taking note of all of these ideas and I plan to compile them into another blog
post as a roadmap for my future post-GSoC contributions.

Another key outcome from those conversations is that I have realized several
opportunities within the project where I can contribute as a Software Engineer.
In particular, due to my early working on Distro Tracker, I'm quite interested
in enabling an event-based, pub/sub mechanism for Debian services. I had some
experience with [RabbitMQ](https://www.rabbitmq.com/) during
[my masters](/assets/documents/masters_dissertation_arthur_del_esposte.pdf) and
I strongly believe using it on the Debian infrastructure could bring many
benefits. Although I've started some conversations about that, I still need
to understand some aspects of the project better and discuss with more
interested people as well.

Like me, many other people are
new to the community and I was glad to see the willingness to contribute
in most of them, such as some GSoC students who intend to continue their
projects, active members who got involved with Debian through previous
Outreachy and GSoC programs, and locals who are taking the opportunity to
get started in the project. Along with them, I intend to actively contribute
to Debian project in the days to come =).

## Technical Report

Here is what I did at DebCamp:

* [Distro-Tracker#43](https://salsa.debian.org/qa/distro-tracker/merge_requests/43):
Add extensions to Bug table field to highlight RC bugs and to add BTS links
for each bug category. My code is already merged and running on production
environment.
* [Distro-Tracker#46](https://salsa.debian.org/qa/distro-tracker/merge_requests/46):
Fixed patch bug action item BTS urls
* [Distro-Tracker#45](https://salsa.debian.org/qa/distro-tracker/merge_requests/45):
Enabled the creation of multiple package tables to be displayed in teams' page.
Each table aims at listing packages related to some specific scope, such as:
  * Packages with RC bugs
  * Packages with newer upstream version
Raphaël Hertzog has already reviewed this Merge Request and suggested some
improvements.
* [Distro-Tracker#44](https://salsa.debian.org/qa/distro-tracker/merge_requests/44):
Improving table rendering performance based on a caching strategy using Redis.
However, we will need to change it since we will not be able to use Redis for
now (see Raphaël's comments on the Merge Request).
* [DebConf19's Website](https://salsa.debian.org/debconf-team/public/websites/dc19/tree/homepage)
Helped Lucas Kanashiro and Valessio Brito to create the first version of
DebConf19's website

Right now, DebConf is happening and so I did not have much time to work on
other technical things this week.

Once again, thanks to Debian for enabling my participation in both DebCamp
and DebConf. I'm really enjoying it.

**Let's get moving on! ;)**

repository: arthurmde/my-website-jekyll

repo_name: arthurmde/my-website-jekyll


text:
I had it during dim sum, but it was also a really quick and easy meal my mom made for us just by throwing some marinated rice on top of the rice already cooking in the rice cooker.  The key is cooking all of the ingredients together with the rice.  All the black bean garlic sauce infuses into the rice at the end and it's just the perfect marriage of savory goodness.

The star of the show are the Chinese fermented black beans, which contain a mix of sweet spices, ginger and even orange peel.  It's really what gives this dish its unique flavor.  For those looking to take a quicker route, there is always a pre-prepared [black bean garlic sauce](http://amzn.to/2tNRLPa), which most people see at the grocery stores.

__Vegan Pork Sparerib Black Bean Rice__

(Serves 2)

- 1 cup white rice, soaked in 1 1/2 cups water
- 1/2 cup vegan pork/beef cubes (can substitute tofu or jackfruit)
- 2 tablespoons [fermented black beans](http://amzn.to/2vNujiy)
- 1 tablespoon garlic, minced
- 1 tablespoon vegan oyster sauce
- 1 teaspoon sugar
- 1/2 teaspoon cornstarch
- 1 tablespoon rice wine
- 1 teaspoon sesame oil
- chili peppers (optional)
- scallions (garnish)

### Directions

1. Start by cooking rice in a claypot or pan.
2. While rice is cooking, combine the vegan pork cubes, black beans, garlic, vegan oyster sauce, sugar, cornstarch, rice wine and sesame oil.
3. Once the rice has absorbed nearly all the liquid, add in the vegan pork mixture.  Cook for an additional 5-6 minutes, until the rice is finished cooking.
4. Garnish with chilies and scallions.

repository: Ibexian/gatsby-advanced-starter

repo_name: Ibexian/gatsby-advanced-starter


text:
---
title: "Americium"
date: 2019-08-28T17:42:15+05:30
draft: false
LatinName: "Americium"
atomicnumber: "95"
classification: "Actinoid"
NumberOfElectrons: "95"
NumberOfProtons: "95"
NumberOfNeutrons: "150"
DiscoveredBy: "Glenn Theodore Seaborg"
DiscoveredIn: 1940
AtomicWeight: "244.0642"
stateAtRoomTemp: Solid
imgurl: "americium.jpeg"
Link: "https://en.wikipedia.org/wiki/Americium"
about: "Americium is a synthetic radioactive chemical element with the symbol Am and atomic number 95. It is a transuranic member of the actinide series, in the periodic table located under the lanthanide element europium, and thus by analogy was named after the Americas."

---

repository: ChivukulaVirinchi/periodic-table

gha_language: HTML

repo_name: ChivukulaVirinchi/periodic-table


text:
# Tutorial title
- Original tutorial by: John Doe
- Your lab name: Moon Lab
- License: CC-SA-BY
- Last Review Date: 01/12/2016
- Last Modified by: Agent Smith

## Summary
Short description of the tutorial itself and a representative image of the tutorial.

## Step 1. Description of the step
Text explaining what to do.

## Step 2. Description of the step
Text explaining what to do.

## Problems faced and solutions

- First Problem description

  > Solution found

- Second Problem description

  > Solution found

## Machines and software used and settings

  * Machine used
  * Spindle Speed
  * Feed Rate
  * Power

## Links for other sources / useful information
  - [Link 1]()
  - [Link 2]()

## Original Design files
* [CAD file]()
* [Electronics Schematic]()
* [Electronics board]()
* [Code]()

repository: antoniogrillo/FabAcademy-Tutorials

repo_name: antoniogrillo/FabAcademy-Tutorials


text:
# Lantern

An isomorphic web application built with modern technologies, which contains basic functionality, fundamental design and workable architecture. Good to be used to create your project quickly.

The architecture design was already done for isomorphic. The views and frontend stuffs which are implemented with React can be rendered on server-side and client-side both and using the same configuration of routes. The good news is, there is no need to learn anti-pattern flux framework(e.g., redux) for isomorphic. Lantern provided an easy way for developers who are already familar with flux pattern.

 The generator is widely used in this project, which is the new technology in new version of ECMAScript(JavaScript) for flow control. The Backend is using generator with Koa web framework, and frontend is using generator in flux framework as well.

Not only technique things, but also UI design is pretty cool in Lantern project. The framework that Semantic UI bring us fantastic frontend user interface, even better than bootstrap and other UI framework.

## Documentation

See our [Wiki](https://github.com/cfsghost/lantern/wiki) for more information.

## Installation

In order to support ES6+ and using a lots of new technologies (ex, generator and classes), Node.js 0.12+ or io.js is required. Once you have such environment, you can start to install modules needed via NPM.

```
npm install .
```

Then you can use webpack to compile and combine all of frontend source code for service.
```
webpack
```

Finally, you can start this web service:
```
node app.js
```

## Development

Starting up in development mode is able to enable hot loading mechanism, you can start service with:
```
node app.js dev
```

## Features

* Fast to setup and easy to customize
* UI is pretty cool and fantasic
* Widely use ES6 and ES7+ (Generator, classes and decorator)
* Isomorphic Architecture
* Provided a lot useful extensions to speed up the development.
* Support permission management
* Support user database system
* Support Hot loading without webpack-dev-server
* Support i18n for multiple language
* Support third-party Authorization (Facebook/Github/Google/Linkedin)
* Support Hot-load mechanism

## Dependencies

* Node.js 0.12+ or io.js
* Koa web framework
* MongoDB
* Fluky - Event-based framework for flux data flow pattern
* babel 6 - Used to support ES6/ES7+
* React v0.14+
* react-router 1.0.0+ - router for React
* [Semantic UI](http://semantic-ui.com/) - Front-end UI toolkit
* Webpack
* Passport
*

## Showcases
* [Hackathon Taiwan](http://hackathon.tw/)

License
-
Licensed under the MIT License

Authors
-
Copyright(c) 2015 Fred Chien<EMAIL_ADDRESS>

repository: HackathonTaiwan/Website

repo_name: HackathonTaiwan/Website


text:
# NAME

udp_open - opens a UDP socket

# SYNOPSIS

**#include <libdill.h>**

**int udp_open(struct ipaddr **\*_local_**, const struct ipaddr **\*_remote_**);**

# DESCRIPTION

UDP is an unreliable message-based protocol. The size of the message is limited. The protocol has no initial or terminal handshake. A single socket can be used to different destinations.

This function creates a UDP socket.

IP address passed in _local_ argument will be used to set source IP address in outgoing packets. Also, the socket can be used to receive packets sent to this address. If port in the address is set to zero an ephemeral port will be chosen and filled into the _local_ address.

IP address passed in _remote_ is the default destination for outbound packets. It is used by **msend()** and **mrecv()** functions which don't allow for specifying the destination address explicitly. It is also used by **udp_send()** and **udp_sendl()** functions if the address parameter of those functions is set to NULL.

# RETURN VALUE

Newly created socket handle. On error, it returns -1 and sets _errno_ to one of the values below.

# ERRORS

* **EADDRINUSE**: The local address is already in use.
* **EADDRNOTAVAIL**: The specified address is not available from the local machine.

* **EINVAL**: Invalid arguments.
* **EMFILE**: The maximum number of file descriptors in the process are already open.
* **ENFILE**: The maximum number of file descriptors in the system are already open.
* **ENOMEM**: Not enough memory.

# EXAMPLE

```
struct ipaddr addr;
int rc = ipaddr_local(&addr, NULL, 5555, 0);
int s = udp_open(&addr, NULL);
```

repository: reqshark/libdill

repo_name: reqshark/libdill


text:
# How-to add a Grid to Search and Identify

## Add show grid to the service configuration

```
    app.registerService('search', SearchService, {
        showGrid: true,
        ...
    });
```

## Add template links in the mapbook.

* For search the templates are named `search-grid-columns` and `search-row`.
* For identify the templates are named `identify-grid-columns` and `identify-row`.

The search example is shown below:

```
    <template name="search-grid-columns" src="./templates/parcel-columns.json" />
    <template name="search-grid-row" src="./templates/parcel-row.html" />
```


text:
# [Employee Directory](https://yaldaemployeechart.herokuapp.com/)

## An application built with React , to display employee information

>This application to view the entire employee directory at once to have quick access to their information.

## Table of contents
* [General info](#general-info)
* [Screenshots](#screenshots)
* [Installation](#installation)
* [Usage](#usage)
* [Credits](#credits)
* [Features](#features)
* [Status](#status)
* [License](#license)
* [Contact](#contact)

## General Info
Employee Directory application is currently doing an api call to randomuser api to generate random list of employees and once you type in the search form it will sort the data.

## Screenshots

## Installation

To install :
* [Visual Code](https://code.visualstudio.com/docs/setup/setup-overview)
* [NodeJS](https://nodejs.org/en/download/)

To use:

In order to install, first download the files. Then you would open terminal and type ```npm install``` . This will install all the npm packages listed under the dependencies in the package.json. Then you will type the command ``` npm start```. This will start the react app at the port 300 in the browser. Finally you will type ```localhost:3000``` in the browser window and use the application.

## Usage

![sample use](./public/assets/images/gif1.gif)

To use this application, you can [click here](https://yaldaemployeechart.herokuapp.com/)

## Credits

This project was bootstrapped with [Create React App](https://github.com/facebook/create-react-app).

Initial readme instructions provided by [Triologyed](https://www.trilogyed.com/)

Built with [Bootstrap](https://getbootstrap.com/)

## Features

* Feature 1: Easy to navigate design.
* Feature 2: Ability to search employee
* deployed with heroku.

To-do list:

*  UI desing can be improved.

## Status
Project is:  _finished_

## License

MIT license
Copyright © 2020 Zahra Ali Aghazadeh

## Contact
Email<EMAIL_ADDRESS>
LinkedIn: [Zahra Ali Aghazadeh](https://www.linkedin.com/in/zahraaliaghazadeh)

feel free to contact me!

repository: zahraaliaghazadeh/employeedirectory

repo_name: zahraaliaghazadeh/employeedirectory


text:
## Fake server provided by: [https://my-json-server.typicode.com/](https://my-json-server.typicode.com/)

## My version of the fake server: [https://my-json-server.typicode.com/ShadowDK89/fake-server](https://my-json-server.typicode.com/ShadowDK89/fake-server)

## Use your own data

Here you can use the template to make your own fake server: [https://github.com/typicode/demo](https://github.com/typicode/demo)


text:
# Hibernate 5 with Wildfly

This is a hello-world of Java EE 7 + Hibernate 5 with the [Wildfly](http://wildfly.org/) application server.
It models a simple datastore of Employees and Departments.  You can use the restful interface to
add some entities and query to see how records are related.  We use an in-memory H2 database and
hbm2ddl create-drop to create the schema afresh when the webapp is deployed.

### How to build and run this project

Start the Wildfly appserver: `./bin/standalone.sh`

Build and deploy: `mvn install wildfly:deploy`

Run a script to populate the tables:
```
./populate_tables.sh
```
Take a look inside that script to see how it posts new records.

Now run some queries.

Get the list of departments:
```
curl localhost:8080/wh1/departments
```

Get the list of employees:
```
curl localhost:8080/wh1/employee
```

Or the employees with a particular last name:
```
curl localhost:8080/wh1/employee?lastName=Young
```
You can use any REST client you like, not just curl.

### Project structure

An inbound REST request passes through these layers, from top to bottom:

* com.example.rest
    * JAX-RS RESTful request interface.
* com.example.service
    * JTA transaction boundaries.
* com.example.dao
    * Data Access Object: JPA persistence context.
* com.example.model
    * JPA Entity.
    * JAX-RS Response body.

In a larger project it would make sense to break out a separate DTO package for JAX-RS Response objects.

### Java EE remarks

Wildfly implements CDI with Weld, JAX-RS with Resteasy, and data binding with Jackson.  Our chosen JPA impl is
Hibernate.

We're using CDI (beans.xml and bean-related annotations) so the Wildfly container will manage the lifecycle
of contextual bean instances.  Since the JPA EntityManager is container-managed, the app doesn't instantiate
the EMF. We use @PersistenceContext in a managed bean (@ApplicationScoped) and the container injects
an EM for us.  The container finds our Entity classes.

The model classes are serving double-duty as JPA Entity and Jackson/JAX-RS Response objects.

### About the database

The datasource is an in-memory H2 database.  When the webapp is deployed, Wildfly parses the persistence
unit.  We specify `hibernate.hbm2ddl.auto=create-drop` and the the schema drop/creation happens at this
point.

Our code defines the webapp datasource using *-ds.xml, which is convenient but for test purposes only.
For production you're supposed to define the datasource in Wildfly standalone.xml (where it can
be managed live in the Wildfly CLI or admin console), not in this deployable war.

For convenience, we also start up an H2 TCP Server when the first restful request is received, so that you
can separately start an [H2 Web Console](http://www.h2database.com/html/quickstart.html#h2_console) and use
the browser to inspect the H2 database.  With Wildfly 10.1.0.Final, I do this:

```
cd <wildfly-install-home>
java -cp modules/system/layers/base/com/h2database/h2/main/h2-1.3.173.jar \
              org.h2.tools.Server -web
```
The default H2 web port is 8082, tcp is 9092.  Open a browser to localhost:8082, and connect using the parameters
specified in the *-ds.xml file, with just one change: in the browser, when you specify JDBC URL, change `mem` to
`tcp://localhost:9092/mem`.

### Technology summary

* Java EE 7
* JDK 8
* JPA 2.1
* CDI 1.2
* JTA 1.2
* Hibernate 5.0.10.Final
* Jackson 2.7.4
* H2 database 1.3
* Tested on Wildfly 10.1.0.Final

gha_language: Java


text:
# Use sparkMeasure with Scala shell or notebooks

Notes on how to use sparkMeasure to collect Spark workload metrics when Scala shell or a Scala notebook.
See also [README](../README.md) for an introduction to sparkMeasure and its architecture.

### Run sparkMeasure using the packaged version from Maven Central

- The alternative, see paragraph above, is to build a jar from master.
    ```
    bin/spark-shell --packages ch.cern.sparkmeasure:spark-measure_2.11:0.13

    // or just download and use the jar (it is only needed in the driver) as in:
    bin/spark-shell --conf spark.driver.extraClassPath=<path>/spark-measure_2.11-0.13.jar
   ```

### Download and build sparkMeasure (optional)

 - If you prefer to build from the latest development version:
    ```
    git clone https://github.com/lucacanali/sparkmeasure
    cd sparkmeasure
    sbt package
    ls -l target/scala-2.11/spark-measure*.jar  # location of the compiled jar

    # Run as in one of these examples:
    bin/spark-shell --jars <path>/spark-measure_2.11-0.14-SNAPSHOT.jar

    # alternative, set classpath for the driver (it is only needed in the driver)
    bin/spark-shell --conf spark.driver.extraClassPath=<path>/spark-measure_2.11-0.14-SNAPSHOT.jar
    ```

### Example: collect and print stage metrics with sparkMeasure

1. Measure metrics at the Stage level, a basic exaple:
    ```
    bin/spark-shell --packages ch.cern.sparkmeasure:spark-measure_2.11:0.13

    val stageMetrics = ch.cern.sparkmeasure.StageMetrics(spark)
    stageMetrics.runAndMeasure(spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show)
    ```

    Example output:
    ```
    Scheduling mode = FIFO
    Spark Context default degree of parallelism = 8
    Aggregated Spark stage metrics:
    numStages => 3
    sum(numTasks) => 17
    elapsedTime => 9103 (9 s)
    sum(stageDuration) => 9027 (9 s)
    sum(executorRunTime) => 69238 (1.2 min)
    sum(executorCpuTime) => 68004 (1.1 min)
    sum(executorDeserializeTime) => 1031 (1 s)
    sum(executorDeserializeCpuTime) => 151 (0.2 s)
    sum(resultSerializationTime) => 5 (5 ms)
    sum(jvmGCTime) => 64 (64 ms)
    sum(shuffleFetchWaitTime) => 0 (0 ms)
    sum(shuffleWriteTime) => 26 (26 ms)
    max(resultSize) => 17934 (17.0 KB)
    sum(numUpdatedBlockStatuses) => 0
    sum(diskBytesSpilled) => 0 (0 Bytes)
    sum(memoryBytesSpilled) => 0 (0 Bytes)
    max(peakExecutionMemory) => 0
    sum(recordsRead) => 2000
    sum(bytesRead) => 0 (0 Bytes)
    sum(recordsWritten) => 0
    sum(bytesWritten) => 0 (0 Bytes)
    sum(shuffleTotalBytesRead) => 472 (472 Bytes)
    sum(shuffleTotalBlocksFetched) => 8
    sum(shuffleLocalBlocksFetched) => 8
    sum(shuffleRemoteBlocksFetched) => 0
    sum(shuffleBytesWritten) => 472 (472 Bytes)
    sum(shuffleRecordsWritten) => 8
    ```

2. An alternative way to collect and print metrics:
    ```scala
    val stageMetrics = ch.cern.sparkmeasure.StageMetrics(spark)
    stageMetrics.begin()

    ...execute one or more Spark jobs...

    stageMetrics.end()
    stageMetrics.printReport()
    ```

3. Print additional accumulables metrics (including SQL metrics) collected at stage-level, Scala:
    ```scala
    stageMetrics.printAccumulables()
    ```

### Collecting metrics at finer granularity: use Task metrics

Collecting Spark task metrics at the granularity of each task completion has additional overhead
compare to collecting at the stage completion level, therefore this option should only be used if you need data with this finer granularity, for example because you want
to study skew effects, otherwise consider using stagemetrics aggregation as preferred choice.

- The API for collecting data at task level is similar to the stage metrics case.
  An example:
    ```scala
    val taskMetrics = ch.cern.sparkmeasure.TaskMetrics(spark)
    taskMetrics.runAndMeasure(spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show)
    ```

### Exporting metrics data for archiving and/or further analysis

One simple use case is to make use of the data collected and reported by stagemetrics and taskmetrics
printReport methods for immediate troubleshooting and workload analysis.
You also have options to save metrics aggregated as in the printReport output.
Another option is to export the metrics to an external system, such as [Prometheus Pushgateway](prometheus.md)

- Example on how to export raw Stage Metrics metrics data in json format
    ```scala
    val stageMetrics = ch.cern.sparkmeasure.StageMetrics(spark)
    stageMetrics.runAndMeasure( ...your workload here ... )

    val df = stageMetrics.createStageMetricsDF("PerfStageMetrics")
    df.show()
    stageMetrics.saveData(df.orderBy("jobId", "stageId"), "/tmp/stagemetrics_test1")

    val accumulablesDF = stagemetrics.createAccumulablesDF("AccumulablesStageMetrics")
    stageMetrics.saveData(accumulablesDF, "/tmp/stagemetrics_accumulables_test1")
    ```

- Example, save aggregated metrics (as found in the printReport output) in json format

    ```scala
    val df = stageMetrics.createStageMetricsDF("PerfStageMetrics")
    val aggregatedDF = stageMetrics.aggregateStageMetrics("PerfStageMetrics")
    aggregatedDF.show()
    stageMetrics.saveData(aggregatedDF, "/tmp/stagemetrics_report_test2")
    ```

- Example on how to export raw Task Metrics data in json format
    ```scala
    val df = taskMetrics.createTaskMetricsDF("PerfTaskMetrics")
    spark.sql("select * from PerfTaskMetrics").show()
    df.show()
    taskMetrics.saveData(df.orderBy("jobId", "stageId", "index"), "<path>/taskmetrics_test3")
    ```


text:
---
id: remote-server-login
title: Remote Server Login (Secure Shell)
layout: page
---

## ssh
_login to a remote server_

```bash
$ ssh [username]@[hostname or ip address]
```

repository: farleykr/helpful-shell

repo_name: farleykr/helpful-shell