text:
---
title: "Map-Reduce With Ruby Using Hadoop"
date: "2010-12-31T09:38:00-08:00"
template: "post"
draft: false
slug: "map-reduce-with-ruby-using-hadoop"
category: "Data processing"
tags:
  - amazon ec2
  - bash
  - cloudera
  - data processing
  - hadoop
  - hadoop cluster
  - hadoop streaming
  - hdfs
  - jclouds
  - map-reduce
  - ruby
  - whirr
description: "Here I demonstrate, with repeatable steps, how to fire-up a Hadoop cluster on Amazon EC2, load data onto the HDFS (Hadoop Distributed File-System), write map-reduce scripts in Ruby and use them to run a map-reduce job on your Hadoop cluster. You will not need to ssh into the cluster, as all tasks are run from your local machine. Below I am using my MacBook Pro as my local machine, but the steps I have provided should be reproducible on other platforms running bash and Java."
socialImage:
  publicURL: "/media/images/2010/12/hadoop_ruby_sq.jpg"
---

Here I demonstrate, with repeatable steps, how to fire-up a Hadoop cluster on Amazon EC2, load data onto the HDFS (Hadoop Distributed File-System), write map-reduce scripts in Ruby and use them to run a map-reduce job on your Hadoop cluster. You will _not_ need to ssh into the cluster, as all tasks are run from your local machine. Below I am using my MacBook Pro as my local machine, but the steps I have provided should be reproducible on other platforms running bash and Java.

*   [Fire-Up Your Hadoop Cluster](#fire-up-your-hadoop-cluster)
*   [Setting Up Your Local Hadoop Client](#setting-up-your-local-hadoop-client)
*   [Defining The Map-Reduce Task](#defining-the-map-reduce-task)
*   [Uploading Your Data To HDFS (Hadoop Distributed FileSystem)](#uploading-your-data-to-hdfs)
*   [Coding Your Map And Reduce Scripts in Ruby](#coding-your-map-and-reduce-scripts-in-ruby)
*   [Running The Hadoop Job](#running-the-hadoop-job)
*   [The Results](#the-results)
*   [Conclusion](#conclusion)
*   [Resources](#resources)

## Fire-Up Your Hadoop Cluster

I choose the [Cloudera distribution of Hadoop](https://www.cloudera.com/hadoop/) which is still 100% Apache licensed, but has some additional benefits. One of these benefits is that it is released by [Doug Cutting](https://en.wikipedia.org/wiki/Doug_Cutting), who started Hadoop and drove it’s development at Yahoo! He also started [Lucene](https://lucene.apache.org/), which is another of my favourite Apache Projects, so I have good faith that he knows what he is doing. Another benefit, as you will see, is that it is simple to fire-up a Hadoop cluster.

 Whirr is built on [jclouds](https://code.google.com/p/jclouds/), meaning other cloud providers should be supported, but only Amazon EC2 has been tested. Once we have Whirr installed, we will configure a _hadoop.properties_ file with our Amazon EC2 credentials and the details of our desired Hadoop cluster. Whirr will use this _hadoop.properties_ file to build the cluster.

If you are on Debian or Redhat you can use either apt-get or yum to install whirr, but since I’m on Mac OS X, I’ll need to [download the Whirr script](https://www.apache.org/dyn/closer.cgi/incubator/whirr/).

The current version of Whirr 0.2.0, hosted on the Apache Incubator site, is not compatible with Cloudera’s Distribution for Hadoop (CDH), so I’m am downloading [version 0.1.0+23](https://archive.cloudera.com/cdh/3/whirr-0.1.0+23.tar.gz).

```bash
mkdir ~/src/cloudera
cd ~/src/cloudera
wget https://archive.cloudera.com/cdh/3/whirr-0.1.0+23.tar.gz
tar -xvzf whirr-0.1.0+23.tar.gz
```

To build Whirr you’ll need to install Java (version 1.6), [Maven](https://maven.apache.org/download.html) ( >= 2.2.1) and [Ruby](https://www.ruby-lang.org/en/) ( >= 1.8.7). If you’re running with the latest Mac OS X, then you should have the latest Java and I’ll assume, due to the title of this post, that you can manage the Ruby version. If you are not familiar with Maven, you can [install it via Homebrew on Mac OS X](/homebrew-intro-to-the-mac-os-x-package-installer) using the brew command below. On Debian use _apt-get install maven2_.

```bash
sudo brew update
sudo brew install maven
```

Once the dependencies are installed we can build the whirr tool.

```bash
cd whirr-0.1.0+23
mvn clean install
mvn package -Ppackage
```

In true Maven style, it will download a long list of dependencies the first time you build this. Be patient.

Ok, it should be built now and if you’re anything like me, you would have used the time to get a nice cuppa tea or a sandwich. Let’s sanity check the whirr script…

```bash
bin/whirr version
```

You should see something like “Apache Whirr 0.1.0+23″ output to the terminal.

Create a _hadoop.properties_ file with the following content.

```
whirr.service-name=hadoop
whirr.cluster-name=myhadoopcluster
whirr.instance-templates=1 jt+nn,1 dn+tt
whirr.provider=ec2
whirr.identity=<cloud-provider-identity>
whirr.credential=<cloud-provider-credential>
whirr.private-key-file=${sys:user.home}/.ssh/id_rsa
whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub
whirr.hadoop-install-runurl=cloudera/cdh/install
whirr.hadoop-configure-runurl=cloudera/cdh/post-configure
```

Replace _<cloud-provider-identity>_ and _<cloud-provider-credential>_ with your Amazon EC2 Access Key ID and Amazon EC2 Secret Access Key (I will not tell you what mine is).

This configuration is a little boring with only two machines. One machine for the master and one machine for the worker. You can get more creative once you are up and running. Let’s fire up our “cluster”.

```bash
bin/whirr launch-cluster --config hadoop.properties
```

This is another good time to put the kettle on, as it takes a few minutes to get up and running. If you are curious, or worried that things have come to a halt then Whirr outputs a whirr.log in the current directory. Fire-up another terminal window and tail the log.

```bash
cd ~/src/cloudera/whirr-0.1.0+23
tail -F whirr.log
```

16 minutes (and several cups of tea) later the cluster is up and running. Here is the output I saw in my terminal.

```bash
Launching myhadoopcluster cluster
Configuring template
Starting master node
Master node started: [[id=us-east-1/i-561d073b, providerId=i-561d073b, tag=myhadoopcluster, name=null, location=[id=us-east-1d, scope=ZONE, description=us-east-1d, parent=us-east-1], uri=null, imageId=us-east-1/ami-d59d6bbc, os=[name=null, family=amzn-linux, version=2010.11.1-beta, arch=paravirtual, is64Bit=false, description=amzn-ami-us-east-1/amzn-ami-2010.11.1-beta.i386.manifest.xml], userMetadata={}, state=RUNNING, privateAddresses=[<IP_ADDRESS>], publicAddresses=[<IP_ADDRESS>], hardware=[id=m1.small, providerId=m1.small, name=m1.small, processors=[[cores=1.0, speed=1.0]], ram=1740, volumes=[[id=null, type=LOCAL, size=10.0, device=/dev/sda1, durable=false, isBootDevice=true], [id=null, type=LOCAL, size=150.0, device=/dev/sda2, durable=false, isBootDevice=false]], supportsImage=Not(is64Bit())]]]
Authorizing firewall
Starting 1 worker node(s)
Worker nodes started: [[id=us-east-1/i-98100af5, providerId=i-98100af5, tag=myhadoopcluster, name=null, location=[id=us-east-1d, scope=ZONE, description=us-east-1d, parent=us-east-1], uri=null, imageId=us-east-1/ami-d59d6bbc, os=[name=null, family=amzn-linux, version=2010.11.1-beta, arch=paravirtual, is64Bit=false, description=amzn-ami-us-east-1/amzn-ami-2010.11.1-beta.i386.manifest.xml], userMetadata={}, state=RUNNING, privateAddresses=[<IP_ADDRESS>], publicAddresses=[<IP_ADDRESS>], hardware=[id=m1.small, providerId=m1.small, name=m1.small, processors=[[cores=1.0, speed=1.0]], ram=1740, volumes=[[id=null, type=LOCAL, size=10.0, device=/dev/sda1, durable=false, isBootDevice=true], [id=null, type=LOCAL, size=150.0, device=/dev/sda2, durable=false, isBootDevice=false]], supportsImage=Not(is64Bit())]]]
Completed launch of myhadoopcluster
Web UI available at https://ec2-72-44-45-199.compute-1.amazonaws.com
Wrote Hadoop site file /Users/phil/.whirr/myhadoopcluster/hadoop-site.xml
Wrote Hadoop proxy script /Users/phil/.whirr/myhadoopcluster/hadoop-proxy.sh
Started cluster of 2 instances
HadoopCluster{instances=[Instance{roles=[jt, nn], publicAddress=ec2-72-44-45-199.compute-1.amazonaws.com/<IP_ADDRESS>, privateAddress=/<IP_ADDRESS>}, Instance{roles=[tt, dn], publicAddress=/<IP_ADDRESS>, privateAddress=/<IP_ADDRESS>}], configuration={fs.default.name=hdfs://ec2-72-44-45-199.compute-1.amazonaws.com:8020/, mapred.job.tracker=ec2-72-44-45-199.compute-1.amazonaws.com:8021, hadoop.job.ugi=root,root, hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.SocksSocketFactory, hadoop.socks.server=localhost:6666}}
```

Whirr has created a directory with some files in our home directory…

```bash
~/.whirr/myhadoopcluster/hadoop-proxy.sh
~/.whirr/myhadoopcluster/hadoop-site.xml
```

This hadoop-proxy.sh is used to access the web interface of Hadoop securely. When we run this it will tunnel through to the cluster and give us access in the web browser via a SOCKS proxy.

You need to configure the SOCKS proxy in either your web browser or, in my case, the Mac OS X settings menu.

_Hadoop SOCKS Proxy Configuration for Mac OS X_

Now start the proxy in your terminal…
_(Note: There has still been no need to ssh into the cluster. Everything in this post is done on our local machine)_

```bash
sh ~/.whirr/myhadoopcluster/hadoop-proxy.sh

   Running proxy to Hadoop cluster at
   ec2-72-44-45-199.compute-1.amazonaws.com.
   Use Ctrl-c to quit.

```

The above will output the hostname that you can access the cluster at. On Amazon EC2 it looks something like _https://ec2-72-44-45-199.compute-1.amazonaws.com:50070/dfshealth.jsp_. Use this hostname to view the cluster in your web browser.

```
https://<hostname>:50070/dfshealth.jsp
```

_HDFS Health Dashboard_

If you click on the link to “Browse the filesystem” then you will notice the hostname changes. This will jump around the data-nodes in your cluster, due to HDFS’s distributed nature. You only currently have one data-node. On Amazon EC2 this new hostname will be the internal hostname of data-node server, which is visible because you are tunnelling through the SOCKS proxy.

_HDFS File Browser_

Ok! It looks as though our Hadoop cluster is up and running.  Otherwise known as HDFS. You can interact with Hadoop and HDFS with the _hadoop_ command. We do not have Hadoop installed on our local machine. Therefore, we can either log into one of our Hadoop cluster machines and run the hadoop command from there, or install hadoop on our local machine. I’m going to opt for installing Hadoop on my local machine (recommended), as it will be easier to interact with the HDFS and start the Hadoop map-reduce jobs directly from my laptop.

Cloudera does not, unfortunately, provide a release of Hadoop for Mac OS X. Only debians and RPMs. They do provide a .tar.gz download, which we are going to use to install Hadoop locally. Hadoop is built with Java and the scripts are written in bash, so there should not be too many problems with compatibility across platforms that can run Java and bash.

Visit [Cloudera CDH Release](https://docs.cloudera.com/display/DOC/Downloading+CDH+Releases) webpage and select [CDH3 Patched Tarball](https://archive.cloudera.com/cdh/3/). I downloaded the same version [hadoop-0.20.2+737.tar.gz](https://archive.cloudera.com/cdh/3/hadoop-0.20.2+737.tar.gz ) that Whirr installed on the cluster.

```
tar -xvzf hadoop-0.20.2+737.tar.gz
sudo mv hadoop-0.20.2+737 /usr/local/
cd /usr/local
sudo ln -s hadoop-0.20.2+737 hadoop
echo 'export HADOOP_HOME=/usr/local/hadoop' >> ~/.profile
echo 'export PATH=$PATH:$HADOOP_HOME/bin' >> ~/.profile
source ~/.profile
which hadoop # should output "/usr/local/hadoop/bin/hadoop"
hadoop version # should output "Hadoop 0.20.2+737 ..."
cp ~/.whirr/myhadoopcluster/hadoop-site.xml /usr/local/hadoop/conf/
```

Now run your first command from your local machine to interact with HDFS. This following command is similar to “ls -l /” in bash.

```
hadoop fs -ls /
```

You should see the following output which lists the root on the Hadoop filesystem.

```
10/12/30 18:19:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Found 4 items
drwxrwxrwx   - hdfs supergroup          0 2010-12-28 10:33 /hadoop
drwxrwxrwx   - hdfs supergroup          0 2010-12-28 10:33 /mnt
drwxrwxrwx   - hdfs supergroup          0 2010-12-28 10:33 /tmp
drwxrwxrwx   - hdfs supergroup          0 2010-12-28 10:33 /user
```

Yes, you will see a depreciation warning, since hadoop-site.xml configuration has been split into multiple files. We will not worry about this here.

## Defining The Map-Reduce Task

We are going write a map-reduce job that scans all the files in a given directory, takes the words found in those files and then counts the number of times words begin with any two characters.

For this we’re going to use a dictionary file found on my Mac OS X /usr/share/dict/words. It contains 234936 words, each on a newline. [Linux has a similar dictionary file](https://en.wikipedia.org/wiki/Words_(Unix)).

## Uploading Your Data To HDFS (Hadoop Distributed FileSystem)

```
hadoop fs -mkdir input
hadoop fs -put /usr/share/dict/words input/
hadoop fs -ls input
```

You should see output similar to the following, which list the _words_ file on the remote HDFS. Since my local user is “phil”, Hadoop has added the file under /user/phil on HDFS.

```
Found 1 items
-rw-r--r--   3 phil supergroup    2486813 2010-12-30 18:43 /user/phil/input/words
```

Congratulations! You have just uploaded your first file to the Hadoop Distributed File-System on your cluster in the cloud.

## Coding Your Map And Reduce Scripts in Ruby

Map-Reduce can actually be thought of as map-group-reduce.  The “map” also outputs the key of the data. Our key will be the two-letter prefix of each word. These keys are used by Hadoop to “group” the data together. The “reduce” then takes each group of data and “reduces” it. In our case the “reduce” will be the counting occurrences of the two-letter prefixes.

Hadoop will do much of the work for us. It will recurse the input directory, open the files and stream the files one line at a time into our “map” script via STDIN. We will output zero, one or many output lines to STDOUT for each line of input. Since we know that our input file has exactly one word per line, we can simplify our script and always output exactly one two-letter prefix for each input line. (EDIT: words with one letter will not result in any output).

The output of our “map” script to STDOUT will have to be Hadoop friendly. This means we will output our “key”, then a tab character then our value and then a newline. This is what the streaming interface expects. Hadoop needs to extract the key to be able to sort and organise the data based on this key.

```
<key><tab><value><newline>
```

Our value will always be “1″, since each line has only one word with only once instance of the two-letter prefix of that word.

For instance, if the input was “Apple” then we would output the key “ap” and value “1″. We have seen the prefix “ap” only once in this input.

You should note that the value can be anything that your reduce script can interpret. For instance, the value could be a string of JSON. Here, we are keeping it very simple.

```
ap<tab>1<newline>
```

Let’s code up the mapper as _map.rb_

```
# Ruby code for map.rb

ARGF.each do |line|

   # remove any newline
   line = line.chomp

   # do nothing will lines shorter than 2 characters
   next if ! line || line.length < 2

   # grab our key as the two-character prefix (lower-cased)
   key = line[0,2].downcase

   # value is a count of 1 occurence
   value = 1

   # output to STDOUT
   # <key><tab><value><newline>
   puts key + "\t" + value.to_s

end
```

Now we have our mapper script, let’s write the reducer.

Remember, the reducer is going to count up the occurences for each two-character prefix (our “key”). Hadoop will have already grouped our keys together, so even if the mapper output is in shuffled order, the reducer will now see the keys in sorted order. This means that the reducer can watch for when the key changes and know that it has seen all of the possible values for the previous key.

Here is an example of the STDIN and STDOUT that map.rb and reduce.rb might see. The data flow goes from left to right.

map.rb
STDIN
map.rb
STDOUT
Hadoop
sorts
keys
reduce.rb
STDIN
reduce.rb
STDOUT

Apple
Monkey
Orange
Banana
APR
Bat
appetite
ap 1
mo 1
or 1
ba 1
ap 1
ba 1
ap 1

ap 1
ap 1
ap 1
ba 1
ba 1
mo 1
or 1
ap 3
ba 2
mo 1
or 1

Let’s code up the reducer as _reduce.rb_

```
# Ruby code for reduce.rb

prev_key = nil
key_total = 0

ARGF.each do |line|

   # remove any newline
   line = line.chomp

   # split key and value on tab character
   (key, value) = line.split(/\t/)

   # check for new key
   if prev_key && key != prev_key && key_total > 0

      # output total for previous key

      # <key><tab><value><newline>
      puts prev_key + "\t" + key_total.to_s

      # reset key total for new key
      prev_key = key
      key_total = 0

   elsif ! prev_key
      prev_key = key

   end

   # add to count for this current key
   key_total += value.to_i

end
```

You can test out your scripts on a small sample by using the “sort” command in replacement for Hadoop.

```
cat /usr/share/dict/words | ruby map.rb | sort | ruby reduce.rb
```

The start of this output looks like this…

```
aa  13
ab  666
ac  1491
ad  867
ae  337
af  380
```

## Running The Hadoop Job

I wrote this bash-based runner script to start the job. It uses Hadoop’s streaming service. This streaming service is what allows us to write our map-reduce scripts in Ruby. It _streams_ to our script’s STDIN and reads our script’s output from our script’s STDOUT.

```bash
#!/bin/bash

HADOOP_HOME=/usr/local/hadoop
JAR=contrib/streaming/hadoop-streaming-0.20.2+737.jar

HSTREAMING="$HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/$JAR"

$HSTREAMING \
 -mapper  'ruby map.rb' \
 -reducer 'ruby reduce.rb' \
 -file map.rb \
 -file reduce.rb \
 -input '/user/phil/input/*' \
 -output /user/phil/output
```

We specify the command to run for the mapper and reducer and use the “-file” parameter twice to attach our two Ruby scripts. It is assumed that all other dependencies are already installed on the machine. In this case we are using no Ruby imports or requires and the Ruby interpreter is already installed on the machines in the Hadoop cluster (it came with the Cloudera Amazon EC2 image). Things become more complicated when you start to run jobs with more dependencies that are not already installed on the Hadoop cluster. This is a topic for another post.

“-input” and “-output” specify which files to read from for input and the directoty to send the output to. You can also specify a deeper level of recursion with more wildcards (e.g. “/user/phil/input/\*/\*/\*”).

Once again, it is important that our SOCKS proxy is running, as this is the secure way that we communicate through to our Hadoop cluster.

```bash
sh ~/.whirr/myhadoopcluster/hadoop-proxy.sh
    Running proxy to Hadoop cluster at ec2-72-44-45-199.compute-1.amazonaws.com. Use Ctrl-c to quit.
```

Now we can start the Hadoop job by running our above bash script. Here is the output the script gave me at the terminal.

```
packageJobJar: [map.rb, reduce.rb, /tmp/hadoop-phil/hadoop-unjar3366245269477540365/] [] /var/folders/+Q/+QReZ-KsElyb+mXn12xTxU+++TI/-Tmp-/streamjob5253225231988397348.jar tmpDir=null
10/12/30 21:45:32 INFO mapred.FileInputFormat: Total input paths to process : 1
10/12/30 21:45:37 INFO streaming.StreamJob: getLocalDirs(): [/tmp/hadoop-phil/mapred/local]
10/12/30 21:45:37 INFO streaming.StreamJob: Running job: job_201012281833_0001
10/12/30 21:45:37 INFO streaming.StreamJob: To kill this job, run:
10/12/30 21:45:37 INFO streaming.StreamJob: /usr/local/hadoop/bin/hadoop job  -Dmapred.job.tracker=ec2-72-44-45-199.compute-1.amazonaws.com:8021 -kill job_201012281833_0001
10/12/30 21:45:37 INFO streaming.StreamJob: Tracking URL: https://ec2-72-44-45-199.compute-1.amazonaws.com:50030/jobdetails.jsp?jobid=job_201012281833_0001
10/12/30 21:45:38 INFO streaming.StreamJob:  map 0%  reduce 0%
10/12/30 21:45:55 INFO streaming.StreamJob:  map 42%  reduce 0%
10/12/30 21:45:58 INFO streaming.StreamJob:  map 100%  reduce 0%
10/12/30 21:46:14 INFO streaming.StreamJob:  map 100%  reduce 88%
10/12/30 21:46:19 INFO streaming.StreamJob:  map 100%  reduce 100%
10/12/30 21:46:22 INFO streaming.StreamJob: Job complete: job_201012281833_0001
10/12/30 21:46:22 INFO streaming.StreamJob: Output: /user/phil/output
```

This is reflected if you visit the job tracker console in web browser.

_jobTracker after successful run_

If you click on the job link you can see lots of information on this job. This job is completed in these images, but with a longer running job you would see the progress as the job runs. I have split the job tracker page into the following three images.

_Map-Reduce Job Tracker Page (part 1)_

_Map-Reduce Job Tracker Page (part 2)_

_Map-Reduce Job Tracker Page (part 3) Graphs_

## The Results

Our map-reduce job has run successfully using Ruby. Let’s have a look at the output.

```
hadoop fs -ls output

Found 3 items
-rw-r--r--   3 phil supergroup          0 2010-12-30 21:46 /user/phil/output/_SUCCESS
drwxrwxrwx   - phil supergroup          0 2010-12-30 21:45 /user/phil/output/_logs
-rw-r--r--   3 phil supergroup       2341 2010-12-30 21:46 /user/phil/output/part-00000
```

Hadoop output is written in chunks to sequential files part-00000, part-00001, part-00002 and so on. Our dataset is very small, so we only have one 2kb file called part-00000.

```
hadoop fs -cat output/part-00000 | head
aa  13
ab  666
ac  1491
ad  867
ae  337
af  380
ag  507
ah  46
ai  169
aj  14
```

Our map-reduce script counted 13 words starting with “aa”, 666 words starting with “ab” and 1491 words starting with “ac”.

## Conclusion

Yes, it is an overkill to use Hadoop and a (very small) cluster of cloud-based machines for this example, but I think it demonstrates how you can quickly get your Hadoop cluster up and running map-reduce jobs written in Ruby. You can use the same procedure to fire-up a much larger and more powerful Hadoop cluster with a bigger dataset and more complex Ruby scripts.

__Please post any questions or suggestions you have in the comments below. They are always highly appreciated.__

## Resources

*   [Apache Hadoop](https://hadoop.apache.org/)
*   [Cloudera’s Distribution for Apache Hadoop (CDH)](https://www.cloudera.com/hadoop/)
*   [Cloudera Hadoop Training VMWare Image](https://www.cloudera.com/downloads/virtual-machine/)
*   [Map-Reduce Using Perl](https://www.slideshare.net/philwhln/map-reduce-using-perl)
*   [jclouds ](https://code.google.com/p/jclouds/)
*   [Words file on Unix-like operating systems](https://en.wikipedia.org/wiki/Words_(Unix))
*   [Ruby Weekly](https://rubyweekly.com/)

## Related Books

If you are interested in learning more about Hadoop, then I recommend reading
[Hadoop: The Definitive Guide (2nd Edition)](https://www.amazon.com/gp/product/1449389732?ie=UTF8&tag=getafil-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=1449389732) by Tom White.

## Comments

          andy

        January 12, 2011 at 3:49 pm

kudos on the good article

I was able to follow along and everything worked (after correcting for my typos).

One thing that wasn’t clear, one can be in any dir for the ruby & bash file work.  I had done this originally in the hadoop folder.

thanks

              Phil Whelan

            January 12, 2011 at 4:07 pm

Thank you for your comment Andy! I’m glad you followed it all the way through and it worked.

In this example the Ruby scripts for Map and Reduce are in the same directory as the bash script that I created for running the job. They do not need to be. You just need to tell the job starter where to find them. For instance, for the map.rb script I have…

-file map.rb

This is a relative path, meaning it’s in the same directory. It could also be relative in another directory..

-file ../someotherdirectory/map.rb

or we could give an absolute path…

-file /Users/phil/someotherdirectory/map.rb

The bash script can also be written is any directory. The path to the JAR file is absolute, so the only other dependencies are the Map and Reduce scripts as I mentioned above.

I hope that answers your question.

          andy

        January 13, 2011 at 10:45 am

Hi Phil,

one more question,

is there a way to use s3 buckets for the input and output?

              Phil Whelan

            January 13, 2011 at 11:07 am

That’s a great question Andy and the answer is “yes”. You will need to modify your hadoop-site.xml to replace HDFS with S3.

Check out this wiki page…

https://wiki.apache.org/hadoop/AmazonS3

This would make a good follow post, so keep watching this space!

          Navin

        January 16, 2011 at 2:45 am

Phil – thanks very much for taking the time to write this up!  I am new to AWS and EC2 and am running into a problem at the point where I first try and launch clusters – wondering if you can please help?  I have my Access Key ID and Secret Access Key in my hadoop.properties, and the output I get is as follows:

[~/src/cloudera/whirr-0.1.0+23]$ bin/whirr launch-cluster --config hadoop.properties                                                                                rvm:ruby-1.8.7-p299
Launching myhadoopcluster cluster
Exception in thread "main" com.google.inject.CreationException: Guice creation errors:
1) No implementation for java.lang.String annotated with @com.google.inject.name.Named(value=jclouds.credential) was bound.
  while locating java.lang.String annotated with @com.google.inject.name.Named(value=jclouds.credential)
    for parameter 2 at org.jclouds.aws.filters.FormSigner.(FormSigner.java:91)
  at org.jclouds.aws.config.AWSFormSigningRestClientModule.provideRequestSigner(AWSFormSigningRestClientModule.java:66)
1 error
  at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:410)
  at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:166)
  at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:118)
  at com.google.inject.InjectorBuilder.build(InjectorBuilder.java:100)
  at com.google.inject.Guice.createInjector(Guice.java:95)
  at com.google.inject.Guice.createInjector(Guice.java:72)
  at org.jclouds.rest.RestContextBuilder.buildInjector(RestContextBuilder.java:141)
  at org.jclouds.compute.ComputeServiceContextBuilder.buildInjector(ComputeServiceContextBuilder.java:53)
  at org.jclouds.aws.ec2.EC2ContextBuilder.buildInjector(EC2ContextBuilder.java:101)
  at org.jclouds.compute.ComputeServiceContextBuilder.buildComputeServiceContext(ComputeServiceContextBuilder.java:66)
  at org.jclouds.compute.ComputeServiceContextFactory.buildContextUnwrappingExceptions(ComputeServiceContextFactory.java:72)
  at org.jclouds.compute.ComputeServiceContextFactory.createContext(ComputeServiceContextFactory.java:114)
  at org.apache.whirr.service.ComputeServiceContextBuilder.build(ComputeServiceContextBuilder.java:41)
  at org.apache.whirr.service.hadoop.HadoopService.launchCluster(HadoopService.java:84)
  at org.apache.whirr.service.hadoop.HadoopService.launchCluster(HadoopService.java:61)
  at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:61)
  at org.apache.whirr.cli.Main.run(Main.java:65)
  at org.apache.whirr.cli.Main.main(Main.java:91)

PS: The rvm:ruby-1.8.7-p299 is part of my prompt and not in the output.

Thanks in advance for any pointers on getting this resolved.

Regards,

Navin

          Adrian Cole

        January 16, 2011 at 11:24 am

Hi, Navin.

There’s probably an issue with the property syntax inside your hadoop.properties.  Have a look at https://incubator.apache.org/projects/whirr.html and https://cwiki.apache.org/WHIRR/configuration-guide.html

If you still have issues, send your query to the user list, you’ll get on track quickly!

<EMAIL_ADDRESS>
Cheers,

-Adrian

          Navin

        January 16, 2011 at 11:57 am

Um, yes, sorry! A problem with “whirr.credentials” getting truncated to “whirr.cred” while copying keys across.  How embarrassing!

Apologies – and thanks for taking the time to reply …

Navin

              Phil Whelan

            January 16, 2011 at 12:04 pm

Excellent! Glad you got if figured out. When you get through to the end of the example, let us know your thoughts are of using Whirr + Hadoop + Ruby.

                  Navin

                February 3, 2011 at 2:23 pm

Worked my way through the tutorial soon after my last comment and have been meaning to come back and thank you for writing this up!  I am inspired to pick something from this list

https://www.quora.com/What-are-some-good-toy-problems-in-data-science

and play some more.

Not all is well though – I received a bill from AWS for $100 this morning!  While following your tutorial I fumbled my way through signing up for AWS and assumed that I would only ever be operating within a free tier – boy was I wrong!  Some three weeks later and I realise that I’ve had “small” EC2 servers going all this time …

So, may I suggest that you please augment your post to direct the reader to shut down (and terminate) their EC2 instances after they are done with the tute, and, perhaps, include directions on how to ensure that they use Amazon’s free https://aws.amazon.com/free/ tier?

It is evident that I’ve made some silly mistakes while following this but I hope I am not atypical of your garden variety newbie that is curious about big data.  I hope that I find a way to get Amazon to reverse charges (don’t like my chances though).  In any case, my enthusiasm is not dampened – thanks again for getting me started

Navin

                      Navin

                    February 5, 2011 at 11:51 pm

Further update: Amazon replied to my request within 12 hours and, in fact, did reverse charges.  Phew!

          Phil Whelan

        January 21, 2011 at 7:04 pm

I have just posted “Quickly Launch A Cassandra Cluster On Amazon EC2″, which follows a very similar process. https://www.bigfastblog.com/quickly-launch-a-cassandra-cluster-on-amazon-ec2

          cjbottaro

        February 18, 2011 at 9:35 pm

Great post, thank you very much for that.

Next question though… how do you use Cassandra for input/output (while still using Ruby)?

I know you can run Hadoop jobs against Cassandra in Java with the InputFormat they provide, but how to do so using streaming?

          Sid

        April 12, 2011 at 3:44 am

What a great post! Really clearly written and keeps to it’s promise of being able to demonstrate through repeatable steps. Particularly liked the cues for tea breaks for the lengthy download/install steps. Nice work!

          Allen

        May 28, 2011 at 12:19 am

When i would like fire up a customized ec2 instance, i added following parameters into hadoop.properties:

whirr.image-id= us-west-1/ami-**********
jclouds.ec2.ami-owners=******************
whirr.hardware-id= m1.large
whirr.location-id=us-west-1

But it doesn’t work. Any thought? thanks

          Adrian Cole

        May 28, 2011 at 11:09 am

I think you would be best taking this to the whirr user list, where you can let us know what didn’t work:

   https://cwiki.apache.org/confluence/display/WHIRR/MailingLists

There are recipes in the latest version of whirr here, as well:

   https://svn.apache.org/repos/asf/incubator/whirr/trunk/recipes/hadoop-ec2.properties

Unrelated, but if you don’t mind trying 0.5.0, voting it up can get the new rev, including recipes released!

   https://people.apache.org/~tomwhite/whirr-0.5.0-incubating-candidate-1/

Cheers,

A

              Allen

            May 29, 2011 at 1:07 am

Cool, I will try both, post the questions and new Whirr, will update here later on if i get any solution. Thx

              Allen

            May 29, 2011 at 3:54 am

I tried whirr 0.5.0, and it still doesn’t work.

It worked well if my hadoop.properties as below:

whirr.service-name=hadoop
whirr.cluster-name=myhadoopcluster
whirr.instance-templates=1 jt+nn,1 dn+tt
whirr.provider=ec2
whirr.identity=
whirr.credential=
whirr.private-key-file=${sys:user.home}/.ssh/id_rsa
whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub
whirr.hardware-id= m1.large
whirr.location-id=us-west-1
whirr.hadoop-install-runurl=cloudera/cdh/install
whirr.hadoop-configure-runurl=cloudera/cdh/post-configure

But once i added two more lines into hadoop.properties file, it went wrong:

whirr.image-id= us-west-1/ami-***** (my ami)
jclouds.ec2.ami-owners=(my owner id>

I have posted a question on whirr forum and see if i could get any solution. Will update here if i get anything. thx

          Jack Veenstra

        June 8, 2011 at 6:22 pm

There’s a bug in your reduce script.  You output the total only when you get a new key.  So the last key’s total will never be included in the output.

              Phil Whelan

            June 8, 2011 at 11:21 pm

Thanks Jack! You’re right. Well spotted.

          threecuptea

        August 6, 2011 at 6:37 pm

I started using EC2 yesterday and I got this working today thanks to your article.  However, it’s not without twist and turn.

1.  I got https://whirr.s3.amazonaws.com/0.3.0-cdh3u1/util/configure-hostnames not found error when I run ‘whirr launch-cluster’. They haven’t put configure-hostnames for cdh3u1 in s3 yet.   I workaround by adding –run-url-base https://whirr.s3.amazonaws.com/0.3.0-cdh3u0/util.  I got this advice from https://www.cloudera.com/blog/2011/07/cdh3u1-released/

2. I followed the instruction to set up Hadoop client in the the host initiating whirr and got the following error when it tried to connect to name node.

11/08/07 01:03:48 INFO ipc.Client: Retrying connect to server: ec2-107-20-60-75.

compute-1.amazonaws.com/<IP_ADDRESS>:8020. Already tried 9 time(s).

Bad connection to FS. command aborted. exception: Call to ec2-107-20-60-75.compu

te-1.amazonaws.com/<IP_ADDRESS>:8020 failed on local exception: java.net.Socket

Exception: Connection refused

It has to do with security.  I checked the security group jclouds#myhadoopcluster3#us-east-1.  It allows inbound on 80, 50070, 50030 only from the host initiating whirr launch-cluster and allow inbound on 8020, 8021 only from the name node host.   I added rules to allow inbound on 8020, 8021 from the host initiating whirr and apply the rule change.  That doesn’t help.   In my case, the host initiating whirr launch-cluster is a EC2 instance too.

3.  I can ssh to cluster hosts from the host initiating whirr without any key.  iptable is empty and selinux is disabled.  Network rules seems set up outside the linux box. No luck.

4.  I ends up transferring files to name nodes and run map reduce job there. Whirr script create /user/hive/warehouse but no /user/ec2-user.  Need to create that directory and input sub-directory.  You might also add  -jobconf mapred.reduce.tasks=1 since the default is 10 in this case.

Thanks.

              Phil Whelan

            August 9, 2011 at 11:39 am

Hi threecuptea,

Do you mind re-posting this to whirr’s mailing list?

https://incubator.apache.org/whirr/mail-lists.html

There’s a release almost finished which should support cdh much better (v0.6)

          Harit

        August 6, 2011 at 11:08 pm

I guess the reducer is missing the code,because the last line when it completes, has to put the result to the output.

I ran the same logic in Java and then in Ruby using hadoop and realized that my last node is missing in the result data. so I added the following line at the very end of reducer.rb

puts prev_key + separator + key_total.to_s

and it worked.

          Daniel

        September 13, 2011 at 1:43 pm

Hi Phil,

I am using Hadoop Mapreduce to predict secondary structure of a given long sequence. The idea is, I have a chunks of segments of a sequence and they are written into a single file input where each line is one segment. I have used one of the programs for secondary structure predictions as my mapper code (Hadoop Streaming).

The out put of the mapper was successful that it produces the predicted structures in terms of dot-bracket notation. I want to use a simple reducer that glue all the outputs from the mapper in an orderly manner.

For Example, If my input was like

….

And my mapper output is a predicted structure but not in order

What I am looking is a reducer code that sorts and Glue and outputs in a form similar to the following:

……

Any help…Thanks

          Dr. SHyam Sarkar

        September 23, 2011 at 7:47 pm

Hello,

We have following properties set :

whirr.service-name=hadoop
whirr.cluster-name=myhadoopcluster
whirr.instance-templates=1 jt+nn,1 dn+tt
whirr.provider=ec2
whirr.credential=mYar/KSbx+UL+nqGr9hSgGHIOqXC9tjNcuO9UwF/
whirr.identity=<REDACTED_AWS_ACCESS_KEY>
whirr.private-key-file=${sys:user.home}/.ssh/id_rsa
whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub
whirr.hadoop-install-runurl=cloudera/cdh/install
whirr.hadoop-configure-runurl=cloudera/cdh/post-configure
whirr.image-id=ami-3bc9997e
whirr.hardware-id=i-bffa23f8
whirr.location-id=us.west-1c

But we are getting following error:

[ec2-user@ip-10-170-103-243 ~]$ ./whirr-0.3.0-cdh3u1/bin/whirr launch-cluster –config hadoop.properties –run-url-base https://whirr.s3.amazonaws.com/0.3.0-cdh3u0/util
Bootstrapping cluster
Configuring template
Exception in thread “main” java.util.NoSuchElementException
        at com.google.common.collect.AbstractIterator.next(AbstractIterator.java:147)
        at com.google.common.collect.Iterators.find(Iterators.java:679)
        at com.google.common.collect.Iterables.find(Iterables.java:555)
        at org.jclouds.compute.domain.internal.TemplateBuilderImpl.locationId(TemplateBuilderImpl.java:492)
        at org.apache.whirr.service.jclouds.TemplateBuilderStrategy.configureTemplateBuilder(TemplateBuilderStrategy.java:41)
        at org.apache.whirr.service.hadoop.HadoopTemplateBuilderStrategy.configureTemplateBuilder(HadoopTemplateBuilderStrategy.java:31)
        at org.apache.whirr.cluster.actions.BootstrapClusterAction.buildTemplate(BootstrapClusterAction.java:144)
        at org.apache.whirr.cluster.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:94)
        at org.apache.whirr.cluster.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:74)
        at org.apache.whirr.service.Service.launchCluster(Service.java:71)
        at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:61)
        at org.apache.whirr.cli.Main.run(Main.java:65)
        at org.apache.whirr.cli.Main.main(Main.java:91)

Can we get any help ?  What shold we do ?

Thanks,

S.Sarkar

          Joao Salcedo

        February 12, 2012 at 12:52 am

Nice tutorial, Everything work just how it should be !!!

Just a small question , what if I wanna connect to the instance , where I can find the key in order to connect to it.

Cheers,

Joao

          Andrii Vozniuk

        May 15, 2012 at 9:27 am

Phil, thanks for the detailed tutorial!

I had my custom MapReduce application up and running on an EC2 cluster just in a few hours.

I reproduced the steps with whirr-0.7.1 and hadoop-0.20.2-cdh3u4.

              Phil Whelan

            May 15, 2012 at 3:40 pm

Great! I’m glad the process still working and the blog post is still valid. Thanks for the comment.

          bodla dharani kumar

        February 15, 2014 at 10:58 pm

hi to all,

Good Morning,

I had a set of 22documents in the form of text files of size 20MB and loaded in hdfs,when running hadoop streaming map/reduce funtion from command line of hdfs ,it took 4mins 31 secs for streaming the 22 text files.How to increase the map/reduce process as fast as possible so that these text files should complete the process by 5-10 seconds.

What changes I need to do on ambari hadoop.

And having cores = 2

Allocated 2GB of data for Yarn,and 400GB for HDFS

default virtual memory for a job map-task = 341 MB

default virtual memory for a job reduce-task = 683 MB

MAP side sort buffer memory = 136 MB

And when running a job ,Hbase error with Region server goes down,Hive metastore status service check timed out.

Note:[hdfs@s ~]$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-streaming-<IP_ADDRESS>.0.6.0-76.jar -D mapred.map.tasks=2 -input hdfs:/apps/*.txt -output /home/ambari-qa/8.txt -mapper /home/coartha/mapper.py -file /home/coartha/mapper.py -reducer /home/coartha/reducer.py -file /home/coartha/reducer.py

Thanks & regards,

Bodla Dharani Kumar,

repository: philwhln/blog

repo_name: philwhln/blog


text:
# Change Log

All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.

# [3.0.0](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@2.0.8<EMAIL_ADDRESS>(2023-08-03)

- **feat(accordion)!:** add sections prop ([#572](https://github.com/zendeskgarden/react-containers/issues/572)) ([e5cc2e1](https://github.com/zendeskgarden/react-containers/commit/e5cc2e1659fb0ed75a070cb808c21ec6a475f69b))

### BREAKING CHANGES

- updated container-accordion API

## [2.0.8](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@2.0.7<EMAIL_ADDRESS>(2023-07-28)

### Bug Fixes

- **deps:** update non-major package dependencies ([#556](https://github.com/zendeskgarden/react-containers/issues/556)) ([6831969](https://github.com/zendeskgarden/react-containers/commit/6831969ebb4390546f0159c5803121d711ef91bd))

## [2.0.7](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@2.0.6<EMAIL_ADDRESS>(2023-07-21)

**Note:** Version bump only for package @zendeskgarden/container-accordion

## [2.0.6](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@2.0.5<EMAIL_ADDRESS>(2023-06-12)

### Bug Fixes

- **deps:** update non-major package dependencies ([#543](https://github.com/zendeskgarden/react-containers/issues/543)) ([08856fc](https://github.com/zendeskgarden/react-containers/commit/08856fca9b08f7434b91bf1b95b4d2fff497d75f))

## [2.0.5](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@2.0.4<EMAIL_ADDRESS>(2023-04-19)

### Bug Fixes

- **deps:** update non-major package dependencies ([#528](https://github.com/zendeskgarden/react-containers/issues/528)) ([5df36aa](https://github.com/zendeskgarden/react-containers/commit/5df36aa7c5e78dc0da79a95416e915cc8e1348da))

## [2.0.4](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@2.0.3<EMAIL_ADDRESS>(2023-03-17)

### Bug Fixes

- update size snapshots ([#524](https://github.com/zendeskgarden/react-containers/issues/524)) ([67c3f83](https://github.com/zendeskgarden/react-containers/commit/67c3f83a41f89ec3a6dfde986c85405b893f7b74))

## [2.0.3](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@2.0.2<EMAIL_ADDRESS>(2023-03-17)

**Note:** Version bump only for package @zendeskgarden/container-accordion

## [2.0.2](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@2.0.1<EMAIL_ADDRESS>(2023-02-17)

**Note:** Version bump only for package @zendeskgarden/container-accordion

## [2.0.1](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@2.0.0<EMAIL_ADDRESS>(2022-10-10)

**Note:** Version bump only for package @zendeskgarden/container-accordion

# [2.0.0](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@1.1.3<EMAIL_ADDRESS>(2022-06-17)

**Note:** Version bump only for package @zendeskgarden/container-accordion

## [1.1.3](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@1.1.2<EMAIL_ADDRESS>(2022-03-31)

**Note:** Version bump only for package @zendeskgarden/container-accordion

## [1.1.2](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@1.1.1<EMAIL_ADDRESS>(2022-02-28)

**Note:** Version bump only for package @zendeskgarden/container-accordion

## [1.1.1](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@1.1.0<EMAIL_ADDRESS>(2021-11-30)

**Note:** Version bump only for package @zendeskgarden/container-accordion

# [1.1.0](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@1.0.10<EMAIL_ADDRESS>(2021-11-10)

### Features

- **accordion:** add default expanded sections prop ([#397](https://github.com/zendeskgarden/react-containers/issues/397)) ([c62aa96](https://github.com/zendeskgarden/react-containers/commit/c62aa96d9988d1037351385e071add2ecffc3a18))

## [1.0.10](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@1.0.9<EMAIL_ADDRESS>(2021-11-08)

### Bug Fixes

- package lock files out of sync ([#396](https://github.com/zendeskgarden/react-containers/issues/396)) ([db47ef7](https://github.com/zendeskgarden/react-containers/commit/db47ef7e099977a015b8d545bff8be74efc027be))

## [1.0.9](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@1.0.8<EMAIL_ADDRESS>(2021-09-27)

**Note:** Version bump only for package @zendeskgarden/container-accordion

## [1.0.8](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@1.0.7<EMAIL_ADDRESS>(2021-09-03)

**Note:** Version bump only for package @zendeskgarden/container-accordion

## [1.0.7](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@1.0.6<EMAIL_ADDRESS>(2021-09-02)

**Note:** Version bump only for package @zendeskgarden/container-accordion

## [1.0.6](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@1.0.5<EMAIL_ADDRESS>(2021-08-16)

**Note:** Version bump only for package @zendeskgarden/container-accordion

## [1.0.5](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@1.0.4<EMAIL_ADDRESS>(2021-03-19)

### Bug Fixes

- **accordions:** provide valid React keys ([#281](https://github.com/zendeskgarden/react-containers/issues/281)) ([dc58f61](https://github.com/zendeskgarden/react-containers/commit/dc58f6199dccfa63911e4b9971ffc0eb9e42c4e3))

## [1.0.4](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@1.0.3<EMAIL_ADDRESS>(2021-02-25)

**Note:** Version bump only for package @zendeskgarden/container-accordion

## [1.0.3](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@1.0.2<EMAIL_ADDRESS>(2021-01-15)

**Note:** Version bump only for package @zendeskgarden/container-accordion

## [1.0.2](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@1.0.1<EMAIL_ADDRESS>(2020-10-23)

### Bug Fixes

- **accordion:** replace hidden styles to prevent Chromium auto-scroll ([#226](https://github.com/zendeskgarden/react-containers/issues/226)) ([1a489dd](https://github.com/zendeskgarden/react-containers/commit/1a489dda8ec260f314c79b0ba7676597c0114c6b))

## [1.0.1](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@1.0.0<EMAIL_ADDRESS>(2020-06-10)

**Note:** Version bump only for package @zendeskgarden/container-accordion

# [1.0.0](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@0.5.4<EMAIL_ADDRESS>(2020-04-06)

### chore

- **accordion:** update changelog ([#184](https://github.com/zendeskgarden/react-containers/issues/184)) ([b971386](https://github.com/zendeskgarden/react-containers/commit/b971386daca9115bc0bebbd97ef62977e8d3547c)), closes [#176](https://github.com/zendeskgarden/react-containers/issues/176)

### BREAKING CHANGES

- **accordion:** Update container-accordion changelog.

## [0.5.4](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@0.5.3<EMAIL_ADDRESS>(2020-03-26)

**Note:** Version bump only for package @zendeskgarden/container-accordion

## [0.5.3](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@0.5.2<EMAIL_ADDRESS>(2020-03-23)

### BREAKING CHANGES

The `useAccordion` hook's `onChange` handler has been updated to be called with the index of the expanded accordion section. To migrate please update `onChange` usages to handle the newly returned index value.

- **utils:** add typing to getControlledValue ([#173](https://github.com/zendeskgarden/react-containers/issues/173)) ([886ba5b](https://github.com/zendeskgarden/react-containers/commit/886ba5b6bd595c8946ec1fc6e5c2f8ec7e8ac3eb))

## [0.5.2](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@0.5.1<EMAIL_ADDRESS>(2020-02-20)

### Bug Fixes

- **build:** improve bundle size with @babel/runtime ([#162](https://github.com/zendeskgarden/react-containers/issues/162)) ([506504c](https://github.com/zendeskgarden/react-containers/commit/506504c840795f34e420b016b94cef10440a30cb))

## [0.5.1](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@0.5.0<EMAIL_ADDRESS>(2020-02-14)

### Bug Fixes

- **build:** apply babel to all bundles ([#160](https://github.com/zendeskgarden/react-containers/issues/160)) ([826735b](https://github.com/zendeskgarden/react-containers/commit/826735bba881d5247b423ffb61cf9643c6599d16))

# [0.5.0](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@0.4.0<EMAIL_ADDRESS>(2020-02-12)

### Features

- **build:** introduce rollup for CJS and ES module bundles ([#158](https://github.com/zendeskgarden/react-containers/issues/158)) ([58313d4](https://github.com/zendeskgarden/react-containers/commit/58313d486e3bfa023e2c9d090149d7ec358d0cd0))

# [0.4.0](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@0.3.3<EMAIL_ADDRESS>(2020-01-15)

### Features

- move ID utilities to react-uid ([#142](https://github.com/zendeskgarden/react-containers/issues/142)) ([9f30572](https://github.com/zendeskgarden/react-containers/commit/9f3057202c94ca497b11b6f05ef649c87d5a5716))

## [0.3.3](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@0.3.2<EMAIL_ADDRESS>(2019-11-04)

**Note:** Version bump only for package @zendeskgarden/container-accordion

## [0.3.2](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@0.3.1<EMAIL_ADDRESS>(2019-10-22)

### Bug Fixes

- update render prop function return types ([#121](https://github.com/zendeskgarden/react-containers/issues/121)) ([eaea3fd](https://github.com/zendeskgarden/react-containers/commit/eaea3fd61a16085ef480ddbd2d67aa377738db36))

## [0.3.1](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@0.3.0<EMAIL_ADDRESS>(2019-10-16)

**Note:** Version bump only for package @zendeskgarden/container-accordion

# [0.3.0](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@0.2.1<EMAIL_ADDRESS>(2019-10-08)

### Features

- **accordion:** migrate useAccordion to TypeScript ([#109](https://github.com/zendeskgarden/react-containers/issues/109)) ([b17cb34](https://github.com/zendeskgarden/react-containers/commit/b17cb34))

## [0.2.1](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@0.2.0<EMAIL_ADDRESS>(2019-09-25)

**Note:** Version bump only for package @zendeskgarden/container-accordion

# [0.2.0](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@0.1.4<EMAIL_ADDRESS>(2019-09-20)

### Features

- **accordion:** allow useAccordion hook to become controlled ([#104](https://github.com/zendeskgarden/react-containers/issues/104)) ([841a2bd](https://github.com/zendeskgarden/react-containers/commit/841a2bd))

## [0.1.4](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@0.1.3<EMAIL_ADDRESS>(2019-09-19)

**Note:** Version bump only for package @zendeskgarden/container-accordion

## [0.1.3](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@0.1.2<EMAIL_ADDRESS>(2019-09-13)

**Note:** Version bump only for package @zendeskgarden/container-accordion

## [0.1.2](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@0.1.1<EMAIL_ADDRESS>(2019-09-05)

**Note:** Version bump only for package @zendeskgarden/container-accordion

## [0.1.1](https://github.com/zendeskgarden/react-containers/compare/@zendeskgarden/container-accordion@0.1.0<EMAIL_ADDRESS>(2019-07-25)

**Note:** Version bump only for package @zendeskgarden/container-accordion

# 0.1.0 (2019-05-23)

### Features

- **accordion:** introduce useAccordion and AccordionContainer ([#33](https://github.com/zendeskgarden/react-containers/issues/33)) ([6fd5716](https://github.com/zendeskgarden/react-containers/commit/6fd5716))

repository: zendeskgarden/react-containers

branch_name: refs/heads/main

extension: md

gha_language: TypeScript

license_type: permissive

repo_name: zendeskgarden/react-containers


text:
# Gomphillus calycioides f. microcephalus (Taylor) Nyl. FORM

#### Status
SYNONYM

#### According to
Index Fungorum

#### Published in
Syn. Lich. 1: 175 (1860)

#### Original name
Baeomyces microcephalus Taylor

### Remarks
null

repository: mdoering/backbone

path: /life/Fungi/Ascomycota/Lecanoromycetes/Ostropales/Gomphillaceae/Gomphillus/Gomphillus calycioides/ Syn. Gomphillus calycioides microcephalus/README.md

branch_name: refs/heads/master

repo_name: mdoering/backbone


text:
# aspMono
Sample ASP application on Linux using Mono

Usage: This instruction assumes you have docker installed and working

###### Clone repo
git clone https://github.com/Tosyn/aspMono.git

###### Create docker image
cd aspMono
docker build -t aspapp .

###### Start application
docker-compose up

###### Update host machine etc host file : add text below
asp.dev

##### Click link below to view in browser
http://asp.dev:8181/

license: MIT

repository: Tosyn/aspMono

repo_name: Tosyn/aspMono


text:
[![CircleCI](https://circleci.com/gh/nsimao/sfg-pet-clinic.svg?style=svg)](https://circleci.com/gh/nsimao/sfg-pet-clinic)

# Spring Framework Guru Pet Clinic

This repo is for the online course called [Spring Framework 5: Beginner to Guru](https://www.udemy.com/spring-framework-5-beginner-to-guru/?couponCode=GITHUB_SFGPETCLINIC).

Contained in this repo is an opinionated version of the classic reference application - [Spring Pet Clinic](https://github.com/spring-projects/spring-petclinic)

repository: nsimao/sfg-pet-clinic

repo_name: nsimao/sfg-pet-clinic


text:
# Webber
========

### Overview
Webber is a lightweight, HTTP Web Request Helper / utility written in C#.
The idea was to have a very simple and flexible abstraction of doing HTTP requests with minimal lines of code to perform the request.

### Usage
####GET
Performing a GET request is as simple as
```csharp
var response = Webber.Get<SamplePost>("http://jsonplaceholder.typicode.com/posts/1");
```

####POST
Shoot a POST request with a wide-range of flexibility
```csharp
//Simple POST
var webberResponse = Webber.Post<SamplePost>(url, data);

//Get flexible control on overriding the default parameters
var webberResponse = Webber.Post<Tests.SamplePost>(
                url,
                data,
                ContentType.Json,
                EncodingType.Utf8,
                new NetworkCredential("myUserName", "P@ssW0rd"),
                myCustomerHeaders);
```

####Any HTTP Verb
```csharp
var webberResponse = Webber.Invoke(
                url,
                data,
                ContentType.Json, // Specify the request's content-type
                MethodType.Put, // Specify the http verb here
                EncodingType.Utf8, // Specify the request's encoding type here
                new NetworkCredential("myUserName", "P@ssW0rd"), // Add your ICrendential when necessary
                myCustomerHeaders); // Add additional headers the reqest
```

###Response Deserialization
These web requests' responses can be auto-deserialized from a Json. The library only has 1 dependency, that is Json.Net. Here's a good article on how Json.net is better than the built in .net serializers http://www.newtonsoft.com/json/help/html/jsonnetvsdotnetserializers.htm


text:
# NCurses Sorting Visualizer
Visualizer with options to show sorts for bubble, shell,
quick and heap sorts. Also scales to terminal size.

## Build
You can build the program by running `make`

## Options
   - a              Enable all sorts.
   - b              Enable Bubble Sort.
   - s              Enable Shell Sort.
   - q              Enable Quick Sort.
   - h              Enable Heap Sort.
   - r seed         Specify random seed.

> EX: `./sorting -h -r 1337` will run heap sort with a seed of 1337.
> This is the output:

![heap](https://user-images.githubusercontent.com/65222208/112743300-f672c280-8f4a-11eb-83e3-19f1dd70929c.gif)

repository: taidanh/ncurse-sorting

repo_name: taidanh/ncurse-sorting


text:
mdadm Basic Monitor
===

This plugin gets stats from /proc/mdstat and for each md* device returns a 1 if the amount of raid disks does not equal the amount of non degraded disks. If the amount of raid disks equals the amount of non degraded disks a 0 is returned

Requirements
---
* Python 2.6+
* mdstat - https://pypi.python.org/pypi/mdstat/

Installation - v1 Agent
---
1. Check if you have mdstat installed. Run:
  ```
python -mdstat
```
  When you hit return, if the command completes and you get no errors then you have the module already installed and can skip to step 3 below.
Otherwise you'll get the following error:
  ```
/usr/bin/python: No module named mdstat
```

2. Install mdstat:
  ```
pip install mdstat
```

3. Download the [Mdadm.py](Mdadm.py) plugin file into your [Server Density agent plugin directory](/README.md).

4. Restart the agent

Installation - v2 Agent
---
1. Check if you have mdstat installed in the agent venv. Run:
  ```
source /usr/share/python/sd-agent/bin/activate
python -mdstat
deactivate
```
  When you hit return, if the command completes and you get no errors then you have the module already installed and can skip to step 3 below.
Otherwise you'll get the following error:
  ```
/usr/bin/python: No module named mdstat
```

2. Install mdstat:
  ```
source /usr/share/python/sd-agent/bin/activate
pip install mdstat
deactivate
```

3. Download the [Mdadm.py](Mdadm.py) plugin file into your [Server Density agent plugin directory](/README.md).

4. Restart the agent

Recommended alerts
---
* `md*_degraded` - Not equal to 0 where * is equal to your md device number.

Configuration
---
None

repository: serverdensity/sd-agent-plugins

gha_language: Python

repo_name: serverdensity/sd-agent-plugins


text:
# Social-Login-With-Hapi
The source code provided is an example of how to setup social login functionality into your node application.

# Dependencies
The backend will use the following technologies and frameworks:

  - [NodeJS](http://nodejs.org/)
  - [Node-HapiJS](https://github.com/hapijs/hapi)
  - [Node-Bell](https://github.com/hapijs/bell)

# Documentation
You can find the blog article related to this code here [social-login-with-hapi](http://blog.itproven.com/technology/nodejs/social-login-with-hapi/).

License
=========
MIT [http://rem.mit-license.org](http://rem.mit-license.org)

repository: salimkapadia/social-login-with-hapi

repo_name: salimkapadia/social-login-with-hapi


text:
# loan-broker
The repository for the Loan Broker Project

To run git clone https://github.com/semester-groupies/loan-broker.git into a folder.

Cd to the directory, then cd loan-broker and run npm install.

Then in the terminal run npm start.

The program will be running at http://localhost:3000/form .

repository: semester-groupies/loan-broker

gha_language: JavaScript

repo_name: semester-groupies/loan-broker


text:
# flickr-mirror

This probably isn't useful yet. Currently it can index a folder structure like

```
data.json
photos/
<PHONE_NUMBER>/
    data.json
    photo_o.jpg
    photo_z.jpg
  ...
sets/
  413133334/
    data.json
```

which is currently produced by [hawx/hall-of-mirrors][].

Once the data has been indexed you can spin up a local server to view your photos.

[hawx/hall-of-mirrors]: https://github.com/hawx/hall-of-mirrors/

repository: hawx/flickr-mirror

repo_name: hawx/flickr-mirror


text:
---
layout: post
title: "vagrant after apt-get upgrade"
published: true
---

It is the solution I found: https://github.com/mitchellh/vagrant/issues/733

check output of lsmod | grep vboxsf, empty? if yes, need rebuild the guest extentions.

/etc/init.d/vboxadd setup

extension: markdown


text:
## Welcome to Digital Image TimeStamping Projects

Digital image processing with tamper-proof timestamp stegano supports assertions of proof that a digital image existed before a particular time with integrity. I designed digital shadow concept to protect and proof that digital image is not tampered from which created first time.
As current, there is no solution for proof of existence and integrity for digital image. Digital watermarking technology is focused on protect owner licenses but not integrity with proof of existence.
With this solution, you can solve the legal compliance issue for scanning paper document, photo. As tamper-proof timestamping image, photo will be compromised way to digital transformation.
My solution for image integrity has unique way of image processing techniques.
There are two step for image processing to tamper-proof and existence-proof.
First step. Digital Shadow model will calculate each digital image's unique shadow data(footprint of image) which purpose to provide zero knowledge proof. I design zk-SNARKS method applies to digital image.
Second step. Steganography with timestamp/digital signed token data into digital image and detect method. Unlike almost competitors using one-way function(hash) technique, this solution use inverse function to proof of integrity.

### Markdown

Markdown is a lightweight and easy-to-use syntax for styling your writing. It includes conventions for

```markdown
Syntax highlighted code block

# Header 1
## Header 2
### Header 3

- Bulleted
- List

1. Numbered
2. List

**Bold** and _Italic_ and `Code` text

[Link](url) and ![Image](src)
```

For more details see [GitHub Flavored Markdown](https://guides.github.com/features/mastering-markdown/).

### Jekyll Themes

Your Pages site will use the layout and styles from the Jekyll theme you have selected in your [repository settings](https://github.com/stegotiff/docs/settings). The name of this theme is saved in the Jekyll `_config.yml` configuration file.

### Support or Contact

Having trouble with Pages? Check out our [documentation](https://help.github.com/categories/github-pages-basics/) or [contact support](https://github.com/contact) and we’ll help you sort it out.

repository: stegotiff/docs

repo_name: stegotiff/docs


text:
# PollProxy

## Concept

PollProxy allows many actors to subscribe to the result of a repeating polling operation.
A typical use case is monitoring the change of state of an external resource (DB, API, file...).
The behaviour is defined by implementing `PollProxy.Poller`.

Here is an example where we want to poll the numbers starting from 1 every 100 ms to find increasingly large numbers devisible by a given number. So if we start the poller with a modulus of 3 it should give us `[3, 6, 9, 12 ...]`.

```elixir
defmodule PollProxy.Example.ModPoller do
  @behaviour PollProxy.Poller
  alias PollProxy.Example.ModPoller
  defstruct [:modulus, :i]

  def interval, do: 100

  def init(modulus) do
    {:ok, %ModPoller{modulus: modulus, i: 0}}
  end

  def poll(%ModPoller{modulus: modulus, i: last_i} = state) do
    i = last_i + 1
    next_state = %ModPoller{state | i: i}
    case rem(i, modulus) do
      0 -> {:update, i, next_state}
      _ -> {:noupdate, next_state}
    end
  end

  def handle_update(update_data, subscriber, _state) do
    Process.send(subscriber, {:update, update_data}, [])
    :ok
  end

end
```

So `init/n` is called once when the proxy is started and computes the intial state. While polling, `poll/1` is called repeatedly, first with the inial state and later with the states returned by earlies invocations.

If polling discovers an update `poll/1` returns `{:update, update_data, state}` and the proxy will call `handle_update/3` for all subscribers. This call will happen in a spawned process so it will not block the proxy.

## starting and subscribing/unsubscribing
PollProxy is its own application so the proxies are supervised.

### starting manually

```elixir
iex(1)> PollProxy.start_proxy(PollProxy.Example.ModPoller, [7], :mod7_poller)
{:ok, #PID<0.115.0>}
iex(2)> PollProxy.subscribe(:mod7_poller)
:ok
[... a few moments pass ...]
iex(3)> flush
{:update, 7}
{:update, 14}
{:update, 21}
{:update, 28}
{:update, 35}
{:update, 42}
:ok
iex(3)> PollProxy.unsubscribe(:mod7_poller)
:ok
iex(4)> Process.whereis(:mod7_poller)
#PID<0.115.0>
```
### starting lazily/dynamically

If the intial arguments are not known beforehand you can also start a proxy and subscribe in one operation.
The name for the proxy is then computed from the poll_module and the poll_args and does not have to be passed around. When a different process wants to subscribe to a proxy with this poll_module/poll_args signature it subscribe to the running proxy.

Addtionaly the proxie workers are started with `stop_when_empty` enabled, meaning they are not only lazily started but also shut themselves down after the last process unsubscribes.

```elixir
iex(1)> {:ok, server}  = PollProxy.start_proxy_and_subscribe(PollProxy.Example.ModPoller, [7], self())
{:ok, :poll_proxy_1736236754}
[... a few moments pass ...]
iex(2)> flush
{:update, 7}
{:update, 14}
{:update, 21}
{:update, 28}
{:update, 35}
{:update, 42}
:ok
iex(3)> PollProxy.unsubscribe(server)
:ok
iex(4)> Process.whereis(server)
nil
```

### subscribing with synchronization

Sometimes the subscriptions has to be aligned with events that happend before. Like handing over a stream client from a catch-up
situation to a subscription. This can be done with a synchronization function which should return true for the matching last state of the poll proxy.

In this example the poller is going through multiples of 10 and we are waiting for a sync after it passed 20:

```elixir
iex(1)>  {:ok, server}  = PollProxy.start_proxy_and_subscribe(PollProxy.Example.ModPoller, [10], self(), synconize: fn(i) -> i == 20 end)
** (MatchError) no match of right hand side value: {:error, :not_synced, nil}
    (poll_proxy) lib/poll_proxy.ex:23: PollProxy.start_proxy_and_subscribe/4
iex(1)>  {:ok, server}  = PollProxy.start_proxy_and_subscribe(PollProxy.Example.ModPoller, [10], self(), synconize: fn(i) -> i == 20 end)
** (MatchError) no match of right hand side value: {:error, :not_synced, 10}
    (poll_proxy) lib/poll_proxy.ex:23: PollProxy.start_proxy_and_subscribe/4
iex(1)>  {:ok, server}  = PollProxy.start_proxy_and_subscribe(PollProxy.Example.ModPoller, [10], self(), synconize: fn(i) -> i == 20 end)
{:ok, :poll_proxy_432906863}
```


text:
# OpenZeppelin Solidity
[![NPM Package](https://img.shields.io/npm/v/openzeppelin-solidity.svg?style=flat-square)](https://www.npmjs.org/package/openzeppelin-solidity)
[![Build Status](https://img.shields.io/travis/OpenZeppelin/openzeppelin-solidity.svg?branch=master&style=flat-square)](https://travis-ci.org/OpenZeppelin/openzeppelin-solidity)
[![Coverage Status](https://img.shields.io/coveralls/github/OpenZeppelin/openzeppelin-solidity/master.svg?style=flat-square)](https://coveralls.io/github/OpenZeppelin/openzeppelin-solidity?branch=master)

OpenZeppelin is a library for writing secure [Smart Contracts](https://en.wikipedia.org/wiki/Smart_contract) on Ethereum.

With OpenZeppelin, you can build distributed applications, protocols and organizations:
- using common contract security patterns (See [Onward with Ethereum Smart Contract Security](https://medium.com/bitcorps-blog/onward-with-ethereum-smart-contract-security-97a827e47702#.y3kvdetbz))
- in the [Solidity language](https://solidity.readthedocs.io/en/develop/).

> NOTE: New to smart contract development? Check our [introductory guide](https://medium.com/zeppelin-blog/the-hitchhikers-guide-to-smart-contracts-in-ethereum-848f08001f05#.cox40d2ut).

## Getting Started

OpenZeppelin integrates with [Truffle](https://github.com/ConsenSys/truffle) and [Embark](https://github.com/embark-framework/embark/).

## truffle

To use with Truffle, first install it and initialize your project with `truffle init`.

```sh
npm install -g truffle
mkdir myproject && cd myproject
truffle init
```

## Embark

To use with Embark, first install it and initialize your project with `embark new MyApp`.

```sh
npm install -g embark
embark new MyApp
cd MyApp
```

## Installing OpenZeppelin

After installing either Framework, to install the OpenZeppelin library, run the following in your Solidity project root directory:

```sh
npm init -y
npm install -E openzeppelin-solidity
```

**Note that OpenZeppelin does not currently follow semantic versioning.** You may encounter breaking changes upon a minor version bump. We recommend pinning the version of OpenZeppelin you use, as done by the `-E` (`--save-exact`) option.

After that, you'll get all the library's contracts in the `node_modules/openzeppelin-solidity/contracts` folder. You can use the contracts in the library like so:

```solidity
import 'openzeppelin-solidity/contracts/ownership/Ownable.sol';

contract MyContract is Ownable {
  ...
}
```

If you are using Embark, you can also import directly from github:

```solidity
import "github.com/OpenZeppelin/openzeppelin-solidity/contracts/ownership/Ownable.sol#v1.9.0";

contract MyContract is Ownable {
  ...
}
```

## Architecture
The following provides visibility into how OpenZeppelin's contracts are organized:

- **access** - Smart contracts that enable functionality that can be used for selective restrictions and basic authorization control functions. Includes address whitelisting and signature-based permissions management.
	- **rbac** - A library used to manage addresses assigned to different user roles and an example Role-Based Access Control (RBAC) interface that demonstrates how to handle setters and getters for roles and addresses.
- **crowdsale** - A collection of smart contracts used to manage token crowdsales that allow investors to purchase tokens with ETH. Includes a base contract which implements fundamental crowdsale functionality in its simplest form. The base contract can be extended in order to satisfy your crowdsale’s specific requirements.
	- **distribution** - Includes extensions of the base crowdsale contract which can be used to customize the completion of a crowdsale.
	- **emission** - Includes extensions of the base crowdsale contract which can be used to mint and manage how tokens are issued to purchasers.
	- **price** - Includes extensions of the crowdsale contract that can be used to manage changes in token prices.
	- **validation**  - Includes extensions of the crowdsale contract that can be used to enforce restraints and limit access to token purchases.
- **examples** - A collection of simple smart contracts that demonstrate how to add new features to base contracts through multiple inheritance.
- **introspection**  - An interface that can be used to make a contract comply with the ERC-165 standard as well as a contract that implements ERC-165 using a lookup table.
- **lifecycle** - A collection of base contracts used to manage the existence and behavior of your contracts and their funds.
- **math** - Libraries with safety checks on operations that throw on errors.
- **mocks** - A collection of abstract contracts that are primarily used for unit testing. They also serve as good usage examples and demonstrate how to combine contracts with inheritence when developing your own custom applciations.
- **ownership** - A collection of smart contracts that can be used to manage contract and token ownership
- **payment** - A collection of smart contracts that can be used to manage payments through escrow arrangements, withdrawals, and claims. Includes support for both single payees and multiple payees.
- **proposals** - A collection of smart contracts that reflect community Ethereum Improvement Proposals (EIPs). These contracts are under development and standardization. They are not recommended for production, but they are useful for experimentation with pending EIP standards. Go [here](https://github.com/OpenZeppelin/openzeppelin-solidity/wiki/ERC-Process) for more information.

- **token** - A collection of approved ERC standard tokens -- their interfaces and implementations.
	- **ERC20** - A standard interface for fungible tokens:
		- *Interfaces* - Includes the ERC-20 token standard basic interface. I.e., what the contract’s ABI can represent.
		- *Implementations* - Includes ERC-20 token implementations that include all required and some optional ERC-20 functionality.
	- **ERC721** - A standard interface for non-fungible tokens
		- *Interfaces* - Includes the ERC-721 token standard basic interface. I.e., what the contract’s ABI can represent.
		- *Implementations* - Includes ERC-721 token implementations that include all required and some optional ERC-721 functionality.

## Tests
Unit test are critical to the OpenZeppelin framework. They help ensure code quality and mitigate against security vulnerabilities. The directory structure within the `/tests` directory corresponds to the `/contracts` directory. OpenZeppelin uses Mocha’s JavaScript testing framework and Chai’s assertion library. To learn more about how to tests are structured, please reference OpenZeppelin’s Testing Guide.

## How To Use And Modify OpenZeppelin Contracts
When using OpenZeppelin to build your own distributed applications, for security reasons we encourage you NOT to modify the framework’s base contracts, libraries, and interfaces. In order to leverage and extend their functionality, we encourage you to inherit from them or compose them together with your own contracts.

The Solidity programming language supports multiple inheritance. This is very powerful yet it can also be confusing: the more complexity you introduce to your distributed applications through multiple inheritance, the greater your application’s attack surface is.

You’ll notice in the `/mocks` directory there are a collection of abstract contracts used primarily for unit testing purposes that can also be used as the foundation for your own custom implementations. These mock contracts demonstrate how OpenZeppelin’s secure base contracts can be used with multiple inheritance.

To learn more about combining OpenZeppelin contracts with your own custom contracts using multiple inheritance we encourage you to read the following: [On crowdsales and multiple inheritance](https://blog.zeppelin.solutions/on-crowdsales-and-multiple-inheritance-af90c694e35f)

## Security
OpenZeppelin is meant to provide secure, tested and community-audited code, but please use common sense when doing anything that deals with real money! We take no responsibility for your implementation decisions and any security problem you might experience.

The core development principles and strategies that OpenZeppelin is based on include: security in depth, simple and modular code, clarity-driven naming conventions, comprehensive unit testing, pre-and-post-condition sanity checks, code consistency, and regular audits.

If you find a security issue, please email [security@openzeppelin.org](mailto:security@openzeppelin.org).

## Developer Resources

Building a distributed application, protocol or organization with OpenZeppelin?

- Read documentation: https://openzeppelin.org/api/docs/open-zeppelin.html

- Ask for help and follow progress at: https://slack.openzeppelin.org/

Interested in contributing to OpenZeppelin?

- Framework proposal and roadmap: https://medium.com/zeppelin-blog/zeppelin-framework-proposal-and-development-roadmap-fdfa9a3a32ab#.iain47pak
- Issue tracker: https://github.com/OpenZeppelin/openzeppelin-solidity/issues
- Contribution guidelines: https://github.com/OpenZeppelin/openzeppelin-solidity/blob/master/CONTRIBUTING.md
- Wiki: https://github.com/OpenZeppelin/openzeppelin-solidity/wiki

## License
Code released under the [MIT License](https://github.com/OpenZeppelin/openzeppelin-solidity/blob/master/LICENSE).

gha_language: Solidity

gha_license_id: MIT


text:
# dgc-lib
Deep gradient compression lib.
## Introduction
Base on deep gradient compression paper.
[DGC paper](https://arxiv.org/pdf/1712.01887.pdf).
## Requirements
dgc-lib requires CUDA and NCCL head.
## Build
To build the static library :
```shell
$ make -j src.staticlib
# or default
$ make -j
```
To build the shared library :
```shell
$ make -j src.lib
```
If CUDA or NCCL is not installed in the default path, you can build with :
```shell
$ make CUDA_HOME=<path to cuda install> NCCL_INCLUDE=<path to nccl include>
```
dgc-lib will be compiled in `build/` unless `BUILD_PREFIX` is set.
## Install
```shell
$ make install
```
dgc-lib will be installed in `install/` unless `INSTALL_PREFIX` is set.
## Tests
dgc-test requires CUDA、NCCL and MPI.
```shell
$ make -j test
```
If CUDA 、NCCL or MPI is not installed in the default path, you can build with :
```shell
$ make -j test CUDA_HOME=<path to cuda install> NCCL_HOME=<path to nccl install> MPI_HOME=<path to mpi install>
```
Run with MPI on 8 processes, each process use one GPU:
```shell
mpirun -np 8 ./build/bin/dgc_test 1048576 1024
```

repository: seiriosPlus/Fleet

branch_name: refs/heads/develop

repo_name: seiriosPlus/Fleet


text:
![](https://img.shields.io/github/license/laplacetw/botlegram) [![Documentation Status](https://readthedocs.org/projects/botlegram/badge/?version=latest)](https://botlegram.readthedocs.io/en/latest/?badge=latest) [![Build Status](https://travis-ci.org/laplacetw/botlegram.svg?branch=master)](https://travis-ci.org/laplacetw/botlegram)

# Botlegram
A Telegram Bot Package for Python.

- Release v0.0.1
- Based on [Telegram Bot API](https://core.telegram.org/bots/api).
- Stay flexible and keep function naming consistent with the API.

It’s good to use. 🙆♂

### Quick Start

    import botlegram

    BOT_TOKEN = "{{your_bot_token}}"
    WEBHOOK = "{{your_host}}" + BOT_TOKEN + '/'
    bot = botlegram.Bot(BOT_TOKEN, WEBHOOK)
    >>> * [Bot] Webhook is already set

Now we have a Telegram bot 🤖

### Try it on Heroku
- [echo Bot](https://github.com/laplacetw/botlegram/files/5273117/echoBot.zip)

### Read The Docs

[https://botlegram.readthedocs.io/](https://botlegram.readthedocs.io/)

repository: laplacetw/botlegram-doc

repo_name: laplacetw/botlegram-doc


text:
# Screenshot Utility
Unity asset package that allows you to take screenshots, speed up and slow down time, and hide HUD.

![Screenshot](Images/cap.PNG)

## Usage
- Download the latest [release](https://github.com/thomasnakagawa/UnityScreenshotUtility/releases)
- Import the package, the example folder is optional.
- Place the ScreenshotUtility prefab in your scene.
- Run your scene and press F12.
- Find your screenshot in projectRoot/Screenshots
- Use the inspector to change the file name, path and shortcut keys

repository: thomasnakagawa/UnityScreenshotUtility

repo_name: thomasnakagawa/UnityScreenshotUtility


text:
# Discriminator-Actor-Critic
Note that all commands must be run from with the `dac` folder.
## Setup
1. Setup a Python 2 environment. (Commands for virtualenv given below)

	```
    pip install virtualenv
    virtualenv . -p $PYTHON2_PATH
    source bin/activate
	```
Note: You can use the command `which python2` to find the PYTHON2_PATH.

2. Installation (NOTE: need mujoco 1.31 setup -> download the zip from https://www.roboti.us/ and put the unzipped mjpro131 folder in the ~/.mujoco folder used for the mujoco license for gym-v1):

	```
    pip install -r requirements.txt
    ```

## Generate expert trajectories
DAC requires the expert trajectories to already exist. We use the OpenAI imitation repo to generate trajectories. (This is the original GAIL repo).

    git clone https://github.com/openai/imitation
    cd imitation
    python -m scripts/im_pipeline pipelines/im_pipeline.yaml 0_sampletrajs
    cp imitation_runs/modern_stochastic/trajs/* ../trajs
## Run DAC

    python dac.py --env_id=$environment_name --expert_path=$path/to/expert/traj
Example (Ant-v1):

    python dac.py --env_id=Ant-v1 --expert_path=trajs/trajs_ant.h5

The environment name can be `Hopper-v1`, `HalfCheetah-v1`, `Ant-v1`, or `Walker2d-v1`

The path to the expert trajectory should be a path to the correspond file in the `trajs` folder.

repository: vluzko/dac-iclr-reproducibility

repo_name: vluzko/dac-iclr-reproducibility


text:
# HEVENS webpack boilerplate

This is standard webpack configuration for frontend development that we use.

If `yarn` confuses you, use `npm` instead of it, but one day the developer community will move to yarn because it is
faster and safer that much NPM can never be by its design. The packages base remain the same. Refer to [yarn website](https://yarnpkg.com/).

## Features

- pug.js for templating
- SCSS for styling
- ES2015 for scripting
- cache busting with main.css?hash or hash in font, video and image names
- source maps
- auto images generation with configurable quality
- auto srcset generation for given image

## Usage

### I'm a frontend developer

1. `yarn run start`
2. Navigate your browser to `localhost:8080` or other port if few webpack-dev-servers are running
3. Make changes
4. `yarn run build`
5. Perform VCS commit

### I'm a backend developer

1. Make changes
2. `yarn run build`
3. In your php templates, replace names of assets with updated ones that contain new hash indicating build date, e.g. `build/assets/main-1500369947463.min.css`. This is done for reliable cache busting.
4. Perform VCS commit

### If you use your brain

Open `package.json` and see what commands are `scripts` field. You may use same commands or even cooler. Visit [webpack docs](https://webpack.js.org/concepts/) or [russian video course on Webpack](https://www.youtube.com/watch?v=X6qde-zvw00)