text:
---
layout: post
title: "010: Training the Digital Generation"
subtitle: with Scott Vincent
tags: [students, learning, entrepeneurship]
comments: true
---

Scott Vincent is the CEO and founder of [Digital Futures](https://digitalfutures.com/), a platform for graduates looking to enter the digital economy. Prior to starting this, Scott was recognised as one the U.K.'s Top 50 Entrepreneurs by The Telegraph for his first venture, Parker Fitzgerald, a risk management consultancy he started in 2009. With clients such as UBS, Credit Suisse and Citi Group, they were subsequently acquired by Accenture. After taking some time out to reconnect with his family, we caught up with Scott to learn more about his latest venture, Digital Futures.

Applications to join the 2021 Digital Futures Graduate Academy are now open. You can register on their [website](https://digitalfutures.com/).

<iframe src="https://anchor.fm/herethefuture/embed/episodes/010-Training-the-Digital-Generation---Scott-Vincent-en8red" height="102px" width="400px" frameborder="0" scrolling="no"></iframe>


text:
template
========

This is my template project. Lately, most of my C projects look the same. So,
to save me from some serious work (setting up XCode projects is a nightmare)
I made this little template and supplied a script to do all the neccessary
substitutions.

Usage
-----

### Initial setup:

Just clone the template project and call the `subst.sh` script. The script
takes a single parameter: The name for your new project. It uses `git config`
to determine your name and email, and inserts the current year into all license
templates (which are BSD, by the way).

    git clone git@github.com:jpommerening/template.git myproject
    cd myproject
    ./subst.sh myproject
    git commit

### Pulling updates:

After doint it's substitutions and renaming, the script commits all modified files
and renames `origin` to `template`.

So, assuming you´re on the `master` branch, if you want to pull updates (in many
cases this might actually work, because `git` is awesome) just use the following
command:

    git pull template master

License
-------

Copyright © @YEAR@, @NAME@ <@EMAIL@>
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

1.  Redistributions of source code must retain the above copyright notice, this
    list of conditions and the following disclaimer.
2.  Redistributions in binary form must reproduce the above copyright notice,
    this list of conditions and the following disclaimer in the documentation
    and/or other materials provided with the distribution.

repository: jpommerening/template

branch_name: refs/heads/master

extension: md

license_type: permissive

repo_name: jpommerening/template


text:
The goal is to remove the maximum of highly used and highly duplicated low-level
code from the spark job code and replace it with methods fully tested whose
names are self-explanatory and readable.

This also provides a monitoring/logger tool.

* [SparkHelper](http://xavierguihot.com/spark_helper/#com.spark_helper.SparkHelper$): Hdfs file manipulations through the Spark API.
* [DateHelper](http://xavierguihot.com/spark_helper/#com.spark_helper.DateHelper$): Wrapper around [joda-time](http://www.joda.org/joda-time/apidocs/) for usual data mining dates manipulations.
* [Monitor](http://xavierguihot.com/spark_helper/#com.spark_helper.Monitor$): Spark custom monitoring/logger and kpi validator.

Compatible with Spark 2.

### HdfsHelper:

The full list of methods is available at
[HdfsHelper](http://xavierguihot.com/spark_helper/#com.spark_helper.HdfsHelper$).

Contains basic file-related methods mostly based on hdfs apache Hadoop
FileSystem API [org.apache.hadoop.fs.FileSystem](https://hadoop.apache.org/docs/r2.6.1/api/org/apache/hadoop/fs/FileSystem.html).

For instance, one don't want to remove a file from hdfs using 3 lines of code
and thus could instead just use `HdfsHelper.deleteFile("my/hdfs/file/path.csv")`.

A non-exhaustive list of exemples:

```scala
import com.spark_helper.HdfsHelper

// A bunch of methods wrapping the FileSystem API, such as:
HdfsHelper.fileExists("my/hdfs/file/path.txt")
assert(HdfsHelper.listFileNamesInFolder("my/folder/path") == List("file_name_1.txt", "file_name_2.csv"))
assert(HdfsHelper.fileModificationDate("my/hdfs/file/path.txt") == "20170306")
assert(HdfsHelper.nbrOfDaysSinceFileWasLastModified("my/hdfs/file/path.txt") == 3)
HdfsHelper.deleteFile("my/hdfs/file/path.csv")
HdfsHelper.moveFolder("my/hdfs/folder")
HdfsHelper.compressFile("hdfs/path/to/uncompressed_file.txt", classOf[GzipCodec])
HdfsHelper.appendHeader("my/hdfs/file/path.csv", "colum0,column1")

// Some Xml/Typesafe helpers for hadoop as well:
HdfsHelper.isHdfsXmlCompliantWithXsd("my/hdfs/file/path.xml", getClass.getResource("/some_xml.xsd"))
HdfsHelper.loadXmlFileFromHdfs("my/hdfs/file/path.xml")

// Very handy to load a config (typesafe format) stored on hdfs at the begining of a spark job:
HdfsHelper.loadTypesafeConfigFromHdfs("my/hdfs/file/path.conf"): Config

// In order to write small amount of data in a file on hdfs without the whole spark stack:
HdfsHelper.writeToHdfsFile(Array("some", "relatively small", "text"), "/some/hdfs/file/path.txt")

// Deletes all files/folders in "hdfs/path/to/folder" for which the timestamp is older than 10 days:
HdfsHelper.purgeFolder("hdfs/path/to/folder", 10)
```

### SparkHelper:

The full list of methods is available at
[SparkHelper](http://xavierguihot.com/spark_helper/#com.spark_helper.SparkHelper$).

Contains basic file/RRD-related methods based on the Spark APIs.

A non-exhaustive list of exemples:

```scala
import com.spark_helper.SparkHelper

// Same as SparkContext.saveAsTextFile, but the result is a single file:
SparkHelper.saveAsSingleTextFile(myOutputRDD, "/my/output/file/path.txt")

// Same as SparkContext.textFile, but instead of reading one record per line,
// it reads records spread over several lines. This way, xml, json, yml or
// any multi-line record file format can be used with Spark:
SparkHelper.textFileWithDelimiter("/my/input/folder/path", sparkContext, "---\n")

// Equivalent to sparkContext.textFile(), but for each line is tupled with its
// file path:
SparkHelper.textFileWithFileName("folder", sparkContext)
// which produces:
RDD(
    ("file:/path/on/machine/folder/file_1.txt", "record1fromfile1"),
    ("file:/path/on/machine/folder/file_1.txt", "record2fromfile1"),
    ("file:/path/on/machine/folder/file_2.txt", "record1fromfile2"),
    ...
)
```

### DateHelper:

The full list of methods is available at
[DateHelper](http://xavierguihot.com/spark_helper/#com.spark_helper.DateHelper$).

Wrapper around [joda-time](http://www.joda.org/joda-time/apidocs/) for
data-mining classic dates manipulations.

A non-exhaustive list of exemples:

```scala
import com.spark_helper.DateHelper

assert(DateHelper.daysBetween("20161230", "20170101") == List("20161230", "20161231", "20170101"))
assert(DateHelper.today() == "20170310") // If today's "20170310"
assert(DateHelper.yesterday() == "20170309") // If today's "20170310"
assert(DateHelper.reformatDate("20170327", "yyyyMMdd", "yyMMdd") == "170327")
assert(DateHelper.now("HH:mm") == "10:24")
assert(DateHelper.currentTimestamp() == "1493105229736")
assert(DateHelper.nDaysBefore(3) == "20170307") // If today's "20170310"
assert(DateHelper.nDaysAfterDate(3, "20170307") == "20170310")
```

### Monitor:

The full list of methods is available at
[Monitor](http://xavierguihot.com/spark_helper/#com.spark_helper.Monitor$)

It's a simple logger/report which contains a report that one can update from
the driver and a success state. The idea is to persist job executions logs and
errors (and forget about grepping unreadable yarn logs).

It's designed for perdiodic spark jobs (handles storage and purge of logs) and
provides a way to handle kpis validation.

Logs are stored on the go which means one can have a direct real time access of
the job logs/status and it's current state (which can overwise be a pain if it
means going through yarn logs, or even for certain production environments going
through additional layers of software logs to get to yarn logs).

One of the issues this logger aims at tackling is the handling of exceptions
reported from executors. An exception within a Spark pipeline doesn't always
mean one want to make the job fail. Or if it's the case, one might still want to
perform a few actions before letting the job crash. The idea is thus to surround
(driver side) a Spark pipeline within a try catch and redirect the exception to
the logger for a clean logging.

This is a "driver-only" logger and is not intended at logging concurrent actions
from executors.

Produced reports can easily be inserted in a notification email whenerver the
job fails, which saves a lot of time to maintainers operating on heavy
production environements.

The produced persisted report is also a way for downstream jobs to know the
status of their input data.

```scala
import com.spark_helper.Monitor

Monitor.setTitle("My job title")
Monitor.addDescription(
  "My job description (whatever you want); for instance:\n" +
  "Documentation: https://github.com/xavierguihot/spark_helper")
Monitor.setLogFolder("path/to/log/folder")

try {

  // Let's perform a spark pipeline which might go wrong:
  val processedData = sc.textFile("file.json").map(/**whatever*/)

  // Let's say you want to get some KPIs on your output before storing it:
  val outputIsValid = Monitor.kpis(
    List(
      Test("Nbr of output records", processedData.count(), SUPERIOR_THAN, 10e6d, NBR),
      Test("Some pct of invalid output", your_complex_kpi, INFERIOR_THAN, 3, PCT)
    ),
    "My pipeline descirption"
  )

  if (outputIsValid)
    processedData.saveAsTextFile("wherever.csv")

} catch {
  case iie: InvalidInputException =>
    Monitor.error(iie, "My pipeline descirption", diagnostic = "No input data!")
  case e: Throwable =>
    Monitor.error(e, "My pipeline descirption") // whatever unexpected error
}

if (Monitor.isSuccess()) {
  val doMore = "Let's do some more stuff!"
  Monitor.log("My second pipeline description: success")
}

// At the end of the different steps of the job, we can store the report in
// HDFS (this saves the logs in the folder set with Monitor.setLogFolder):
Monitor.store()

// At the end of the job, if the job isn't successfull, you might want to
// crash it (for instance to get a notification from your scheduler):
if (!Monitor.isSuccess()) throw new Exception() // or send an email, or ...
```

At any time during the job, logs can be accessed from file
`path/to/log/folder/current.ongoing`

Here are some possible reports generated by the previous pipeline:

```
           My job title

My job description (whatever you want); for instance:
Documentation: https://github.com/xavierguihot/spark_helper
[10:23] Begining
[10:23-10:23] My pipeline descirption: failed
  Diagnostic: No input data!
    org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://my/hdfs/input/path
    at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:285)
    at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:228)
    ...
[10:23] Duration: 00:00:00
```

or

```
           My job title

My job description (whatever you want); for instance:
Documentation: https://github.com/xavierguihot/spark_helper
[10:23] Begining
[10:23-10:41] My pipeline descirption: success
  KPI: Nbr of output records
    Value: 14669071.0
    Must be superior than 10000000.0
    Validated: true
  KPI: Some pct of invalid output
    Value: 0.06%
    Must be inferior than 3.0%
    Validated: true
[10:41-10:42] My second pipeline description: success
[10:42] Duration: 00:19:23
```

## Including spark_helper to your dependencies:

With sbt, add these lines to your build.sbt:

```scala
resolvers += "jitpack" at "https://jitpack.io"

libraryDependencies += "com.github.xavierguihot" % "spark_helper" % "v1.1.1"
```

With maven, add these lines to your pom.xml:

```xml
<repositories>
	<repository>
		<id>jitpack.io</id>
		<url>https://jitpack.io</url>
	</repository>
</repositories>

<dependency>
	<groupId>com.github.xavierguihot</groupId>
	<artifactId>spark_helper</artifactId>
	<version>v1.1.1</version>
</dependency>
```

With gradle, add these lines to your build.gradle:

```groovy
allprojects {
	repositories {
		maven { url 'https://jitpack.io' }
	}
}

dependencies {
	compile 'com.github.xavierguihot:spark_helper:v1.1.1'
}
```


text:
# ULS SIMPLE Example

This example provides demo usage example for ULS within a `docker compose` setup.
It shows a single configured feed for visibility.
You might want to look over to a [real world example.](../complex/README.md)
All docker related controls can be found in [docker-compose.yml](docker-compose.yml)

## ETP-THREAT
|IN|OUT|FILE|
|---|---|---|
|ETP-THREAT|TCP|[etp-threat.env](etp-threat.env)|

repository: pombredanne/uls

branch_name: refs/heads/main

repo_name: pombredanne/uls


text:
# Students Performance Data
---

# Table of contents
1. Motivation
2. Installation
3. Files
4. Results
5. Acknowledgement

# Motivation
In this project we will be taking a look at the performance of students in a particular school.
These are some of the questions i tried to answer,

- What is the parental level of education of the first top 5 females with 100% score in reading.

- What does the data suggest about race and ethnicity and their score.

- Did students who completed the test preparation course do well in all subjects?

---

# Installation
The notebook uses,
Data Analysis
- Pandas
- Numpy

# Data Visualization
- Matplotlib
- Seaborn

The python library was used for this notebook.

---

# Files
The files in the repository answers all the questions asked in the data analysis.

---

# Results
The main findings can be found  [here](https://medium.com/p/f64c60a20624/edit)

---

# Acknowledgement
Credit must be given to [Kaggle](https://www.kaggle.com/)  for the data.

license: MIT

repository: NaaOyoe/Students-Performance

repo_name: NaaOyoe/Students-Performance


text:
---
hrs_structure:
  division: '1'
  volume: '3'
  title: '12'
  chapter: '182'
  section: 182-17
type: hrs_section
tags:
  - Government
menu:
  hrs:
    identifier: HRS_0182-0017
    parent: HRS0182
    name: 182-17 Penalty for violation
weight: 83090
title: Penalty for violation
full_title: 182-17 Penalty for violation
---
**[§182-17] Penalty for violation.** Any person who violates any provision of this chapter, or any regulation adopted pursuant hereto, shall be fined not more than $500 for each offense. If any person after receiving written notice for a violation fails to cure such violation within such time and under such conditions as determined by the rules and regulations, such person shall be subject to a citation for a new and separate violation. There shall be a fine not more than $500 for each additional violation. [L 1978, c 135, §9]

repository: SamMade/BlueAndKrabby

repo_name: SamMade/BlueAndKrabby


text:
# TensorflowLite4Unity

## Project Description
TFLite models are normally not supported by the ml-agents framework. This causes the problem that quantized models can not be currently run by the Unity Inference Engine whereas performance drawbacks are inevitable during the runtime. Currently, tflite models' performances are backed by the GPU delegates which have been implemented in the mean time and are being developed further. Especially, with the development of frameworks like MediaPipe, Unity could need more power on the Android side due to the unsupport of the TFLite models. This plugin helps to integrate TFLite inference into the Unity Engine and also to the R&D teams for preparing quick prototypes as the Unity Engine has an easy environment to accomplish this purpose.

## Roadmap
1.  Simple TFLite interpreter integration(in development).
2.  Reusable plug&play API on the C# side to reduce development time without rewriting the same components again(in development).
3.  Easy integration into the AR Foundations environment to use computer vision neural nets for several tasks for AR scenes(in development).
4.  Performance optimizations.

## Project Structure

[TFARInterface](tflite4unity/TFARInterface) : A Unity interface to use the tflite plugin.

[TFLite Android](tflite4unity/TFARInterface/Assets/TFLiteAndroidProject) : A plugin to support tflite models on the Android side.
TODO: iOS plugin will be added.

## Contributions
Contributions are welcomed and encouraged!

1.  TFLite plugin will be ported to C++ for better performance.
2.  You can add your .tflite models into the /assets folder to share them for reuse(pull request with model details is encouraged!).

## Acknowledgments

Thanks to the [MediaPipe Team](https://github.com/google/mediapipe) for having published their useful models for essential AR interactions!

## Tech Stack

1.  Tensorflow Lite 2.0.0
2.  Unity 2019.2(recommended)
3.  Android Studio


text:
Usage of ET in Hall A C++/Root Analyzer
=======================================

I. GENERAL REMARKS
------------------

The Event Transfer (ET) system was invented by the Jlab DAQ
group for obtaining data online, and for inserting data into the
datastream.  It is documented at:
coda.jlab.org/coda/Users_Guide/2_1_guide/API/ET/3.1/table_of_contents.htm

The CODA classes written by me are "wrappers" around the ET
library (in the case of THaEtClient) and around the EVIO library
(in the case of THaCodaFile).  These two derive from abstract
class THaCodaData.  The main advantages of this scheme is
to allow a common interface bwteen files and ET, and to
provide a simple, safe usage (e.g. users cannot cause deadtime).
More info is at hallaweb.jlab.org/equipment/daq/codaclass.html

The example listed in the above URL, which is also in tstcoda_main.C
in the hana_decode directory, makes it clear how to use the
interface.  The codaclasses are available separately, and
compile on SunOS with CC compiler as well as Linux (RedHat 6.2)
with g++ compiler.

II. SPECIFIC COMPILATION INSTRUCTIONS
-------------------------------------

Follow these steps to implement the CODA interface.
There is more than one way to do it, but here is a simple way.
The code referred to here is presently in
haplix7:/home/rom/haAna/ana_0.5

 --------
| STEP 1 |  The main Makefile needs a flag ONLINE_ET turned on.
 --------   However, this should be off by default so that users
            and user institutions don't need ET library (they only
            read data files).
            Here are the diffs:

11a12,14
> # If compiling for ONLINE_ET analysis using ET system
> #export ONLINE_ET = 1
>
30c33
< DCDIR         = hana_decode_1.3
---
> DCDIR         = hana_decode_1.4
53a57,69
> # ONLIBS is needed for ET
> # CODA environment variable must be set.  Examples are
> #   CODA = /adaqfs/coda/2.2        (in adaq cluster)
> #   CODA = /data7/user/coda/2.2    (on haplix cluster)
>   CODA = /data7/user/coda/2.2
>   LIBET = $(CODA)/Linux/lib/libet.so
>   ONLIBS = $(LIBET) -lieee -lpthread -ldl -lresolv
>
>   ifdef ONLINE_ET
>      LIBS += $(ONLIBS)
>      GLIBS += $(ONLIBS)
>   endif
>
64a81,84
> #ifdef ONLINE_ET
> SRC += THaOnlRun.C
> #endif
>

 --------
| STEP 2 |  THaRun is modified to use abstract class THaCodaData.
 --------   Here are diffs:

[rom@haplix1 ana_0.5]$ diff THaRun.h THaRun_orig.h
23c23
< class THaCodaData;
---
> class THaCodaFile;
33c33
<   THaCodaData*  fCodaData;     //CODA data associated wih this run
---
>   THaCodaFile*  fCodaFile;     //CODA file associated wih this run

(In the .C file, we simply replace fCodaFile by fCodaData)
diff THaRun.C THaRun_orig.C
18d17
< #include "THaCodaData.h"
29c28
<   fCodaData = 0;
---
>   fCodaFile = 0;
39c38
<   fCodaData = new THaCodaFile;  //Do not open the file
---
>   fCodaFile = new THaCodaFile;  //Do not open the file
51c50
<   fCodaData   = new THaCodaFile;
---
>   fCodaFile   = new THaCodaFile;
66c65
<      fCodaData   = new THaCodaFile;
---
>      fCodaFile   = new THaCodaFile;
77c76
<   delete fCodaData;
---
>   delete fCodaFile;
84c83
<   return fCodaData ? fCodaData->codaClose() : 0;
---
>   return fCodaFile ? fCodaFile->codaClose() : 0;
120c119
<   // by fCodaData->codaRead()
---
>   // by fCodaFile->codaRead()
122c121
<   return fCodaData ? fCodaData->getEvBuffer() : 0;
---
>   return fCodaFile ? fCodaFile->getEvBuffer() : 0;
133,135c132,134
<   if( !fCodaData ) fCodaData = new THaCodaFile;
<   if( !fCodaData ) return -3;
<   return fCodaData->codaOpen( fFilename.Data() );
---
>   if( !fCodaFile ) fCodaFile = new THaCodaFile;
>   if( !fCodaFile ) return -3;
>   return fCodaFile->codaOpen( fFilename.Data() );
160c159
<   return fCodaData ? fCodaData->codaRead() : -1;
---
>   return fCodaFile ? fCodaFile->codaRead() : -1;

-------------------------------------------------------------

 --------
| STEP 3 |  A new class THaOnlRun can be created which inherits from
 --------   THaRun, and uses ET system instead of CODA file.  The use
            of word 'File' unfortunately remains, but this is neverthe-
            less a simple way to proceed.  Hopefully Ole will do something
            like this.  Below is the THaOnlRun class.  This code is
            also in haplix7:/home/rom/haAna/ana_0.5
            Of course, this must be added to HallA_LinkDef.h

THaOnlRun.h:
============

#ifndef ROOT_THaOnlRun
#define ROOT_THaOnlRun

//+SEQ,CopyRight,T=NOINCLUDE.

//////////////////////////////////////////////////////////////////////////
//
// THaOnlRun
//
// Description of an online run using ET system.
//
//////////////////////////////////////////////////////////////////////////

//*KEEP,TNamed.
#include "TNamed.h"
#include "THaRun.h"
//*KEEP,TDatime.
#include "TDatime.h"
//*KEND.

#include <limits.h>

class THaCodaData;

class THaOnlRun : public THaRun {

protected:
  TString  fComputer;   // computer where DAQ is running, e.g. 'adaql2'
  TString  fSession;    // SESSION = unique ID of DAQ, usually an env. var., e.g 'onla'
  UInt_t   fMode;       // mode (0=wait forever for data, 1=time out, recommend 1)
  UInt_t   fOpened;

public:
  THaOnlRun();
  THaOnlRun(const char* computer, const char* session, UInt_t mode);
  virtual  Int_t  OpenFile();
  virtual  Int_t  OpenFile( const char* computer, const char* session, UInt_t mode);

  ClassDef(THaOnlRun,0)   //Description of an online run using ET system
};

#endif

THaOnlRun.C
===========

//*-- Author :    Robert Michaels    March 2001

//////////////////////////////////////////////////////////////////////////
//
// THaOnlRun :: THaRun
//
// Description of an online run using ET system to get data in real time.
// Inheriting from THaRun, this is essentially a TNamed object with
// additional support for a run number, filename, CODA file IO,
// and run statistics.
//
//////////////////////////////////////////////////////////////////////////

//*KEEP,THaOnlRun.
#include "THaOnlRun.h"
//*KEND.

#include <iostream>
#include "THaCodaData.h"
#include "THaEtClient.h"
#include "string.h"

ClassImp(THaOnlRun)

//______________________________________________________________________________
THaOnlRun::THaOnlRun() : THaRun()
{
  // Default constructor

  fCodaData = new THaEtClient();
  fMode = 1;
  ClearEventRange();
}

//______________________________________________________________________________
THaOnlRun::THaOnlRun( const char* computer, const char* session, UInt_t mode) : THaRun(ses
sion), fComputer(computer), fSession(session), fMode(mode)
{
  // Normal constructor

  fCodaData = new THaEtClient();
  ClearEventRange();
}

//______________________________________________________________________________
Int_t THaOnlRun::OpenFile()
{
  // Open ET connection.

  if (fComputer.IsNull() || fSession.IsNull()) {
      return -2;   // must set computer and session, at least;
  }

  return fCodaData->codaOpen(fComputer, fSession, fMode);

}

//______________________________________________________________________________
Int_t THaOnlRun::OpenFile( const char* computer, const char* session, UInt_t mode )
{
// Set the computer name, session name, and mode, then open the 'file'.
// It isn't really a file.  It is an ET connection.

  fComputer = computer;
  fSession = session;
  fMode = mode;
  return OpenFile();
}

---------------------------------------------------

 --------
| STEP 4 |  Now one can use a 'setup' macro analogous to the
 --------   original 'setup.C' which Ole made.  Here is setup_onl.C
            in haplix7:/home/rom/haAna/ana_0.5

setup_onl.C
===========
{
//
//  Steering script for Hall A analyzer demo
//  This demo is similar to 'setup.C' secept that it uses
//  a THaOnlRun object to get data from ET system in real time.
//

// Set up the equipment to be analyzed.

THaApparatus* HRSE = new THaElectronHRS("Electron HRS");
gHaApps->Add( HRSE );

THaApparatus* BEAM = new THaBeam("Beamline");
gHaApps->Add( BEAM );

// Set up the event layout for the output file

THaEvent* event = new THaEvent;

// Set up the analyzer - we use the standard one,
// but this could be an experiment-specific one as well.

THaAnalyzer* analyzer = new THaAnalyzer;

// Define the Run to come from computer "adaqcp" (where CODA
// is running), using session "par1" (a unique session name of
// the DAQ system), and in mode 1 (the recommended "time-out" mode).

THaOnlRun* run = new THaOnlRun( "adaqcp", "par1", 1);

// Define the analysis parameters

analyzer->SetEvent( event );
analyzer->SetOutFile( "Afile.root" );
run->SetLastEvent( 10000 );

}

III. SPECIFIC RUN-TIME INSTRUCTIONS
===================================

Running ET, you need to know:

  1. What computer is running DAQ ?  E.g. "adaql2"
  2. What is the session ?  This is a unique identifiying tag of the DAQ.
     Usually $SESSION is an environment variable of the online accounts.
  3. What mode do you wish ?   Available are two modes:
     mode = 0  -->  Analyzer waits forever for data.
     mode = 1  -->  Analyzer times out after typ 30 seconds (e.g. if DAQ stops)

These parameters are passed to ET via the codaOpen method of THaEtClient.

In the "higher level" class THaOnlRun shown above, one can pass them via
parameters in the constructor; this only sets the parameters, it doesn't
try to codaOpen.
THaOnlRun* run = new THaOnlRun( "adaqcp", "par1", 1);

The call to codaOpen then gets done in THaAnalysis when it does this
Int_t status = run.OpenFile();

CODA environment variable must be set.  Examples are
   CODA = /adaqfs/coda/2.2        (in adaq cluster)
   CODA = /data7/user/coda/2.2    (on haplix cluster)
User's $LD_LIBRARY_PATH must include $CODA/$OSNAME/lib (OSNAME= SunOS or Linux)
Example:  OSNAME=Linux ; export OSNAME
LD_LIBRARY_PATH=/app/root/lib:$CODA/$OSNAME/lib:.   ; export LD_LIBRARY_PATH

A typical session in the analyzer may look like the following,
where the macros setup_onl.C and next_onl.C are listed here
and also, at present, in haplix7:/home/rom/haAna/ana_0.5

analyzer [0] .x setup_onl.C          // sets up ET system.
analyzer [1] analyzer->Process( *run )
Run 0                   // run is "zero" because you're in middle of run
ET rate 50.3 Hz in  6 sec, avg 50.3 Hz
ET rate 28.9 Hz in  7 sec, avg 46.0 Hz
1000
Event limit reached.
Processed 1000 events.

analyzer [2] evlen->Draw()

analyzer [3] .x next_onl.C         //  this gets more events.
number of events ?
4000
set out file
analyzer [4] analyzer->Process( *run )
ET rate  1.5 Hz in 103 sec, avg 34.9 Hz

(etc)

The histogram 'evlen' will automatically increment.

----------------------------------------

setup_onl.C was already listed above.

next_onl.C is similar but only redifines the run:

{
#include <iostream>
// next_onl.C :  Continue running....

int nevent;
cout << "number of events ? "<<endl;
cin >> nevent;

THaOnlRun* run = new THaOnlRun( "adaqs2", "tst1", 1);
analyzer->SetEvent( event );
analyzer->SetOutFile( "Afile.root" );
cout << "set out file  "<<endl<<flush;
run->SetLastEvent( nevent );

}

repository: JeffersonLab/analyzer

repo_name: JeffersonLab/analyzer


text:
# Google Cloud Composer

* Based on Apache Airflow
* It is a manged Apache Airflow service
* Create, schedule, monitor and manage workflows

## Documentation
* https://airflow.apache.org/
* https://airflow.apache.org/start.html
* https://cloud.google.com/composer/docs/

## What is Apache Airflow
* Open source
* programmatically author, schedule and montior workflows
* Based on python
* Setup non trivial (without using Cloud Composer)

## Composer / Airflow
Support of built-in services thise services are supported by **operators**.
* BigQuery
* GCS
* Pub/Sub
* Dataflow

Composer provides Operators to communicated with those services.

## DAG
* Directed Acyclic Graph, which means nothing else then we should not have cycles in our workflows.

## Compose Environment
* Each environment has its own Cloud Storage

Environments can be created via:
* Google Cloud SDK
* Google Cloud Platform Console
* Google Cloud Composer APIs

## Example
A typical workflow also called DAG would look like
* Look for new HDFS files
* Copy the data to Cloud Storage
* Looking for new data load them into bigquery
* Transform the data
* Extract information
* Notify slack

## Write a Workflow / Airflow DAG
A workflow is pythong file with 5 main sections

### Import
import dependencies needed for the workflow.
```
import datetime
from airflow import DAG
from airflow import models
```

### Arguments
construct default arguments argruments.
```
default_dag_args= {
    'retries':1
}

# other global variables which we want to use in our DAG
bg_dataset_name : 'datasetname'
output_file: 'gs://test/folder'
```
### Setup
Setup our DAG by giving it a name, setup a interval and give it our arguments
```
dag = DAG(
    'name',
    scheduler_interval = datetime.timedelta(days =1)
    default_args = default_dag_args
)
```

* schedule interval cannot be set in the default args
### Tasks
Tasks is the job itself which should run in the DAG.
Multiple tasks are possible. For example

1. Run a BigQuer query that eports the results to a new dataset
2. Export the new dataset to Cloud Storage

```
# perform a biquery operation
bigquery_operator.BigQueryOperator(
    task:id = 'big_query_query'
)

#export the query results
bigquery_to_gcs.BigQueryToCloudStorageOperator(
    task:id = 'export_to_gcs'
)
```

detailed description can be found here https://cloud.google.com/composer/docs/how-to/using/writing-dags#pythonoperator

### Ordering
define in which order the Tasks are executed. We want to run the query first before we save the results to gcs.
```
big_query_query >> export_to_gcs
```

## Security
* IAM support
* To access the airflow special permissions are required

## How to deploy a DAG
Each composer environment has a dedicated Google Cloud Storage DOG folder. The deployment is very simple we just upload the DAG to this DAGs folder.

We simply uplado the DAG File which is a .py file. After uploading it, it takaes a few minutes until Airflow picks up the DAG.

After that the DAG should be listed in the Airflow UI.

## Web UI
Google provies a link to the Airflow Web UI.

## PYPI
Packages can be installed via the composer UI.
Add packages and version directly in the Composer UI.

## cloud scheduler vs cloud composer

## Notes from the web
Now let's say your CSV file is overwritten at 01:00 UTC every day, and you want to run the same Dataflow job to process it every time when its overwritten. If you don't want to manually run the job exactly at 01:00 UTC regardless of weekends and holidays, you need a thing to periodically run the job for you (in our example, at 01:00 UTC every day). Cloud Composer can help you in this case. You can provide a config to Cloud Composer, which includes what jobs to run (operators), when to run (specify a job start time) and run in what frequency (can be daily, weekly or even yearly).

It seems cool already, however, what if the CSV file is overwritten not at 01:00 UTC, but anytime in a day, how will you choose the daily running time? Cloud Composer provides sensors, which can monitor a condition (in this case, the CSV file modification time). Cloud Composer can guarantee that it kicks off a job only if the condition is satisfied.

Google Cloud Composer is a big step up from Cloud Dataflow. Cloud Composer is a cross platform orchestration tool that supports AWS, Azure and GCP (and more) with management, scheduling and processing abilities.

Cloud Dataflow handles tasks. Cloud Composer manages entire processes coordinating tasks that may involve BigQuery, Dataflow, Dataproc, Storage, on-premises, etc.

repository: SaschaHeyer/Google-Cloud-Platform-Notes

path: /Big Data/Cloud Composer.md

filename: Cloud Composer.md

repo_name: SaschaHeyer/Google-Cloud-Platform-Notes


text:
waveform_collection
===================

[![](https://readthedocs.org/projects/uaf-waveform-collection/badge/?version=master)](https://uaf-waveform-collection.readthedocs.io/)

Package containing convenience functions to collect seismic/infrasound waveforms
and metadata from IRIS/WATC/AVO servers or local files (miniSEED, etc.).

Installation
------------

It's recommended that you install this package into a new or pre-existing
[conda](https://docs.conda.io/projects/conda/en/latest/index.html) environment.
(If you choose the latter option, ensure that your environment contains all of
the packages listed in the [Dependencies](#dependencies) section.)

To create a new conda environment for use with this and other _uafgeotools_
packages, execute the following terminal command:
```
$ conda create -n uafinfra -c conda-forge obspy
```
This creates a new environment called `uafinfra` with ObsPy and its dependencies
installed.

To install _waveform_collection_, execute the following terminal commands:
```
$ conda activate uafinfra  # Or your pre-existing env
$ git clone https://github.com/uafgeotools/waveform_collection.git
$ cd waveform_collection
$ pip install -e .
```
The final command installs the package in "editable" mode, which means that you
can update it with a simple `git pull` in your local repository. This install
command only needs to be run once.

Dependencies
------------

Python packages:

* [ObsPy](http://docs.obspy.org/)

...and its dependencies, which you don't really have to be concerned about if
you're using conda!

Usage
-----

Documentation is available online
[here](https://uaf-waveform-collection.readthedocs.io/).

Access the package's functions with (for example)
```python
from waveform_collection import gather_waveforms
```
and so on. For a usage example, see
[`example.py`](https://github.com/uafgeotools/waveform_collection/blob/master/example.py).

Authors
-------

(_Alphabetical order by last name._)

David Fee

Julia Gestrich

Darren Tan

Liam Toney

Andrew Winkelman

gha_language: Python

gha_license_id: MIT


text:
# NODE PDF2IMAGE

Convert pdf to image with imagemaick.

## Intallation

In root folder of your project : `npm install angelodlfrtr/node-pdf2img`

Then import it : `var PDF2img = require('node-pdf2img');`

## Usage

### Configuration

When you create a new PDF2img instance, you can pass config object :
```js
var default_config = {
  tmp_path: "/tmp",
  quality: 90,
  convert_bin: "/usr/bin/convert"
};

var instance = new PDF2img(default_config);
```

### Convert from URI

```js
var pdfinstance = new PDF2img();

pdfinstance.fromUri('http://somepdfuri').then(function(response_stream){

  // Response stream is png file read stream
  // Temp png source file is destroyed when 'end' event is fired on stream

  response_stream.on('data', function(data){
    // do something with data
  });

  response_stream.on('end', function(){
    console.log('end');
  });

}, function(err){
  console.log(err);
});
```

### Convert from file path

```js

pdf.fromFile('/tmp/test.pdf').then(function(response_stream){

  // In this case, source file is not destroyed

}, function(err){
  console.log(err);
});

```

### Convert from buffer

```js
var fs = require('fs');

fs.readFile('/tmp/test.pdf', function(err, buf){

  pdf.fromFile(buf).then(function(response_stream){

    // ...

  }, function(err){
    console.log(err);
  });
});

```

### Additionnal options

For `fromFile` and `fromUri` methods, you can pass as second argument a config object :

```js
var custom_config = {
  quality: 100,
  destroy_source: true // Destroy source file (Default unless source is a file path)
};

var promise = pdf.fromFile('/tmp/test.pdf', custom_config);

```

## LICENSE

Copyright (c) 2016 Angelo Delefortrie

MIT License

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

 IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.


text:
# Wizard with Progress Indicator

What if you want to provide the user with a progress indicator that allows them to represent where they are in the wizard? Since the `step-manager` yields all of the step names as an `Array`, you can dynamically create a progress indicator. You can even allow the user to navigate to a specific step by clicking on the location in the indicator.

{{docs/cookbook/wizard/demo-wizard-with-progress-indicator}}

repository: alexlafroscia/ember-steps

gha_language: JavaScript

repo_name: alexlafroscia/ember-steps


text:
# Shrug link shortener client

### configurations

Set up your application configurations.

To do that rename `env.example` to `env.development` and set your own configurations if needed.

Note: For test environment use `env.test` as configurations file name.

### `npm i`

Installs NPM packages.

### `npm run dev-server`

Runs the app in the development mode.

Open [http://localhost:8080](http://localhost:8080) to view it in the browser.

### `npm test`

Launches the test runner.

### `npm run build:prod`

Builds the app for production to the `public/dist` folder.

### `npm start`

Once build process is complete, this command runs Express server.

Open [http://localhost:3000](http://localhost:3000) in the browser to see the result.

Note: If you set another port in configuration file use that port in URL.


text:
---
layout: feature
title: 'Mood'
shortdef: 'mood'
udver: '2'
---

  Values:
  Adm
  Cnd
  Des
  Imp
  Ind
  Int
  Irr
  Jus
  Nec
  Opt
  Pot
  Prp
  Qot
  Sub

Mood is a feature that expresses modality and subclassifies finite
verb forms.

### `Ind`: indicative or realis

The indicative can be considered the default mood. A verb in
indicative merely states that something happens, has happened or will
happen, without adding any attitude of the speaker.

#### Examples

* [cs] _Studuješ na univerzitě._ "You study at the
  university."
* [de] _Du studierst an der Universität._ "You study at
  the university."
* [fr] _Tu le fais._ "You do it."
* [tr] *eve gidiyor* "she is going home"
* [tr] *eve gitti* "she went home"
* [et] _Sa ei tule._ "You are not coming."
* [pt] Ela foi para casa. “she went home.”
* [sq] _Ti flet shqip._ "You speak Albanian."

### `Imp`: imperative

The speaker uses imperative to order or ask the addressee to do the
action of the verb.

#### Examples

* [cs] _Studuj na univerzitě!_ "Study at the university!"
* [de] _Studiere an der Universität!_ "Study at the university!"
* [tr] *eve git* "go home!"
* [tr] *eve gidin* "go home!" (plural)
* [tr] *eve gitsin* "[let him] go home!" (3rd person imperative)
* [sa] ब्रूहि राजः / _brūhi rājaḥ_ "tell the king"

### `Cnd`: conditional

The conditional mood is used to express actions that would have taken
place under some circumstances but they actually did not / do not
happen. Grammars of some languages may classify conditional as tense
(rather than mood) but e.g. in Czech it combines with two different
tenses (past and present).

#### Examples

* [cs] Kdybych byl chytrý, studoval bych na
  univerzitě. "If I were smart I would study at the
  university" (note that only the auxiliary _bych_ is specific to
  conditional; the active participle _byl_ is also needed to
  analytically form the conditional mood, however, it will only be
  tagged as participle because it can also be used to form past tense
  indicative.)
* [tr] *eve gittiyse* "if she went home"
* [tr] *eve gidiyorsa* "if she is going home"
* [tr] *eve giderse* "if she goes home"
* [tr] *eve gidecekdiyse* "if she was going to go home"

### `Pot`: potential

The action of the verb is possible but not certain.
This mood corresponds to the modal verbs _can, might, be able to_.
Used e.g. in Finnish. See also the optative.

#### Examples

* [tr] *eve gidebilir* "she can go home"
* [tr] *eve gidemeyebilir* "she may not be able to go home"

### `Sub`: subjunctive / conjunctive

The subjunctive mood is used under certain circumstances in
subordinate clauses, typically for actions that are subjective or
otherwise uncertain. In German, it may be also used to convey the
conditional meaning.

#### Examples

* [fr] Je veux que tu le fasses "I want you to do it" lit. I
  want that you it do<tt>.Sub</tt>

### `Jus`: jussive / injunctive

The jussive mood expresses the desire that the action happens; it is thus close to both imperative and optative.
Unlike in desiderative, it is the speaker, not the subject who wishes that it happens.
Used e.g. in Arabic. We also map the Sanskrit injunctive to `Mood=Jus`.

#### Examples

* [sa] मैवं वोचः / maivaṁ vocaḥ "Do not speak this way"

### `Prp`: purposive

Means “in order to”, occurs in Amazonian and Australian languages, such as Arabana.

#### Examples

* [ard] _Antha yukarnda puntyi manilhiku._ "I am going to get some meat."

### `Qot`: quotative

The quotative mood is used e.g. in Estonian to denote direct speech.
The boundary between this mood and the non-first-hand [Evident]()iality is blurred.

#### Examples

* [et] _Sa ei tulevat._ "You are reportedly not coming."

### `Opt`: optative

Expresses exclamations like "May you have a long life!" or "If only I were rich!"
In Turkish it also expresses suggestions.
In Sanskrit it may express possibility (cf. the potential mood in other languages).

#### Examples

* [tr] *eve gidelim* 'let's go home'
* [sa] अप्रधानः प्रधानः स्यात् / _apradhānaḥ pradhānaḥ syāt_ "the unimportant person may be (become) important"

### `Des`: desiderative

The desiderative mood corresponds to the modal verb “want to”: “He
wants to come.” Used e.g. in Japanese or Turkish.

#### Examples

* [ja] 食べたい / _tabetai_ “want to eat”

### `Nec`: necessitative

The necessitative mood expresses necessity and corresponds to the modal verbs "must, should,
have to": "He must come."

#### Examples

* [tr] _eve gitmeli_ “she should go home”
* [tr] _eve gitmeliydi_ “she should have gone home”

### `Int`: interrogative

Verbs in some languages have a special interrogative form tha is used in yes-no questions.
This is attested, for instance, in the Turkic languages. Celtic languages have it for the
copula but not for normal verbs.

#### Examples

* [ug] يېدىڭىزمۇ؟ / _yëdingizmu?_ “Have you eaten?”
* [ga] _Nach in aghaidh easa atá sé ag snámh?_ “Isn't he swimming against the tide / fighting a losing battle?”
* [gd] _Rudeigin mu dheidhinn sgrìob a Venezuela, an e?_ “Something about a trip to Venezuela, isn't it?”

### `Irr`: irrealis

The irrealis mood denotes an action that is not known to have happened. As such, it is a roof term
for a group of more specific moods such as conditional, potential, or desiderative. Some languages
do not distinguish these finer shades of meaning but they do distinguish realis (which we tag with
the same feature as indicative, `Ind`) and irrealis.

#### Examples

* [quc] _Xaq ta ne kimbʼe iwukʼ._ “Let me be with you.” (“Que fuera yo con ustedes.”)

### `Adm`: admirative

Expresses surprise, irony or doubt. Occurs in Albanian, other Balkan languages,
and in Caddo (Native American from Oklahoma).

#### Examples

* [sq] _Ti fliske shqip!_ “You (surprisingly) speak Albanian!”

repository: UniversalDependencies/docs

branch_name: refs/heads/pages-source

gha_language: HTML

repo_name: UniversalDependencies/docs


text:
# Crataegus reverchonii var. mohrii (Beadle) J.B.Phipps VARIETY

#### Status
ACCEPTED

#### According to
The Catalogue of Life, 3rd January 2011

#### Published in
null

#### Original name
Crataegus mohrii Beadle

### Remarks
null

repository: mdoering/backbone

repo_name: mdoering/backbone


text:
---
layout: post
title:  "That's GOOD science!"
author: Jens Preussner
tags: welcome
cover: "assets/dyes.jpg"
categories: misc
---

At least, this is what the first page of my brand new notebook states! No, I'm not judging my own research here, the notebook was just a giveaway from the [German Stem Cell Network](http://www.gscn.org/Conferences/2015/Home.aspx) conference last week in Frankfurt and in some way inspired me to start keeping notes related to my work, ideas and projects.
The beautiful cover shows some Petri dishes with names of dyes, from which I only heard GFP before -  not a shame for a computational biologist, who spends most of the time in front of a white screen with black letters on it.
But Hold on! Sometimes its even a black screen with colored letters on it - indicating a source code and a (hopefully) productive time!

Going back to the title, the question *What IS good science?* popped into my mind and I remembered [a great paper](http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1001745) by Greg Wilson on best practises for scientific computing.
In case you're just starting with computational biology or bioinformatics or never heard about how to write what is considered good software, just jave a look at the paper and stay tuned!

Greg Wilson, D.A. Aruliah, C. Titus Brown, Neil P. Chue Hong, Matt Davis, Richard T. Guy, Steven H.D. Haddock, Kathryn D. Huff, Ian M. Mitchell, Mark D. Plumbley, Ben Waugh, Ethan P. White and Paul Wilson. Best Practices for Scientific Computing. **PLOS Biology** (2014), doi: 10.1371/journal.pbio.1001745

extension: markdown


text:
# exchange-ews-thunderbird
thunderbird addons provide exchange support: mail, calendar, global address list (contacts)
## Summary

- **Mailews**: main addon provide exchange mail function through ews interface, will also provide service of ews to dependent addons.

- **Calendarews**: provide exchange calendar function and depend on the service of ews provided by mailews

- **Contactews**: provide read-only global address book of exchange service, user can search through address book or email address editing textbox

## Installation

Install mailews first, if need calendar support, install [**lightning addon**](https://addons.mozilla.org/en-Us/thunderbird/addon/lightning/) first, then install calendarews for calendar, contactews for global address book.

## Usage

Goto menu **Options**->**Account Settings**, from the drop down list “**Account Actions**”, choose “**Add Ews Account**”, input the **email**, **user name** and **password**, either **discover** the ews address or input the url directly. Select the authentication method (Office365 should be **Basic**, other may choose **NTLM**), the **discover** button will also try to detect the authentication method.

After exchange account created, correspondent calendar and address book will be automatically created.

## Build

### Components
- **mailews/libews**: library implement exchange web service protocol, based on soap library  [**gsoap**](http://www.cs.fsu.edu/~engelen/soap.html) Building libews needs to static link to two libraries [**libcurl**](https://curl.haxx.se/libcurl/) and [**openssl**](https://www.openssl.org/), please download and build static libraries, then install the static library to a proper directoyr, for example to **/usr/local**. To Build on different platform:
  * **Mac**:configure and make the library using below scripts
  ```Bash
  ./configure --disable-debug --enable-shared --disable-static<EMAIL_ADDRESS>--with-curl=/usr/local --with-openssl=/usr/local/ssl
  make
  ```
  * **Linux**: configure and make the library using below scripts
  ```Bash
  ./configure --disable-debug --enable-shared --disable-static  --with-curl=/usr/local --with-openssl=/usr/local/ssl
  make
  ```
  * **Windows**: Please use mailews/libews/vs visual studo project file to build

- **mailews**: exchange mail addon source
-  **contactews**: exchange addressbook addon source
- **calendarews**: exchange calendar addon source

### Sample mozilla build config
```
ac_add_options --enable-application=mail
ac_add_options --enable-extensions=default,mailews,contactews,calendarews
ac_add_options --enable-debug
ac_add_options --disable-tests
ac_add_options --enable-calendar
```

repository: apechenev/exchange-ews-thunderbird

repo_name: apechenev/exchange-ews-thunderbird


text:
## JupyterLab for data science and knowledge graphs

JupyterLab image based on the [jupyter/docker-stack](jupyter/docker-stack) scipy image, compatible with OpenShift deployment, with additional packages and kernels installed for data science and knowledge graphs.

#### Installed kernels

* Python 3.8 with autocomplete and suggestions ([LSP](https://github.com/krassowski/jupyterlab-lsp))
* [IJava](https://github.com/SpencerPark/IJava) with current default OpenJDK
* [SPARQL kernel](https://github.com/paulovn/sparql-kernel)

#### Installed Jupyterlab extensions

- [jupyterlab-git](https://github.com/jupyterlab/jupyterlab-git)
- [jupyterlab-lsp](https://github.com/krassowski/jupyterlab-lsp)

Volumes can be mounted into `/home/jovyan` folder.

## Build and publish

Build:

```bash
docker build -t ghcr.io/maastrichtu-ids/jupyterlab .
```

Push:

```bash
docker push ghcr.io/maastrichtu-ids/jupyterlab
```

Using a specific `Dockerfile`:

```bash
docker build -f Dockerfile -t ghcr.io/maastrichtu-ids/jupyterlab .
```

## Run locally

Test run:

```bash
docker run --rm -it --user $(id -u) -p 8888:8888 -e JUPYTER_TOKEN=password -v $(pwd):/home/jovyan ghcr.io/maastrichtu-ids/jupyterlab
```

> Use `JUPYTER_TOKEN` or `JUPYTER_NOTEBOOK_PASSWORD` for password

Run as `root` user:

```bash
docker run --rm -it --user root -p 8888:8888 -e GRANT_SUDO=yes -e JUPYTER_TOKEN=password -v $(pwd):/home/jovyan ghcr.io/maastrichtu-ids/jupyterlab
```

> You should now be able to install anything in the JupyterLab container, try:
>
> ```bash
> sudo apt install vim
> ```
>

Check `docker-compose.yml` to run with Docker Compose:

```yaml
  jupyterlab:
    container_name: jupyterlab
    image: ghcr.io/maastrichtu-ids/jupyterlab
    user: root
    environment:
      - JUPYTER_TOKEN=dba
      - GRANT_SUDO=yes
    volumes:
      - ./data/jupyterlab:/home/jovyan
    ports:
      - 8888:8888
```

**Potential permission issue:**

The official Jupyter/docker-stack image uses the `jovyan` user by default which does not have admin rights (`sudo`).

This can cause issues when writing to the shared volumes, to fix it you can change the owner of the folder or start JupyterLab as root user:

Create the folder with right permission

```bash
mkdir -p data/jupyterlab
sudo chown -R 1000:1000 data/jupyterlab
```

And run JupyterLab with admin rights

## Deploy on OpenShift

* See this template to deploy [JupyterLab on OpenShift with restricted user](https://github.com/MaastrichtU-IDS/dsri-openshift-applications/blob/main/okd4-templates-restricted/template-jupyterlab-restricted.yml).

* See this template to deploy [JupyterLab on OpenShift with `sudo` privileges](https://github.com/MaastrichtU-IDS/dsri-openshift-applications/blob/main/okd4-templates-anyuid/template-jupyterlab-root.yml).

If you are working or studying at Maastricht University you can easily [deploy this notebook on the Data Science Research Infrastructure (DSRI)](https://maastrichtu-ids.github.io/dsri-documentation/docs/deploy-jupyter).

repository: jiaoshoujie/jupyterlab

repo_name: jiaoshoujie/jupyterlab


text:
For documentation about the servers and protocols go here: [https://github.com/Kinnay/NintendoClients/wiki](https://github.com/Kinnay/NintendoClients/wiki).

* [Features](#features)
* [Contributing](#contributing)
* [API Reference](#api-reference)

## Features
This package is able to do everything that's required to access a game server. It also provides a framework to host your own game servers. For example scripts, check out the [github repository](https://github.com/Kinnay/NintendoClients). The following servers are currently supported:

* Wii U:
	* Game servers (`nex`)
	* `account.nintendo.net`
	* `idbe-wup.cdn.nintendo.net`

* Switch:
	* Game servers (`nex`)
	* `dauth-lp1.ndas.srv.nintendo.net`
	* `aauth-lp1.ndas.srv.nintendo.net`
	* `e0d67c509fb203858ebcb2fe3f88c2aa.baas.nintendo.com`

## Contributing
If you would like to contribute code or knowledge in any way, feel free to open a pull request or issue on [github](https://github.com/Kinnay/NintendoClients). If you open a pull request, please try to follow the current code style as much as possible:

* Use tabs for indentation, and spaces for alignment.
* Use double quotes for strings, unless the string contains a double quote itself. In that case, using single quotes is ok.
* Use the following naming convention:

```python
GLOBAL_CONSTANT = "hi"

class ClassName:
	def method_name(self):
		self.variable_name = 0
```

* There is no hard limit on line lengths, but try to keep it somewhat reasonable.

* Before you submit a pull request, run the tests to make sure that you did not break something. Simply run `pytest` in the root directory of the repository. This shouldn't take longer than a few seconds.
* If you fix a bug, consider writing a new test to make sure that this bug does not happen again.
* If you implement a new function or feature, add it to the documentation. After that, write a test to make sure that it works correctly. The test should only be based on the documentation, not on the code. We don't want to test implementation details. If the documentation is not clear enough to write a test, update the documentation.

## API Reference

* nintendo
	* common
		* [crypto](reference/common/crypto)
		* [http](reference/common/http)
		* [streams](reference/common/streams)
		* [tcp](reference/common/tcp)
		* [tls](reference/common/tls)
		* [types](reference/common/types)
		* [udp](reference/common/udp)
		* [util](reference/common/util)
		* [websocket](reference/common/websocket)
		* [xml](reference/common/xml)
	* enl
		* [crypto](reference/enl/crypto)
	* nex
		* [aauser](reference/nex/aauser)
		* [account](reference/nex/account)
		* [authentication](reference/nex/authentication)
		* [backend](reference/nex/backend)
		* [common](reference/nex/common)
		* [datastore](reference/nex/datastore)
		* [datastore_smm](reference/nex/datastore_smm)
		* [datastore_smm2](reference/nex/datastore_smm2)
		* [debug](reference/nex/debug)
		* [errors](reference/nex/errors)
		* [friends](reference/nex/friends)
		* [health](reference/nex/health)
		* [hpp](reference/nex/hpp)
		* [kerberos](reference/nex/kerberos)
		* [matchmaking](reference/nex/matchmaking)
		* [messaging](reference/nex/messaging)
		* [monitoring](reference/nex/monitoring)
		* [nattraversal](reference/nex/nattraversal)
		* [nintendonotification](reference/nex/nintendonotification)
		* [notification](reference/nex/notification)
		* [prudp](reference/nex/prudp)
		* [ranking](reference/nex/ranking)
		* [ranking2](reference/nex/ranking2)
		* [remotelog](reference/nex/remotelog)
		* [rmc](reference/nex/rmc)
		* [screening](reference/nex/screening)
		* [secure](reference/nex/secure)
		* [streams](reference/nex/streams)
		* [subscriber](reference/nex/subscriber)
		* [utility](reference/nex/utility)
	* pia
		* [lan](reference/pia/lan)
		* [settings](reference/pia/settings)
		* [types](reference/pia/types)
	* [aauth](reference/aauth)
	* [baas](reference/baas)
	* [dauth](reference/dauth)
	* [games](reference/games)
	* [idbe](reference/idbe)
	* [miis](reference/miis)
	* [nnas](reference/nnas)
	* [sead](reference/sead)
	* [switch](reference/switch)

repository: HookedBehemoth/NintendoClients

repo_name: HookedBehemoth/NintendoClients


text:
# Working Title CJ4 v0.10.4

Welcome to the Working Title CJ4 v0.10.4.  This patch removes the temporary flaps lift workaround as well as addresses a couple other critical issues.

## Installation
Installation is easy, simply copy the `workingtitle-aircraft-cj4` folder inside the zip file to your MSFS Community folder.

**Important: We recommend that you fully delete the previous `workingtitle-aircraft-cj4` folder before copying this release.**

## Default Livery Conflicts
The alternate default liveries that now ship with MSFS are not compatible with the CJ4 yet at this time. They currently ship with panel.cfg files that do not include our FADEC module. Please use the standard livery or a known compatible 3rd party or community livery.

# Changes

- Course needle for ILS should now be properly set during nav-to-nav transfers
- Incompatible livery warning should no longer show improperly with compatible liveries if FADEC load is delayed
- Flaps lift workaround has been removed

# ⚠ Known Issues
* PTCH mode will not level off at an altitude and it can have some quirky behaviors.  This is currently a sim AP issue.
* Some external applications that use the GPS/Flight plan SimVars may not function correctly or as expected when FP Sync is off.
* Loading and saving flights can have bad results.
* Custom liveries can render FADEC inoperative if they ship with a panel.cfg. You must uninstall them or remove their panel.cfg from the livery folder. This is a limitation of the Asobo livery system.
* Autopilot modes cannot be triggered via keybindings or controllers and must currently be triggered in the cockpit with the mouse. External binding applications are adding support for LVars and HEvents. Used SimVars are documented [here](https://github.com/Working-Title-MSFS-Mods/fspackages/wiki/Sim-Variables)
* Sometimes a heading to altitude instruction on takeoff will display further than the first RNAV fix on an RNAV departure procedure; in these cases the workaround is to cross-check the DP chart and remove the erroneous waypoint either by deleting the heading to altitude fix or dropping the first RNAV fix onto the magenta line in the LEGS page.
* Due to sim autopilot bank rate limitations, the aircraft may overshoot on certain RNP approaches with tight turns. If you encounter this, we recommend handflying the approach with the given lateral and vertical guidance.
* If for whatever reason, you find that VNAV is not behaving as expected, try and turn it off and on again.

repository: Working-Title-MSFS-Mods/fspackages

path: /docs/workingtitle-cj4/Working Title CJ4 v0.10.4 Changes.md

filename: Working Title CJ4 v0.10.4 Changes.md

repo_name: Working-Title-MSFS-Mods/fspackages


text:
---
title: "Tiles editor progress report #3"
excerpt: "A new progress report on the TileMap and TIleSet editors rework."
categories: ["progress-report"]
author: Gilles Roudiere
image: /storage/app/uploads/public/607/066/eee/607066eee70c4289567474.png
date: 2021-04-12 08:34:53
---

It's time for a third progress report on the TileMap and TileSet editors rework! Updates are likely less visually appealing than in the [previous progress report](https://godotengine.org/article/tiles-editor-progress-report-2), but a lot of groundwork has been done since then.

*See other articles in this Godot 4.0 tiles editor series:*

1. [Tiles editor progress report #1](https://godotengine.org/article/tiles-editor-rework)
2. [Tiles editor progress report #2](https://godotengine.org/article/tiles-editor-progress-report-2)
3. (you are here) [Tiles editor progress report #3](https://godotengine.org/article/tiles-editor-progress-3)
4. [Tiles editor progress report #4](https://godotengine.org/article/tiles-editor-progress-4)
5. [Tiles editor progress report #5](https://godotengine.org/article/tiles-editor-progress-report-5)

### TileSet rework

A significant part of my work those past two months has mostly been about reimplementing most of the already existing TileSet features, while introducing more flexibility to it. Most of TileSet's systems (rendering, collisions, physics, navigation, …) are now using a concept of "layers" (this might be renamed). This allows you to customize the properties that the tiles themselves expose. As an example, you could now define several PhysicsBodies per tile with different collision layers/masks. This was not possible before.

The remaining of this section describes the TileSet changes I made. Right now, there is no way to edit the tiles' properties in an efficient way. They are only editable using a dedicated inspector for now, but I plan to implement a way to paint values over tiles in the TileSet editor. However, most properties can now be visualized in the tile atlas.

#### Physics

TileSet's physics have been reimplemented from scratch. The new implementation allows to define several physics layers. Each *TileSet's physics layer* has a collision layer property, a collision mask property and a physics material.
Then, each tile can define one or several shapes for each *TileSet's physics layer*.

![TileSet physics](/storage/app/uploads/public/607/05d/819/60705d8191412365837988.gif)

#### Navigation

Navigation was also implemented again. As it was quite limited before, I did a little bit of work on NavigationServer. We decided in 4.0 to [get rid of the Navigation2D and Navigation3D nodes](https://github.com/godotengine/godot/pull/46786). To replace them, you can now assign navigation layers to navigation regions. Then, when using `NavigationServer2D.map_get_path(...)`, you can provide a mask allowing you to choose which layers to include/exclude from the path request. Thoses changes consequently allow TileMap nodes to be independent from any parent Navigation2D nodes.

In a way similar to *TileSet's physics layers*, *TileSet's navigation layers* may allow you to set different navigation layers.

![TileSet navigation](/storage/app/uploads/public/607/05d/fb7/60705dfb7e92f692135683.gif)

TileMap can now display navigation regions as debugging information (it's hard to see on the picture here, as it's light green over green).

Note that navigation in TileMap is still a little bit rough on the edges. The pathfinding algorithm doesn't work well with the high number of regions a TileMap would generate. This will need further improvements to be more usable.

#### Custom data

As it was quite a requested feature, TileSet now support *custom data layers*. Each custom data layer can be assigned a type and a name. Once this is done, each tile of the TileSet can be assigned a value for each custom data layer.

![custom_data.gif](/storage/app/uploads/public/607/064/9da/6070649dac6f3069466787.gif)

Right now, the tile inspector displays the index instead of the custom data name, but this may change in the future.

#### Terrains

*Terrains* are the more powerful replacement of autotiles. Unlike autotiles, terrains can support transitions from one terrain to another, as a tile may define several terrains at once.

Terrains are grouped into terrain sets. Each terrain set is assigned a mode from "Match corners and side", "Match corners" and "Match sides". They define how terrains are matched to each other in a terrain set. (This corresponds to the previous bitmask modes autotiles used: 2×2, 3×3 or 3×3 minimal, and is similar to what Tiled features.) Terrains will support hexagonal and isometric shapes.

Unlike before, where autotiles were a specific kind of tiles, terrains are only a set of properties assigned to atlas tiles. These properties are then used by a dedicated TileMap painting mode that selects tiles featuring terrain data in a smart way. This means any terrain tile can be either painted as terrain or as a single tile, like any other.

I am still working on the terrain painting tool, so it's not done yet. Nonetheless, here's a sneak peek of what it looks like for now:

![TileSet terrains](/storage/app/uploads/public/607/05f/665/60705f665e652945333713.gif)

### Other changes

You might have noticed that, compared to the previous screenshots of the TileSet editor, some tabs are now missing (Atlases, TileSet layers, TileProperties and scenes). This is due to the fact that, internally, things have been modified. While I used to think the scenes painting would be quite different from the atlases one, they are quite similar in the way they are basically "something to paint" on the TileMap.

As a consequence, I implemented the concept of *TileSetSource* to group them. A TileSet source is a group of tiles to be painted on the TileMap. Right now, only the `TileSetAtlasSource` (getting its tiles from a single atlas texture) is implemented, but I'd like to implement a `TileSetScenesCollectionSource` (storing a set of scenes) and a `TileSetSingleTilesCollectionSource` (storing tiles with a single texture each).

This allowed me to simplify the bottom interface, with the list of TileSetSources on the left, and their editors on the right.
The editing of TileSet layers, as seen above, have been moved to the inspector. That's likely a better place to edit them as they are something global to the TileSet resource.

![2021-04-09-160807_826x189_scrot.png](/storage/app/uploads/public/607/05f/d88/60705fd88dea3038798751.png)

Similarly, the TileMap editor view is now more compact, as I moved the tabs to the toolbar.

![Tabs in the toolbar](/storage/app/uploads/public/607/05f/ac6/60705fac69254963527894.png)

### Coming next

I will first have to finish my work on the terrains system, then work on ensuring backwards compatibility with 3.x.
We will try to merge the pull request after that, so that other contributors and early adopters can have a look into it and test it in real world conditions.

repository: godotengine/godot-website

repo_name: godotengine/godot-website


text:
# Matrix HOME

Matrix HOME is a home automation project built around Raspberry Pi 3 and [Matrix Creator One](https://creator.matrix.one) board.

## Features
* Offline wakeword listening
* Speech to text using Google API
* Fuzzy matching Google STT result (closest match to predetermined list of phrases)
* Visual feedback via LED array
* Send commands to Philips HUE or IR devices (TV, speakers, etc...) based on speech

## Prerequisites
1. Raspberry Pi
2. Creator Matrix One board
3. Philips Hue Bridge & some lamps (optional, HUE actions can be disabled easily in the code)
4. IR devices (you will need to add the relevant LIRC IR commands in the app.py yourself)

## Setting up & running
1. Follow the [docs](https://matrix-io.github.io/matrix-documentation/) how to set up and configure Matrix CORE for rpi.
2. Checkout the project and update [submodule](https://github.com/akukolu/matrix-led) from remote
3. Create some .pmdl hotword models with [snowboy](https://snowboy.kitt.ai/) and place them in `snowboy/models` or use the default one (Alexa)
4. [Set up Google API credentials](https://cloud.google.com/video-intelligence/docs/common/auth)
5. `pip install -r requirements.txt`
6. Press the button in Philips Hue bridge. You only need to do this step when running the app for the first time.
7. `python app.py ip.to.your.hue.bridge`

## Changelog
* `0.3.0` use proper speech to text api
* `0.2.0` move snowboy wakeword functionality to a package, python3 is now required
* `0.1.0` first "mvp" version

## Roadmap
* Make the project runnable in any device (for example Google AIY kit)

## Testing
`python -m unittest`

## License
Apache

repository: akolu/matrix-home

repo_name: akolu/matrix-home


text:
# Tagus CMS

This is a simple CMS using Javascript only! It was developed with Nodejs/Express, Mongo DB and React

## Getting Started

To install the npm package:
```
npm install --save tagus-cms

```

To access your Tagus CMS back office simply navigate to '/tagus-admin'

### Prerequisites

This was developed running node v6.11.1, so at least this is the needed version to run it.

### Installing

Start your standard express app, and extend it with tagus-cms!

```
const express = require('express');
const bodyParser = require('body-parser');
const path = require('path');
const morgan = require('morgan');
const hbs = require('hbs');
const app = express();
const tagusCMS = require('./tagus/tagus-cms');
const portNumber = process.env.PORT_NUMBER;

app.use(morgan('dev'));
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));
app.use(express.static('SiteName'));

tagusCMS.extend(app, {
  media: {
    path:'SiteName/img',
    dir: '/img',
    root: 'SiteName'
  },
  views: {
    path: [path.join(__dirname, 'SiteName/views')],
    engine: 'hbs'
  },
  public: 'SiteName',
  mongoConnectionString: process.env.MONGO_CONNECTION_STRING,
  domain: process.env.DOMAIN,
  authSecretKey: process.env.AUTHSECRETORKEY,
  email: {
    email: process.env.EMAIL_ADDRESS,
    pass: process.env.EMAIL_PASSWORD
  }
});

app.listen(portNumber, function () {
  console.log("listening to " + portNumber);
});
```

The extend function accepts your Express instance and an object with the following mandatory settings.

```
{
    media: {
        path: "The path where your media files are",
        dir: "The actual directory of your media files,
        root: "The root of your static website files"
    },
    views: {
        path:"The path of your view template files",
        engine: "The view engine you are using"
    },
    public: "The public folder of your website",
    mongoConnectionString: "The connection string for your Mongo db",
    domain: "The domain of your website",
    authSecretKey: "Tagus CMS uses passport and tokens to manage some permissions. Create an auth key to encrypt your tokens",
    email: {
        email: "An email address. This will be necessary in order to send the new users their initial password, so that they can access the Tagus CMS back office",
        pass: "Your email account password"
    }
}
```

## Contributing

Feel free to open the necessary PR!

I'm using the 'https://github.com/snot-dev/project-tagus' repo to develop this app. Feel free to check it out!

## Authors

* **Gonçalo Assunção** - [snot-dev](https://github.com/snot-dev)

## License

This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details

## Acknowledgments

Thanks to the following guys, for the help and support:

* **Gonçalo Cardoso** - [gcardoso89](https://github.com/gcardoso89)
* **João Ramalho** - [ramalhovfc](https://github.com/ramalhovfc)
* **Leandro Mendes** - [Pr0fil3](https://github.com/Pr0fil3)
* **Tiago Fernandes** - [vftiago](https://github.com/vftiago)

The Tagus Logo was created by:

**Adolfo Ferreira** - http://www.adolfoferreira.com/

Special thanks to, for the many design inputs:

**Jesús Alfonso Sánchez** - http://www.jasnchz.com/

Enjoy!

repository: snot-dev/tagus-cms

repo_name: snot-dev/tagus-cms


text:
## Count Substrings : Editorial by Richi

* __Algorithm and Implementation__

This is a pretty simple question, but an interesting thing to note here is that, this has to be solved in less that O(n2) time complexity(Since the value of n goes
up to 10^5 and its square would be 10^10 which would give us a time-out error). I didn't notice that before and wrote a working solution(which obviously is TLE). If you want
to check it out, head up to : [Richi's Failed Solution](https://www.codechef.com/status/CSUB,ilovemetoo).

Anyways, As always, try solving for some cases on your own and see how things work out.

For example, I tried with _10101_ which should give output as 6, and I saw it as :

> Count from left to right, If you encounter a 1, add 1 to count(our answer) along with number of ones encountered so far(since they would form a substring with this current 1 as
their ending point).
So you maintain two variables, count and number of ones till now.

_Difficulty_ : 1/5.

gha_language: Java


text:
# Instructions for Logging Issues

## 1. Search for Duplicates

[Search the existing issues](https://github.com/coveo/uabot/issues) before logging a new one.

## 2. Do you have a question?

The issue tracker is for **issues**, in other words, bugs and suggestions.

## 3. Did you find a bug?

When logging a bug, please be sure to include the following:

 * If at all possible, an *isolated* way to reproduce the behavior
 * The behavior you expect to see, and the actual behavior

## 4. Do you have a suggestion?

We also accept suggestions in the issue tracker.

In general, things we find useful when reviewing suggestions are:

* A description of the problem you're trying to solve
* An overview of the suggested solution
* Examples of how the suggestion would work in various places
  * Code examples showing e.g. "this would be an error, this wouldn't"

# Instructions for Contributing Code

## Contributing bug fixes

uabot is currently accepting contributions in the form of bug fixes. A bug must have an issue tracking it in the issue tracker that has been approved ("Milestone == Community") by the Coveo team. Your pull request should include a link to the bug that you are fixing. If you've submitted a PR for a bug, please post a comment in the bug to avoid duplication of effort.

## Contributing features

Features (things that add new or improved functionality to uabot) may be accepted, but will need to first be approved (marked as "Milestone == Community" by a uabot coordinator with the message "Approved") in the suggestion issue. Features with language design impact, or that are adequately satisfied with external tools, will not be accepted.

Design changes will not be accepted at this time. If you have a design change proposal, please log a suggestion issue.

## Housekeeping

Your pull request should:

* Include a description of what your change intends to do
* Be a child commit of a reasonably recent commit in the **master** branch
    * Requests need not be a single commit, but should be a linear sequence of commits (i.e. no merge commits in your PR)

* It is desirable, but not necessary, for the tests to pass at each commit
* Have clear commit messages
    * e.g. "Refactor feature", "Fix issue", "Add tests for issue"
* Include adequate tests
    * At least one test should fail in the absence of your non-test code changes. If your PR does not match this criteria, please specify why
    * Tests should include reasonable permutations of the target fix/change
    * Include baseline changes with your change
    * All changed code must have 100% code coverage

repository: coveooss/uabot

gha_language: Go

repo_name: coveooss/uabot


text:
# Network Slimming (Pytorch 1.0)

This repository contains an pytorch implementation for the following paper
[Learning Efficient Convolutional Networks Through Network Slimming](http://openaccess.thecvf.com/content_iccv_2017/html/Liu_Learning_Efficient_Convolutional_ICCV_2017_paper.html) (ICCV 2017).

- python 3.5+
- pytorch 1.1.0

## Channel Selection Layer
`ChannelSelection` layer is used to help the pruning of ResNet. This layer stores a parameter indexes which is initialized to an all-1 vector.
During pruning, it will set some places to 0 which correspond to the pruned channels.

## Baseline
Train preactivation resnet20 as baseline.
```bash
python3 main.py --arch=preact_res20 --tag=baseline
```

## Train with Sparsity
Add L1 regularization to BatchNorm2d layer's weight

```bash
python3 main.py -sr --s 0.0001 --arch=preact_res20
```

`--s` will be used in `update_bn` function
```python
def update_bn(model, s=0.0001):
    for m in model.modules():
        if isinstance(m, nn.BatchNorm2d):
            m.weight.grad.data.add_(s * torch.sign(m.weight.data))  # L1
```

Best accuracy model in `./ckpts/preact_res20/best_acc`

You can compare distribution of `BatchNorm2d` weight in tensorboard.

![images](./images/bn_weights.png)

## Prune model trained with sparsity
Run following command and pruned model will be saved as `./ckpts/pruned/prect/preact_res20_pruned.pth.tar`.

```bash
python3 resprune.py --ckpt_dir=./ckpts/preact_res20_sr/best_acc --arch=preact_res20

Loading checkpoint from ckpts/preact_res20_sr/best_acc/ckpt_24696_acc90.960_loss0.2836.pth.pth.tar
BN layer name: layer1.0.bn1	pruned 43.75%(7/16)
BN layer name: layer1.0.bn2	pruned 18.75%(3/16)
BN layer name: layer1.0.bn3	pruned 0.00%(0/16)
BN layer name: layer1.1.bn1	pruned 89.06%(57/64)
BN layer name: layer1.1.bn2	pruned 37.50%(6/16)
BN layer name: layer1.1.bn3	pruned 0.00%(0/16)
BN layer name: layer2.0.bn1	pruned 43.75%(28/64)
BN layer name: layer2.0.bn2	pruned 0.00%(0/32)
BN layer name: layer2.0.bn3	pruned 0.00%(0/32)
BN layer name: layer2.1.bn1	pruned 87.50%(112/128)
BN layer name: layer2.1.bn2	pruned 0.00%(0/32)
BN layer name: layer2.1.bn3	pruned 0.00%(0/32)
BN layer name: layer3.0.bn1	pruned 19.53%(25/128)
BN layer name: layer3.0.bn2	pruned 0.00%(0/64)
BN layer name: layer3.0.bn3	pruned 0.00%(0/64)
BN layer name: layer3.1.bn1	pruned 51.17%(131/256)
BN layer name: layer3.1.bn2	pruned 0.00%(0/64)
BN layer name: layer3.1.bn3	pruned 0.00%(0/64)
BN layer name: bn	pruned 68.75%(176/256)
cfg 19: [9, 13, 16, 7, 10, 16, 36, 32, 32, 16, 32, 32, 103, 64, 64, 125, 64, 64, 80]
Pruned model save to: ./ckpts/pruned/preact_res20_pruned.pth.tar
Pruned rate: 0.4
Model acc:90.960 loss: 0.284 time: 1.526s
Pruned model acc:17.660 loss: 7.252 time: 1.398s
Time reduce: 8.3854% (1.53s -> 1.40s)
Model params reduce: 8.4535% (220394.00 -> 201763.00)
Model size reduce: 53.4249% (1.72mb->0.80mb)
```

## Fine tune pruned model
```bash
python3 main.py --refine=./ckpts/pruned/preact_res20_pruned.pth.tar --arch=preact_res20 --tag=fine_tune_pruned
```

repository: Sanster/network-slimming

repo_name: Sanster/network-slimming


text:
# Tune Book

The following analysis is based on the abc string that is in the textarea at the bottom of this page.

## Number of Tunes

It might be useful to know in advance how many tunes are in a tunebook before rendering them:

<num-tunes ref="numTunes"></num-tunes>

## Get Tune Info

The following assumes you've created a `TuneBook` object like this:

```
var tuneBook = new ABCJS.TuneBook(tunebookString)
```

### Extract Tune by ID

If you know the ID, then you can use the following to get a particular tune in a tunebook. The id is the value in the `X:` field.

<tune-book-info ref="tuneById" type="id"></tune-book-info>

### Extract Tune by Title

If you know the title, then you can use the following to get a particular tune in a tunebook. The title is the value in the *first* `T:` field.

<tune-book-info ref="tuneByTitle" type="title"></tune-book-info>

### Get all Tune Info

You can directly access the array of tunes in a tune book with:

```javascript
var arrayOfTunes = tunebook.tunes;
```

### Get all measures separately

To parse the string and return each measure:

```javascript
var measureArray = abcjs.extractMeasures(tunebookString);
```

## Test Data

Paste in any ABC you want here and see how that affects the analysis above:

<example-tune-book v-if="abcjsReady" :callbacks="callbacks"></example-tune-book>

<script>
	import { waitForAbcjs } from '../../../wait-for-abcjs';
	export default {
		async mounted() {
            await waitForAbcjs()
            this.abcjsReady = true;
			setTimeout(() => {
				this.callbacks = [this.$refs.numTunes, this.$refs.tuneById, this.$refs.tuneByTitle];
			}, 500);
		},
		data() {
			return {
				abcjsReady: false,
				callbacks: [],
			};
		},
	}
</script>

repository: paulrosen/abcjs

gha_license_id: NOASSERTION

repo_name: paulrosen/abcjs


text:
## Working on Java server version

```java
import twitter4j.Status;
import twitter4j.Twitter;
import twitter4j.TwitterException;
import twitter4j.TwitterFactory;

import java.io.IOException;
import java.util.List;

public class test {
    public static void main(String[] args) throws TwitterException, IOException {
        // gets Twitter instance with default credentials
        Twitter twitter = new TwitterFactory().getInstance();
        try {
            List<Status> statuses;
            String user;
            if (args.length == 1) {
                user = args[0];
                statuses = twitter.getUserTimeline(user);
            } else {
                user = twitter.verifyCredentials().getScreenName();
                statuses = twitter.getUserTimeline();
            }
            System.out.println("Showing @" + user + "'s user timeline.");
            for (Status status : statuses) {
                System.out.println("@" + status.getUser().getScreenName() + " - " + status.getText());
            }
        } catch (TwitterException te) {
            te.printStackTrace();
            System.out.println("Failed to get timeline: " + te.getMessage());
            System.exit(-1);
        }
    }
}
```

## Description
A Twitter hot tweets viewer to show at most 10 "representative tweets" selected from the tweets of a twitter user.
There are two types of "representativeness": (1) the author's definition -- the number of like; and (2) Twitter's popularity.

## Installation
Please add a twitter_config file as advised [here](https://github.com/BoyCook/TwitterJSClient#usage) before the following steps.

`npm install`

`npm start`

## Implementation
[Twitter is running OAuth](https://dev.twitter.com/oauth/overview/introduction), so we need a server to fetch tweets from Twitter.  As for the server setup, I use node.js and express.js web framework.  In order to deal with Twitter's Oauth and pull data from Twitter, [Twitter Libraries](https://dev.twitter.com/resources/twitter-libraries) can help a lot, and I go with [TwitterJSClient](https://github.com/BoyCook/TwitterJSClient) by @BoyCook.  During the development, I use Gulp to automate and enhance my workflow.  I also perform unit tests on the APIs of the server with Mocha and Chai.  At last, I deploy the application to AWS EC2.

## Architecture
### Representativeness: (1) the number of like

![alt](https://github.com/drjkuo/JujiOA/blob/master/readme/apiv2.png)

The diagram above shows the system architecture of first type of "representativeness": the author's definition -- the number of like.  When an user visit the landing page of our server, the server returns indexV2.html.  After the user enters one twitter handle, top 10 liked tweets will be displayed in the decreasing order.

#### Client side
On the client side, in the main.js file, the display of top 10 liked tweets is done as follows: (1) The keypress "Enter" triggers an AJAX request and expects to receive data consisting of an array of tweet IDs; (2) On receiving the expected data, [Twitter's factory function](https://dev.twitter.com/web/javascript/creating-widgets) twttr.widgets.createTweet() help us to dynamically generates the desired widgets; (3) The browser will show top 10 liked tweets in a decreasing order.
```javascript
  function apiV2 (event){
    if (event.which == 13 || event.keyCode == 13) {
      event.preventDefault();
      var twitterID = document.getElementById("twitterID").value;
      // remember to replace localhost with your site IP
      var url = "http://localhost:3000/hot2?twitterID="+twitterID;
      $.getJSON( url, function( data ) {
        noShow.style.display= (data.length === 0) ? "" : "none";
        for (var i=0; i<tweetNum; i++) {
          // clear div
          document.getElementById(`id${i}`).innerHTML = "";
          // Twitter factory api for dynamically generating widgets
          twttr.widgets.createTweet(
            data[i],
            document.getElementById(`id${i}`),
            {
              align: 'left'
            })
          ......
      }
```

#### Server side
On the server side, in the index.js file, I leverage twitter.getUserTimeline by @BoyCook to pull the most recent 200 tweets of the designated twitter ID from Twitter.  Then, I sort those tweets by their number of liked and return top 10 liked tweets to the client.  One enhancement could be made by replacing sort() with the help of a priority queue.
```javascript
app.get('/hot2', function (req, res) {
  twitter.getUserTimeline({ screen_name: req.query.twitterID, count: '200'},
    error,
    function (data) {
      toBeSent2 = [];
      data = JSON.parse(data);
      // Sort tweet IDs by the number of liked
      function cmp(a, b) { return b.cnt - a.cnt; };
      var tweetArr = [];
      for (var i=0; i<data.length; i++) {
        tweetArr.push({id: data[i].id_str, cnt: data[i].favorite_count});
      }
      tweetArr.sort(cmp);
      // Top 10 liked tweets
      var topNumber = 10;
      for (var i=0; i<topNumber; i++) {
        if (tweetArr[i]) toBeSent2.push(tweetArr[i].id);
      }
      res.json(toBeSent2);
    }
  );
});
```

### Representativeness: (2) Twitter's popularity

![alt](https://github.com/drjkuo/JujiOA/blob/master/readme/apiv1.png)

The diagram above shows the system architecture of second type of "representativeness": [Twitter's popularity](https://dev.twitter.com/rest/reference/get/search/tweets).  When an user visit another landing page of our server, the server returns index.html.  After the user enters one twitter handle, top 10 popular tweets will be displayed, where Twitter does not indicate how they evaluate the popularity.

#### Client side
The client side works very similar to the abovementioned.  The difference is sending the request to another API.

#### Server side
On the server side, in the index.js file, I leverage twitter.getSearch by @BoyCook to pull the most 10 popular tweets of the designated twitter ID from Twitter, and return top 10 liked tweets to the client.

```javascript
app.get('/hot', function (req, res) {
  var q = "from:"+req.query.twitterID;
  twitter.getSearch({ 'q': q , 'result\_type':'popular', 'count': '10'},
    error,
    function (data) {
      toBeSent = [];
      data = JSON.parse(data);
      for (var i=0; i<data.statuses.length; i++) {
        toBeSent.push(data.statuses[i].id_str);
      }
      res.json(toBeSent);
  });
});
```

## Unit testing
Unit testing on the three APIs (GET /, GET /hot, GET /hot2) of the server.
![alt](https://github.com/drjkuo/JujiOA/blob/master/readme/unitTest.png)

## Build tools
I use Gulp to help on JS file uglify and html minify.

## Future work
Due to the limitation (max 200 tweets) of tweetsTwitter's timeline API, if we would like to pull more than 200 of most recent tweets, we need to check with [Working with Timelines](https://dev.twitter.com/rest/public/timelines) to paginate large result sets.

repository: drjkuo/hotTweets

repo_name: drjkuo/hotTweets


text:
## Running Locally

-   Install Dependecies: `npm install`
-   Run Bot: `npm run start:dev`

## Set Environment Variables Locally

File: `.env`

```
ENVIRONMENT=DEVELOPMENT
PATH_URL=api.mercadolibre.com
BOT_TOKEN=8888888888:xxxxxxxxxxxxxxx// PRODUCTION
BOT_TOKEN_DEV=8888888888:xxxxxxxxxxxxxxx // DEVELOPMENT
APP_URL=<https://<your-app-name>.azurewebsites.net/api/<your-api-name>?code=<your-api-function-key>> // Isn't required in development environment
```

## How To Get a Telegram Bot Token

Find telegram bot named "@botfarther", he will help you with creating and managing your Bot.

## Automatic Deployments

- [Setup Automatic Deploy](https://docs.microsoft.com/en-us/azure/azure-functions/functions-how-to-github-actions?tabs=dotnet)
- Modify your .yml: `github/workflows/master_telegram-experimental.yml`

```
AZURE_FUNCTIONAPP_NAME: <your azure function app name>
AZURE_FUNCTIONAPP_PUBLISH_PROFILE: <your github secret>
NODE_VERSION: '14.x' <default>
```

repository: silverbryan/mercadolibre-bot

repo_name: silverbryan/mercadolibre-bot


text:
flask-bones [![Build Status](https://travis-ci.org/cburmeister/flask-bones.png?branch=master)](https://travis-ci.org/cburmeister/flask-bones) [![Coverage Status](https://coveralls.io/repos/cburmeister/flask-bones/badge.png?branch=master)](https://coveralls.io/r/cburmeister/flask-bones?branch=master) [![Stories in Ready](https://badge.waffle.io/cburmeister/flask-bones.png?label=ready&title=Ready)](https://waffle.io/cburmeister/flask-bones)
===========

I've been reusing this pattern for Flask applications and decided to stop repeating myself.

## Setup

1. Install required services:

    ```
    $ brew install memcached
    $ brew install redis
    $ brew install postgresql
    ```

2. Install Python packages:

    ```
    $ pip install -r requirements.txt
    ```

3. Set necessary environment variables:

    ```
    $ export SECRET_KEY=46-2346-24986-2384632-2039845-24
    $ export DATABASE_URL=postgresql://$USER@localhost/flask_bones
    $ export SERVER_NAME=$HOST:5000
    ```

4. Install Javascript dependencies:

    ```
    $ cd app/static
    $ bower install
    ```

5. Setup database and seed with test data:

    ```
    $ python manage.py shell
    >>> import tests
    >>> tests.make_db()
    ```

6. Run a local smtp debugging server:

    ```
    $ sudo python -m smtpd -n -c DebuggingServer localhost:25
    ```

7. Run the celery worker:

    ```
    $ python runcelery.py -A app.tasks worker
    ```

8. Run local server:

    ```
    $ python manage.py runserver --host <IP_ADDRESS>
    ```

## Features

1. Caching with Memcached

    ```bash
    from app.extensions import cache

    # cache something
    cache.set('some_key', 'some_value')

    # fetch it later
    cache.get('some_key')
    ```

2. Email delivery with Mailgun

    ```bash
    from app.extensions import mail
    from flask.ext.mail import Message

    # build an email
    msg = Message('User Registration'<EMAIL_ADDRESS>recipients=[user.email])
    msg.body = render_template('mail/registration.mail', user=user, token=token)

    # send
    mail.send(msg)
    ```

3. Asynchronous job scheduling with Celery & Redis

    ```bash
    from app.extensions import celery

    # define a job
    @celery.task
    def send_email(msg):
        mail.send(msg)

    # queue job
    send_email.delay(msg)
    ```

4.  Password security that can keep up with Moores Law

    ```bash
    from app.extensions import bcrypt

    # hash password
    pw_hash = bcrypt.generate_password_hash('password')

    # validate password
    bcrypt.check_password_hash(pw_hash, 'password')
    ```

6. Deploy on Heroku with ease

    ```bash
    # provision new stack
    $ heroku create

    # configure environment
    $ heroku config

    # deploy
    $ git push heroku master
    ```

7. Easily swap between multiple application configurations

    ```bash
    from app.config import dev_config, test_config
    app = Flask(__name__)

    class dev_config():
        DEBUG = True

    class test_config():
        TESTING = True

    # configure for testing
    app.config.from_object(test_config)

    # configure for development
    app.config.from_object(dev_config)
    ```

8. Form validation & CSRF protection with WTForms

    ```bash
    # place a csrf token on a form
    {{ form.csrf_token }}

    # then validate
    form.validate_on_submit()
    ```

10. Scale with Blueprints

    ```bash
    # app/user/__init__.py
    user = Blueprint('user', __name__, template_folder='templates')

    # app/__init__.py
    app = Flask(__name__)
    app.register_blueprint(user, url_prefix='/user')
    ```

11. Automated tests and continuous integration support for TravisCI

    ```bash
    # run the test suite
    python tests.py

    #from travis.yml
    coverage run tests.py
    ```

    ```bash
    # load the database with test data
    from tests import make_db
    make_db()
    ```

12.  Use any relational database using the SQLAlchemy ORM

    ```bash
    from app.user.models import User

    # fetch user by id
    user = User.get_by_id(id)

    # save current state of user
    user.update()

    # fetch a paginated set of users
    users = User.query.paginate(page, 50)
    ```

13. Merge and compress your javascripts and stylesheets

    ```bash
    # create a bundle of assets
    js = Bundle(
        'js/jquery.js',
        'js/bootstrap.min.js',
        filters='jsmin',
        output='gen/packed.js'
    )
    ```

    ```bash
    # serve up a single minified file
    {% assets "js_all" %}
        <script type="text/javascript" src="{{ ASSET_URL }}"></script>
    {% endassets %}
    ```

repository: nezaj/flask-bones

repo_name: nezaj/flask-bones