text:
#JOBS

DELETE THIS SECTION WHEN THE JOBS ARE DONE.

DOC
- THIS DOC HAS NOT BEEN SPELL CHECKED!
- UPDATE INPUT SECTION AFTER REWRITEING THE CODE WITH SPECIFICS ABOUT FORMATS
- would be good to add some images of analysis showing non-normal distributions etc

CODE
- AT END. ADD IN GOLBAL PRINT STATEMENTS TO TELL THE USER WHAT IS HAPPENING. GENERALLY I WANT TO MINIMISE THESE AND PUSH MOST PRINTING TO LOG FILE.
- AT END. CREATE A VERBOSE GLOBAL LOG FILE TO SHOW WHAT HAPPENED IN THE RUN
- WHY DOES EDGE WEIGHT COME UP AS 0 SOMETIMES OR None OTHERS? CHECK AND FIX
- IS EDGE WEIGHT BROKEN? I'M SEEING LOTS OF 0. MAKE SURE THIS ISNT A BROKEN CALCULATION.
- IS THERE A MORE ELEGANT WAY OF CHECKING IF A NODE HAS AN ATTRIBUTE OTHER THAN except KeyError:?
- ADD TIMESTAMP ENTITY TO NEO WHEN EACH STATS SCRIPT IS RUN
- NEED TO ADD A PARAMETER OPTION TO UPDATE CALCULATIONS EVEN IF THEY HAVE PREVIOUSLY BEEN MADE
- NEED TO BE ABLE TO SWAP BAD/GOOD FACET IN APP OR SELECT BOTH BAD. IF BOTH GOOD IT'S NO MERGE.
- NEED MANCLUSTER TO PRODUCE MEDIAN EXAMPLES OF CLUSTER SAMPLES TO AID DECISION MAKING.

## v1.1 Suggestions

- add ontology script as another early stage filtering mechanism. Hopefully this script will broadly pull pairs into the analysis pipeline.
- prevent missing data (reported in graph_make.py) by altering input.py order of data aquisition. Check this is the only issue here.
- add more progress bars

# BioSamples Attribute Curator Documentation

**A semi-automated curation tool to identify and harmonise erronious attributes in the BioSamples database.**

### Scope

The digital BioSamples database aims to aggregate sample metadata from all biological samples used throughout the world and serve as a central hub for linking and relating this metadata to biological data. To this end, it does not enforce information requirements, thereby lowering the barrier to entry for researchers; this also widens the scope of metadata collection. The downside to a zero validation design is the potential for low quality input which has a negative impact on search and information output. Therein lies the need for this software. As of October 2017 BioSamples has over 4.7\times10^6 samples sharing over 27,000 attributes and is quickly growing.

The scope of this software does not extend to curation of the values associated with these attributes but it does explore the value data as one potential way of comparing attributes. Here we aim to use a holistic semi-automated approach to clean up the attributes and merge those which have been provided in error. This includes harmonising various cases of lexical disparity including case differences, spelling errors and pluralisation). It also uses ontology comparisons (of attibutes and associated values) to explore merge oppotunities as well as various statistical distances (including attribute coexistance and lexical similarity scoring of value information).

After identification of possible curatable attribute pairs, we extrapolate clusters of samples to ensure curations are only applied to relevent samples. This improves our confidence and flexibility. Automated and semi-automated analysis are recorded and remembers in a clearly accesible way to avoid duplicate computation and provide data for correlation analysis (and other machine learning methods). This latter analysis should dramatically improve the power of this tool and is an essential component in it's design.

### Beyond Immediate Scope

* Ontology cleanup- although not directly considered this will inevitably be improved through use of this tool.
* Profiling of samples- although clustering methods herein could be adapted to meet this goal.
* Checking external links and references- this should be encorporated when possible (relates to profiling samples).
* Replacing value information- although the methods herein can be adapted this would ideally operate as a separate pipline as specific considerations differ.

## Scripts

### main.py

Controlling script. Imports other scripts. Decides when to generate new input. Launches analysis scripts in correct order. Amalgamates results. Stores and edits output files.

#### Organisation
Broadly the pipeline operates as such:

input -> pair filtering -> analysis -> clustering -> human input -> curation items

* input- generate input files required if they are old
* pair filtering- pairwise scripts that are fast enough to analyse every pair and suggest merge oppotunities^*
* analysis- including coexistance graph analysis and value matching distances. These scripts also generate pairwise distances but they tend to be slower. Therefore they are applied to a pre-trimmed list of pairs.
* clustering- samples affected by suggested merges are clustered to ensure homogeneity before merging. At this stage it is also possible to apply the curations to specific samples.
* human input- human curators are presented with various data to make merge decisions.
* curation items- the output is a list of sample specific attribute merges ready to be turned into curation items.

^* note that the lexical script does much more than filter results. It also XXXXXXXX

Scriptwise pipe:

input.py -> lexical.py -> coexistance.py -> values.py -> cluster.py

#### Output

### input.py

Generates inputs for pipeline and stores output into ./data as csv or json. This script makes 4 files:

1. attributes.csv
csv list of attributes and their frequency of occurance in the database

2. samples.csv
csv with sampleID folowed by all the attributes that sample contains

3. coexistences.json
list of all pairs that coexist in the same sample and the frequency at which they do so

4. values
list of dictionaries containing key:value pairs as attribute:[value], with values stored as a list

The 'attribute.csv' is generated quickly (XXXXXX time) as it only requires 1 Solr request. This file is only used for XXXXXX. 'samples.csv' is generated by iterative page requests through BioSamples v3 API and scrapes each page. This process is multithread enabled to speed up the process but this is still a process that takes longer than 8 hours. The temporary threads this process produces are concatinated to produce 'samples.csv'. The sampleID is then stripped from each line and the output is then used to feed the modules that count the coexistance which ultimately builds 'coexistences.json'.

This script then does some pre-processing of the data. It converts the json into csv format (coexistences.csv)

### graph_make.py

This pre-processing script uses __attributes.csv__ and __coexistences.json__ to calculate weighting and create a networkX graph. This output is stored in three files:

1. coexistences.csv
This is a csv converted directly from the json using re. While this may be prone to breaking and would ideally be replaced with a json reader, unfortunately there are issues due to the flat structure of the json file.
2. coexistencesProb.csv
Pandas is used to lookup attribute frequencies and calculte expected coexistance counts. These can be compared to expected coexistance counts to calculate a weight (see below for more details).
3. coexistences.gexf
NetworkX graph output suiltable for reading by Gephi (see _A note on Gephi_).

#### Why do we require probability weighting?

If two attributes coexist frequently within samples we may presume they are related and provide distinct information. We could conclude that these facets should not be merged becuase they provide distinct information (this can be confired by checking value data too). However, numerous attributes in the BioSamples Database (such as organism, synonym, model, package, organismPart and sampleSourceName) will coexist with each other in samples more frequently than less numerous ones. Therefore, to derrive the significance of the coexistance count we must normalise against individual attribute frequencies.

In order to do this we calculate the following steps:

1. probability of attribute = no of instances / total instances
1. expected coexistance count = probability of attribute1 * probability of attribute2 * total instances
1. difference = observed coexistance count - expected coexistance count
1. weight = difference / sum of differences

As BioSamples Data input is not randomly generated the vast majority of attributes do not contain any coexistance within samples and the vast majority that do have a positive difference (observed higher than expected). The graph is undirected even though Gephi insists on adding direction depending on which attribute is the 'source' or 'target' but this is only a visualisation bug. All weights add up to 1 (as per caculations outlined above) and the graph contains both positive and negative weights (negative when expected is higher than observed). These negative weights are often intuitively relevent (e.g. organismPart and serovar with a difference of -27689 or environmentBiome and organismPart with a difference of -89490) and highly positive weights are also intuitive (e.g. depth and elevation with a difference of 99617).

#### Missing value warnings.

Missing attributes may occur if the pair has an attribute that is not in attributes.csv. The number of missing pairs is recorded. To minimise this, attributes.csv is generated as the last step in the input.py. This should be checked to ensure significant data isn't missing. This process may need to be more stringent in later versions.

#### A note on using Gephi.

Gephi is an open-source free graph visualisation program for windows, mac and linux. Download at https://gephi.org

1. When you load this into Gephi you must first insist that the 'weight' column becomes the edge weight. This can be done in the Data Laboratory with the 'Copy Data to Other Column' button intuitively. I also add these values to the label column so that I can see the values if I request them in the graph.

2. I suggest starting by changing the size of the node to be equivelent to 'Degree' aka how many edges that facet has. I followed this method https://stackoverflow.com/questions/36239873/change-node-size-gephi-0-9-1

For general help see this: https://gephi.org/tutorials/gephi-tutorial-visualization.pdf

3. The next step is the layout. There are three relevent algorithms for expanding the nodes. Here are my observations on each, WARNING this is a very reductive description:

YifanHu(Proportional)

Seems the most intuitive so you can work out why things appear where they do. this is because the whole thing is a minimisation calculation based on the weights. Thats why its a circle, why big stuff tends to stay in the middle and the weakest linked stuff floats to the outer asteroid belt of junk. I reccomend starting with this because you can undertand what it is doing but it is not the best for getting discreet clusters.

OpenOrd

This does not give an intuitive result but it does make nice clusters for various reasons. It is supposedly the quickest but all these three work well enough with the dataset we have. It is great to watch it going through the various stages and would be perfect for a live demo.

ForceAtlas(2)

Default settings are nice and gives you somthing inbetween the two above. I changed a few parameters to get the largest nodes out the the edge of the screen so I could focus on smaller clusters in the middle. To do this switch on LinLog mode.

### lexical.py

Current output:

"merge_confirmed_"+TIMESTAMP+".csv"
"merge_after_manual_review_"+TIMESTAMP+".csv"

analysis pipeline currently takes:
'data/mergeHighConfidence.csv' and 'data/mergeLowConfidence.csv'

Trish's version:

needs:
* "../master-data/cocoa_facets.csv"
* "en_US","../master-data/biosamples-lexical-dictionary/lowercase_mywords.txt"

### coexistance.py

needs graph_make.py to be run first
input:
* query pairs from files starting with merge
* fold_diff_weighted_network.gexf

pulls in all csv starting with 'merge'

output:
mergeHighConfidence_stats.csv
mergeLowConfidence_stats.csv

I'm hoping I can iteratively add to these files as I feed them to each stats module.

has -r ability vs the usual continue mode
- need to add all its outputs here and what they mean

### values.py

- takes data/values.csv as input (currently I'm working with a temp file that Trish generated I need to build this with the input script)
- lots more info is calculated for the fuzzy score if needed!
- has -r ability vs the usual continue mode
- need to add all its outputs here and what they mean

### auto_cluster.py

This scipt performs clustering up to the point of k (cluster number) determination. Whilst various automated methods exist to determine k, they are unsuitable on the highly variable BioSample dataset. Therefore, this script aims to equip the user with as much information as possible to make a judgement on k. After k has been manually entered the mancluster.py script can then perform further calculations and provide information on individual clusters such as median sample representations of that cluster. This module requres the input data from 'samples.csv'. If this program is run in recalculate mode (-r argument passed) it will strip Sample-Cluster relationships that were previously calculated rather than skipping previously calculated Pairs. Therefore when ran in normal mode, the scipt will essentially pick up where it left off.

The clustering that can be done automatically prior to a user entered k includes:

* tally of no. of samples which have each or both attributes
* generation of a binary matrix (samples vs attribute presence/absence)
* 2D reduction (MCA) of the binary matrix and plotting of scatter graph
* hiarachical clustering and plotting dentrogram

N.B. When running in normal mode the script won't know if there are any missing samples as it will skip over any Pairs that have previously been calculated. Therefore, it is important to schedule this script in -recalculate mode to ensure extra samples are linked to on a regular basis.

#### The output of this script for each Pair in the Neo4j graph

Added to Pair node:
1. total no of samples in pair
1. no of samples in attribute 1
1. no of samples in attribute 2
1. no of shared samples in pair
1. file path for MCA scatter plot
1. file path for hiarachical clustered dendrogram

Relationships added:
1. Adds a direct relationship (HAS_ATTRIBUTE) from the Pair nodes to relevant Sample nodes. These are not removed when k has been determined in a later step.

The scatter plot from MCA is generated and saved in 'data/plots/...' for later recall along with the coordinates from the MCA analysis (files named mca_id.log). The log file is a csv with columns x and y coordinates as well as the frequency (s)of samples at that coordinates. These are samples with identical attributes.

#### Further work

- add more automated k methods aiming to identify a fully automated method.
- review dendrogram plot asthetics
- after k has been manually determined k-means clustering needs to work from xD data not 2D data hence autoclus needs to pass on the binary matrix?
- calculates the number of samples affected by the merge. this needs to be used to rank when human curators see facets.

### man_cluster.py

This script will only trigger when a user has entered a k. It should be able to run automatically but mainly on demand via the app. When k is entered this script should fire and produce results ASAP for that facet. If the user changes their mind it should trigger again. The automatic firing of this should update and mop up anything that is old or hasn't been calculated.

#### Choosing a clustering method

Everyone loves K-means but unfortiantely it is not an appropreate method for clustering dichotomous data. This is because the Euclidean distance will only be a count of the binary differences between samples. This means that inappropreate ties may occur which cannot be overcome in iterative cluster assignments. Hence, we shouldn't use k-means for clsutering binary data.

Appropriate alternatives are hierarchical clustering (as implemented by autocluster.py), two step clustering or spectral clustering. Or one can use MCA (which is better than PCA for binary data) to reduce the binary data into a number of dimensions which also serves to convert it from being binary (this is the first step of spectral clustering anyway). Then k-means is an appropreate clustering method too.

In summary, generally dimension reduction (via calculation of eigenvalues) is required before clustering binary data. This step is done in autocluster to 2D. More dimensions improve resolution but this hinders plotting. It is however useful when used as a backend to clustering.

### Neo4J Curation Database

This stores the output from all the scripts.

- pairs are sorted alphabetically prior to node creation to prevent duplication of reverse pairs.
- before I have the lexical script working I need to manually create the pairs from the lexical script to give me a platform to get the other scripts working.


text:
---
layout: website
title: Plays
menu-active: plays
permalink: /plays/
---

Why Hashitomi and Kokaji?

By Tom Hare

        The modern repertory of noh comprehends more than two hundred plays,
        depending on how one counts them, so what is it that makes these two
        particular plays, Kokaji and Hashitomi, the focus of
        this project?

        Books and electronic media on noh, in both Japanese and Western
        languages, have generally categorized their content according to the
        five-play system (goban-date) that noh troupes long ago adopted
        in their ordering of formal full-day programs: 1) “god-plays”, 2)
        “warrior-plays”, 3) “wig-plays” (or “third-category plays”), 4)
        “miscellaneous” or “fourth-category-plays” and 5) “demon plays” or
        “finale-plays”. These categories are not very precise. Not all the
        so-called demon plays are actually about demons. Some are about
        lower-ranking gods (maybe “daemon” would be a better way to translate
        the Japanese term) and although the “wig plays” are mostly focused on
        women, with actors in wigs, not all plays focused on women (even wearing
        wigs) are categorized as wig plays (and the main actor often wears a wig
        in other kinds of plays too). “Fourth-category plays are also called
        “miscellaneous plays,” judiciously allowing a catch-all group for plays
        that don’t comfortably fit anywhere else.

        Hashitomi is a wig play and Kokaji is generally
        considered a demon play or “fifth-category play, but in some cases it is
        considered a “fourth-category play.” The central figure in
        Kokaji is actually a god (or daemon) rather than a demon
        proper, which may account for its occasional placement in the fourth
        category.

        For all these inconsistencies, though, the goban-date system has had a
        productive role in structuring occasions of performance, dating back
        into the seventeenth century, if not earlier. But this was still not the
        earliest way of categorizing the repertory of noh, or as it was known
        then, “sarugaku.” In the late fourteenth century, a somewhat different
        typology existed, at least in the writings of Zeami (1363-1443), with
        gods, women, warriors and demons, to be sure, but also the deranged (in
        most cases, women), priests, Chinese “types”, and “roles in which the
        main character doesn’t wear a mask.” Perhaps that was simply the way
        this most prominent noh actor in all the history of noh thought about
        categories of sarugaku performance. But even Zeami wasn’t fully
        committed to such a schema, since he changed to a more abstract system
        of “Three Modes” (Old Man, Warrior, and Woman) in the latter decades of
        his life.

        The choice of Hashitomi and Kokaji for this site
        doesn’t slot neatly into either of the schemes outlined above, but for
        all that, it provides an instructive perspective on all of them, with
        ties to the historical development of sarugaku into noh as well. Before
        we can compare the plays in this context, though, we should look at them
        individually, to highlight their particular characteristics, both
        typical and unusual. Neither play is mentioned frequently or prominently
        in the historical record of noh, so much of our evidence must come
        internally, from the plays themselves, but that is the best way to
        understand their structure and content anyway.

Hashitomi (or Hajitomi)

        As is often the case with noh, neither Hashitomi nor
        Kokaji can be confidently attributed to a particular author or
        composer. One of the lists of the noh repertory compiled in the
        seventeenth-century claims that Zeami wrote Hashitomi, but this
        is highly unlikely. The author named in other lists of roughly the same
        time claims the play was written by one Naitō Saemon, or Naitō Tōzaemon,
        or Ninagawa; little is known about these figures apart from similar
        references to the authorship of two other noh plays by the said Naitō,
        so perhaps he was the writer of Hashitomi after all.[^1] But
        even if we cannot be fully confident in such an identification, the
        question of authorship nonetheless affords us a good perspective on not
        only Hashitomi and its reception but, more broadly, on the
        solidification of performance conventions in noh, inter-arts comparisons
        and what has been termed the “folklore” of The Tale of Genji.

        The name Naitō Saemon or Tōzaemon itself suggests lineage in a military
        family, and is very different in character from the names of
        professional noh actors we can identify from the fourteenth and
        fifteenth century. It is likely that as a noh or sarugaku playwright,
        Naitō (or whoever composed Hashitomi) was a talented amateur,
        and the play bears various marks of the amateur, not just this name, as
        we shall see. And we should say from the start that “amateur” here is
        not to be taken negatively.

        If Hashitomi bears a number of hallmarks of amateur
        composition, this makes it even more amenable to the identification of
        general characteristics of wig plays, even as it illustrates the
        historical development of noh in the fifteenth century more clearly than
        some of the more celebrated and intricately constructed plays of
        professionals like Zeami.

        The central character of Hashitomi, for instance, is not
        readily identifiable. She bears certain associations with the famous
        Lady of the Evening Faces, Yūgao no Ue, in The Tale of Genji, but she
        could also be understood to be the spirit of the so-called yūgao plant,
        a type of bottle gourd with a white blossom. Such a spread of
        associations comes about primarily because of the choice of poems and
        other passages from Genji quoted, and sometimes altered, in the play.

        The quotations in question come from a variety of contexts in Genji. One
        is a sophisticated comparison between the moon and a departing lover,
        figuring the lady who longs for him on his departure to the mountain
        horizon. [^2] Another requests that pinks (or dianthus flowers) deign to
        cast a thought, in the form of a dewdrop, onto the stakes of a
        dilapidated fence: this is an elaborate metaphor asking a highborn lord
        to look in upon his little daughter, even though her now dead mother was
        of undistinguished rank. [^3] Both of these allude to important plot
        points early in the narrative of Genji, in connection with the ill-fated
        Lady of the Evening Faces, but both need the context of the tale itself
        to fill out the incidents in question. That, though, does not seem to be
        what the writer of Hashitomi intended. And yet, he was not just
        a magpie, picking up a snatch of poem here and a snatch of poem there.
        He was aiming for a different kind of sophisticated effect, if we are to
        judge from his use of a third poem taken from Genji. This poem appears
        in the play at the most conspicuous place possible, right after the
        shite’s jonomai at the play’s climax. Or, to be more precise, we should
        say that the poem sandwiches the dance, because the first line of it
        (altered in an interesting way) comes before the dance and then, once
        the dance is finished, the full poem is sung. In Genji, too, the poem in
        question comes at a crucial place; there, it serves as the pivot from
        simple curiosity to a fatal romantic encounter. The poem reads:

            Yorite koso sore ka to mo mime tasokare ni

            honobono mitsuru hana no yūgao

            Who is it, in the deepening dusk?

            Only by going up close might you see just who —

            the white gourd blossom

            fading out of view.

        In Genji, this poem comes in response to another poem, on a slip of
        paper attached to a real yūgao blossom:

            Kokoro-ate ni sore ka to zo miru shiratsuyu no

            hikari soetaru yūgao no hana

            Venturing a guess,

            it looks to me just like “the very one”:

            light suffuses the white dewdrops

            on the blossoms of white gourd.

        This episode in Genji is one of its most characteristic, pairing an
        elite teenage male, a player in the erotic competition of the classical
        romance, with a beauty bereft of the political and social connections
        needed to maintain her position, any position, in the society of the
        novel. The initial vector for contact is classical poetry, ostensibly
        about flowers, and these two poems allow the two lovers-to-be to break
        the ice and thus to further the plot of the novel. The hidden female
        speaker of kokoro-ate ni is hinting that she understands it is
        the Shining Prince, Genji himself that passes by her cottage, with its
        yūgao flowers, and Genji, in his response, begs an invitation
        to get a closer look.

        The poems also draw on a broader cultural matrix, that of an idealized
        classical culture of the royal court. In addition to the figural use of
        the gourd blossom, which links the Lady of the Evening Faces with Genji,
        the cultural associations of the flower, as codified and thematized in
        the practice of court poetry, create a rich world of associations. They
        also set this world in soft focus, by ambiguity and interpretive
        polyvalency. The word tasogare, “dusk”, for instance, provides
        the mise-en-scène for the gourd to blossom — it blooms at nightfall, and
        each blossom perdures for but a single night. This foreshadows the
        brevity of a tragically brief love affair at its very inception. It also
        impresses the time with an emotional character: if tasogare is
        dusk, it is the time when one begins to fail to recognize passers-by.
        Indeed, ta so gare (or kare)” in ancient Japanese
        means “Who is that?”

        This grounding of the poem is used to specific ends in the novel, but
        when we consider the use of this and other poems from the classical
        tradition in Hashitomi, we find that those are not necessarily
        the same ends.

        There is, for instance, a crucial single-syllable change in the way the
        play quotes the Genji verse. Instead of yorite koso, “Only by
        going up close”, the play reads orite koso, which means “only
        by breaking/snapping off”. That could probably be construed to mean
        “take someone captive, romantically (or abusively),” but it is much more
        prominently a word used of gathering sprigs or branches of flowers in
        bloom. This interpretation fits easily into the general aesthetic
        context of Hashitomi, which is occasioned, according to the
        words of the waki in the beginning of the play, by a Buddhist service
        propitiating the flowers that adorn ritual altars (the
        rikka kuyō). With this reference, the play alludes to an
        intellectual debate popular in medieval Japan in which the boundaries of
        sentiency, and thus, of the potential for enlightenment or salvation,
        were disputed. The debate is not fully articulated in the play, but it’s
        clear enough that the composer felt plants and trees were sentient
        beings, and deserving of reverence. In this, he was of the same opinion,
        apparently, as a number of other noh playwrights, including the most
        celebrated.

        With this, then, we begin to see Hashitomi in a somewhat
        broader context than merely that of an elegant tableau vivant lifted
        from an old tale. It then reveals links not only to Buddhist thought
        about the natural world, but also to an important inter-arts connection
        with the art of flower arranging.

        This connection is, in fact, given material expression as the raison
        d’être for a variant performance style for the play, in all five modern
        schools. That variant style, or kogaki specifies a particular
        prop that is not used in more conventional performances. The prop is a
        very large, very formal flower arrangement in an ancient style called
        rikka or “standing flowers”, quoted directly from Buddhist
        antecedents. This arrangement is stationed center-stage for the first
        part of the play when it is performed according to this variant, and it
        adds a beautiful and conspicuous supplement to the already heavily
        floral character of the play.

        There isn’t a clear documentary trail that would tell us when the play
        was first done in this manner, but in that it straddles the worlds of
        noh performance and flower arranging, it points to the aesthetic
        cultivation of a time when amateurs could compose noh plays, and have
        them performed, and integrate them with other overt expressions to their
        cultural capital and inter-arts sophistication. And it is not merely in
        the context of noh and flower arranging that we can see this in
        examining Hashitomi. In addition, we can reconsider the way the
        play deploys the several classical Japanese poems it quotes. If they are
        not used to reimagine a scene from classic literature (as other famous
        noh plays such as Izutsu or Atsumori do), they are nonetheless taken
        from self-conscious and serious aesthetic aspirations. If the poems in
        question don’t all come from a particular narrative context in Genji,
        they nonetheless share a common theme. The poem I mentioned earlier,
        about “the pinks . . . casting a thought, in the form of a dewdrop, onto
        the stakes of a dilapidated fence,” though not from the “Lady of the
        Evening Faces” episode in Genji, nonetheless comes from a foreshadowing
        of that episode in the previous chapter of the tale. It seems very
        likely — and here we would follow Janet Goff in her fine study of noh
        based on Genji — that the composer of Hashitomi might have
        discovered his thematic material for the play in the handbooks created
        for practitioners of linked verse (many of them amateurs), where
        appropriate tags from classical poetry and narrative, were collected and
        arranged topically for the use of latter-age versifiers.

        One of the best known of those handbooks, Renju gappekishū, indeed,
        quotes the two poems from the encounter in Genji with the same variant
        first line of the response poem, orite koso, that is cited in
        Hashitomi. [^4]

        One sees a similar propensity to playing a game of poetic allusion in
        some of the Chinese verses quoted in Hashitomi. Those verses
        come mostly from a collection of both Chinese and Japanese verse put
        together in the eleventh century, Wakan rōeishū, and seem less
        successfully integrated into the play than the Genji verses. All the
        same, they point to the same propensity for canvassing a broad
        inter-arts spectrum in order to compose the play.

        The title Hashitomi (or Hajitomi) has not been the
        exclusive title of this play in such documentation as exists since the
        seventeenth century. It has occasionally been known as Yūgao or Yūgao no
        Ue, provoking some confusion because there is another play of the same
        name in the repertory. Modern scholarly consensus has distinguished the
        two plays, leaving the other with the name Yūgao, pointing as well to
        the latter play’s more consistent and narrative-oriented use of source
        poems from The Tale of Genji. For this reason, the possibility has been
        mooted that that play was created by Zeami or at least by someone
        faithful to the guidelines he articulated.

        That said, though, one can still identify in Hashitomi many of
        those same guidelines: the use of familiar poems from old classics is
        itself something Zeami recommended, and the focus on dance, with the
        fully orthodox
        jonomai
        of the final act sandwiched between lines from the central poem in the
        play is a common feature of many of the classic plays in the repertory.
        (And although the professional actors and playwrights of the fifteenth
        century cannot be directly credited with the jonomai, they do
        seem to have deployed its ancestor, the tennyo no mai as early
        as the 1420s.)

        From this perspective, Hashitomi differs considerably from
        Kokaji. Although we have even less information on who composed
        Kokaji than Hashitomi, and although we have no
        documentary evidence to show that it is an old play (of, say, the late
        fourteenth or early fifteenth century). It nonetheless represents an
        older approach to noh, or sarugaku performance, than Hashitomi,
        in that it is a miracle play: the claims of sarugaku tradition hold that
        it originated in the miraculous manifestation of a native deity at the
        foot of a pine tree, the Pine of Divine Revelation (yōgō no matsu) in the ancient capital Nara. Many of the oldest plays in the
        repertory show features of this legend. Some of them are classed as god
        plays according to the goban-date system I mentioned at the
        beginning of this essay. Others, though, often more interesting, are
        plays such as Kokaji, which fall into the final category, the
        so-called “demon” plays. The shite in Kokaji, however, is no
        demon, but rather the god Inari Myōjin, whose avatar is a fox.

      <h3 id="plays-Kokaji">Kokaji

        Something that often distinguishes “god plays” from “demon plays” in the
        five-genre system is interest. A few of the god plays are wonderful, but
        many others are rigidly formal and austere (not to say boring). The
        “demon plays” are never boring, and that is in part because of a plot.
        It’s not a complicated multi-layered plot usually, but there is a
        “completion of action” in such a plot: a task is set, as in
        Kokaji, and, despite challenging obstacles, the task is
        accomplished. Often, as in Kokaji, the accomplishment is due at
        least in part to the aid of a god, a very deus-ex-machina.

        The task in Kokaji is the forging of a sword worthy of royal
        commission. The obstacle is the insistence, on the part of the best
        sword smith in the land, Munechika, that he must have an equally skilled
        partner in order to hammer such a sword into perfection. There is no one
        the Munechika can call upon, so he prays to the god Inari. Then, rather
        abruptly, a mysterious figure (wearing the mask of a beautiful boy)
        calls out to him. In a session that is part coaching and part
        annunciation, the boy encourages Munechika to make preparations to forge
        the sword, doing so with recourse to Japanese proverbs and sententious
        Chinese verse.

        In a long formal narrative (the kuri-sashi-kuse sequence), the
        boy recounts tales from both China and Japan, of illustrious swords. (No
        pressure!) Then, without more ado, he instructs Munechika to prepare the
        forge, both physically and ritually, only to disappear into the “evening
        clouds of Mt. Inari.” We can already surmise that this story is going to
        have a happy ending, but that doesn’t keep another actor, this time a
        kyōgen actor, from coming on stage as a minor local deity, to rehearse
        the plot once again.

        So now to the accomplishment of the task: The plot is resolved in our
        minds, but the excitement of the play is really just getting off the
        ground, for now there is a sequence of musical pieces, some
        instrumental, some with chorus accompanied by the ensemble, during which
        the god Inari manifests himself to his human partner Munechika. This
        appearance of a divine figure echoes the miraculous origins of sarugaku
        itself, but in this case the appearance is only part of the miracle, for
        it takes material form in the sword that Munechika forges with his godly
        partner. Appropriately, the god’s costume will coruscate over metallic
        thread or appliqué, the mask will arrest our attention with a daemonic
        intensity, and to top it all off, we will find a huge red wig crowned
        with a glittering circlet topped, in turn, with a shining image of a
        fox. This is what we came for, the appearance of a god, or a demi-god,
        and the accomplishment of a great task.

        The sequence of instrumental pieces that mark the last act of the play,
        the raijo, the hayafue and the maibataraki,
        is a tour-de-force, and marks off a special group of plays in the noh
        repertory as of particular excitement. These plays often bring a full
        day’s performance to an end. They are flashy and satisfyingly simple (to
        enjoy, if not to perform). They’ve been a staple of sarugaku performance
        since the late fourteenth-century and have their roots in the oldest
        identifiable traditions of the modern schools of noh.

        With such archaic roots, miracle plays like Kokaji represent a
        kind of Ur-noh. They are discernible in the oldest traditions of the
        Yamato troupes that are the lineal ancestors of modern noh performance,
        and although Kokaji itself may not be that old, it is
        representative of an aesthetic of simple plots with definitive and
        felicitous conclusions. When we look back at Hashitomi from
        such a perspective, we are struck by now different it is, how the
        conclusion is anything but definitive — we don’t really know if the
        central character we’ve been watching for an hour or so is the exquisite
        ghost of the court of Genji or, rather, a transcendent manifestation of
        a gourd blossom. She evaporates as if in a dream.

        Her costume will be very beautiful, no doubt, and her mask perhaps
        virtuosically executed, but nonetheless generic. And on top of that,
        unlike Kokaji, with its two masks, Hashitomi requires
        only one. Viewers are likely as not going to be drawn to the fragile
        edifice of a gourd-flower arbor that is the main prop on stage in
        typical performances, as much as they are to the actors themselves. Or
        to an elaborate sculpture of blossoms that is then to be situated center
        stage — with the full acquiescence of the actors — for the entire first
        act of the play, when the play is performed in that special variant
        rikka style. You might say that the visual aesthetic of the
        play is diffused, like the identity of the shite, over the entire stage.
        In this sense, Hashitomi is more a play for expert members of
        the audience, than is Kokaji. As we have seen, it was likely
        composed by an amateur, someone interested not just in noh, but also
        flower arranging, someone eager to show off his knowledge of poems from
        The Tale of Genji, perhaps an aficionado of linked verse with a taste
        for Chinese learning as well.

        All these features of Hashitomi make it an excellent resource
        for investigating the value-added aesthetic of plays written to contrast
        with those about gods and demons from the old Yamato tradition. These
        newer plays are more complex thematically and formally more
        self-conscious than those. They were incubated in the aesthetic world of
        the early to mid- fifteenth century, and exhibit an interest in mystery
        and subtlety, often discussed using the term yūgen. Although
        the Yamato tradition of performance lives on in Hashitomi as
        well as in Kokaji, it is transformed by the influence of other
        approaches to performance associated with the area near Lake Biwa,
        closer to Kyoto than the Yamato troupes, as well as by the example of
        mature classic plays such as Izutsu and Matsukaze.

        That these two plays bear the hallmarks of different aesthetic,
        geographical and ideological strains makes them a particularly
        instructive pair for study.

* Footnotes must be added below (see https://github.com/sul-cidr/noh/wiki/Level-0-HTML-components#footnotes-using-markdown-feature)
{:footnotes}
[^1]: The play Shunzei Tadanori, which remains in the Noh repertory, as well as the play Kodama Ukifune, which has fallen out of performance, have been attributed to Naitô.  (Goff, p. 186.)
[^2]: Yama no ha no kokoro mo shirade yuku tsuki wa uwa no sora nite kage ya taenan
“Will the moon go off into the sky and fade from sight, all unawares of the longing heart at the mountain’s edge?“
[^3]: Yamagatsu no kakiho aru to mo oriori ni aware wa kakeyo nadeshiko no tsuyu “Though the stakes in this rustic fence are bent this way and that from time to time, allow your thoughts to settle down upon it, dewdrops on the pinks.“
[^4]: There is another variation in the Noh play from the poem as quoted in Genji. This one is less significant in that it merely changes honobono mitsuru to honobono mieshi. I translated the former version “fading out of view”, but a more explicit version would be something like “which one saw vaguely.” Honobono mieshi has less explicit agency, and could be rendered, “which appeared vaguely.”

license: MIT

repository: sul-cidr/noh

branch_name: refs/heads/develop

extension: md

gha_language: HTML

gha_license_id: MIT

license_type: permissive

repo_name: sul-cidr/noh


text:
[Back to Plugins Menu](..)
# Using Sagemaker with Flyte - Sagemaker plugin

## Prerequisites
Before following this example, make sure that
- SageMaker plugins are [enabled in flytepropeller's config](https://github.com/lyft/flytepropeller/blob/f9819ab2f4ff817ce5f8b8bb55a837cf0aeaf229/config.yaml#L35-L36)
- You have your [AWS role set up correctly for SageMaker](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.html#sagemaker-roles-amazonsagemakerfullaccess-policy)
- [AWS SageMaker k8s operator](https://github.com/aws/amazon-sagemaker-operator-for-k8s) is installed in your k8s cluster

## Contents
1. [Train & Optimize a Builtin Algorithm](builtin)
1. [Train & Optimize a Custom algorithm](custom)

repository: nuclyde-io/flytesnacks

branch_name: refs/heads/master

repo_name: nuclyde-io/flytesnacks


text:
---
categories:
- Machine Learning
- AndrewNg
date: 2019-09-26 16:59:34
tags: Machine Learning
title: 过拟合
---

# Notes of Andrew Ng’s Machine Learning —— (7) Overfitting

## The Problem of Overfitting

Consider the problem of predicting $y$ from $x \in \R$. The leftmost figure below shows the result of fitting a $y=\theta_0+\theta_1x$ to a dataset. We see that the data doesn't really lie on straight line, and so the fit is not very good.

![img](https://tva1.sinaimg.cn/large/006y8mN6ly1g7aopbltb1j30f0046dg2.jpg)

Instead, if we had added an extra feature $x^2$, and fit $y=\theta_0+\theta_1+\theta_2x^2$, then we obtain a slightly better fit to the data (See middle figure).

Naively, it might seem that the more features we add, the better. However, there is also a danger in adding too many features: The rightmost figure is the result of fitting a $5^{th}$ order polynomial $y = \sum_{j=0} ^5 \theta_j x^j$. We see that even though the fitted curve passes through the data perfectly, we would not expect this to be a very good predictor for unseen examples.

The figure on the left shows an instance of **underfitting**—in which the data clearly shows structure not captured by the model

> **`Underfitting`**, or *high bias*, is when the form of our hypothesis function h maps poorly to the trend of the data. It is usually caused by a function that is too simple or uses too few features.

The figure on the right is an example of **overfitting**.

> **`Overfitting`**, or *high variance*, is caused by a hypothesis function that makes accurate predictions for examples in the training set, but it does not generalize well to make accurate predictions on new, previously unseen examples. It is usually caused by a complicated function that creates a lot of unnecessary curves and angles unrelated to the data.

`Overfitting` is applied to both linear and logistic regression. There are two main options to address the issue of overfitting:

1) Reduce the number of features:

- Manually select which features to keep.
- Use a model selection algorithm (studied later in the course).

2) Regularization

- Keep all the features, but reduce the magnitude of parameters $\theta_j$.
- Regularization works well when we have a lot of slightly useful features.

## Cost Function

If we have overfitting from our hypothesis function, we can reduce the weight that some of the terms in our function carry by increasing their cost.

Say we wanted to make the following function more quadratic:
$$
\theta_0 + \theta_1x + \theta_2x^2 + \theta_3x^3 + \theta_4x^4
$$

We'll want to eliminate the influence of $\theta_3x^3$ and $\theta_4x^4$ . Without actually getting rid of these features or changing the form of our hypothesis, we can instead modify our **cost function**:
$$
min_\theta\ \dfrac{1}{2m}\sum_{i=1}^m (h_\theta(x^{(i)}) - y^{(i)})^2 + 1000\cdot\theta_3^2 + 1000\cdot\theta_4^2
$$

We've added two extra terms at the end to inflate the cost of $\theta_3$ and $\theta_4$. Now, in order for the cost function to get close to zero, we will have to reduce the values of $\theta_3$ and $\theta_4$ to near zero. This will in turn greatly reduce the values of $\theta_3x^3$ and $\theta_4x^4$ in our hypothesis function. As a result, we see that the new hypothesis (depicted by the pink curve) looks like a quadratic function but fits the data better due to the extra small terms $\theta_3x^3$ and $\theta_4x^4$.

![img](https://tva1.sinaimg.cn/large/006y8mN6ly1g7aq081c3hj30gh091wfl.jpg)

We could also *regularize all of our theta parameters* in a single summation as:

$$
\mathop{min}\limits_{\theta} \dfrac{1}{2m}\ \sum_{i=1}^m (h_\theta(x^{(i)}) - y^{(i)})^2 + \lambda\ \sum_{j=1}^n \theta_j^2
$$

The $\lambda$, or lambda, is the **regularization parameter**. It determines how much the costs of our theta parameters are inflated.

Using the above cost function with the extra summation, we can smooth the output of our hypothesis function to reduce overfitting.

Note. If lambda is chosen to be too large, it may smooth out the function too much and cause underfitting.

## Regularized Linear Regression

We can apply regularization to both linear regression and logistic regression.

### Gradient Descent

We will modify our gradient descent function to separate out $\theta_0$ from the rest of the parameters because we do not want to penalize $\theta_0$.
$$
\begin{array}{l}
Repeat\quad\{\\
\qquad \theta_0:=\theta_0-\alpha\frac{1}{m}\sum_{i=1}^m\big(h_\theta(x^{(i)})-y^{(i)}\big)x_0^{(i)}\\
\qquad \theta_j:=\theta_j-\alpha\Big[\Big(\frac{1}{m}\sum_{i=1}^m\big(h_\theta(x^{(i)})-y^{(i)}\big)x_j^{i}\Big)+\frac{\lambda}{m}\theta_j\Big]\qquad j\in\{1,2,\cdots,n\}\\
\}
\end{array}
$$
The term $\frac{\lambda}{m}\theta_j$ performs our regularization. With some manipulation our update rule can also be represented as:
$$
\theta_j:=\theta_j(1-\alpha\frac{\lambda}{m})-\alpha\frac{1}{m}\sum_{i=1}^{m}\Big(h_\theta(x^{(i)})-y^{(i)}\Big)x_j^{(i)}
$$
The first term in the above equation, $(1-\alpha\frac{\lambda}{m})$ will always be less than $1$. Intuitively we can see it as reducing the value of $\theta_j$ by some amount on every update. Notice that the second term is now exactly the same as it was before.

### Normal Equation

Now let's approach regularization using the alternate method of the non-iterative normal equation.

To add in regularization, the equation is the same as our original, except that we add another term inside the parentheses:
$$
\begin{array}{l}
\theta=(X^TX+\lambda \cdot L)^{-1}X^Ty\\
\textrm{where}\quad L=\left[\begin{array}{cccc}
0\\
&1\\
&&1\\
&&&\ddots\\
&&&&1
\end{array}\right]
\end{array}
$$
$L$ is a matrix with $0$ at the top left and $1$'s down the diagonal, with $0$'s everywhere else. It should have dimension $(n+1)×(n+1)$. Intuitively, this is the identity matrix (though we are not including $x_0$), multiplied with a single real number λ.

Recall that if $m<n$, then $X^TX$ is non-invertible. However, when we add the term $\lambda\cdot L$, then $X^TX+\lambda\cdot L$ becomes invertible.

## Regularized Logistic Regression

We can regularize logistic regression in a similar way that we regularize linear regression. As a result, we can avoid overfitting. The following image shows how the regularized function, displayed by the pink line, is less likely to overfit than the non-regularized function represented by the blue line:

![img](https://tva1.sinaimg.cn/large/006y8mN6ly1g7axj9vr5xj30do07edh2.jpg)

Recall that our cost function for logistic regression was:
$$
J(\theta) = - \frac{1}{m} \sum_{i=1}^m \Big[ y^{(i)}\ \log \big(h_\theta (x^{(i)})\big) + (1 - y^{(i)})\ \log \big(1 - h_\theta(x^{(i)})\big) \Big]
$$

We can regularize this equation by adding a term to the end:
$$
J(\theta) = - \frac{1}{m} \sum_{i=1}^m \large[ y^{(i)}\ \log (h_\theta (x^{(i)})) + (1 - y^{(i)})\ \log (1 - h_\theta(x^{(i)}))\large] + \frac{\lambda}{2m}\sum_{j=1}^n \theta_j^2
$$

The second sum, $\sum_{j=1}^n \theta_j^2$ **means to explicitly exclude** the bias term, $\theta_0$. I.e. the $\theta$ vector is indexed from $0$ to $n$ (holding $n+1$ values, $\theta_0$ through $\theta_n$), and this sum explicitly skips $\theta_0$, by running from $1$ to $n$, skipping $0$. Thus, when computing the equation, we should continuously update the two following equations:

![img](https://tva1.sinaimg.cn/large/006y8mN6ly1g7axue4ykpj30de06h75k.jpg)

repository: cdfmlr/clownote

gha_language: Stylus

repo_name: cdfmlr/clownote


text:
## AWS Encryption SDK for Javascript

The AWS Encryption SDK for Javascript provides a fully compliant,
native Javascript implementation of the [AWS Encryption SDK](https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/introduction.html)

This library is currently under public preview. Feel free to check out the code and give it a spin, but be aware that the APIs are still in flux. We'd love to hear your feedback on the APIs before they're fully nailed down.

## Client Packages

| Package | Description |
|:--------|:------------|
| [@aws-crypto/client-browser](https://npmjs.com/package/@aws-crypto/client-browser) | Client SDK for **Web applications** |
| [@aws-crypto/client-node](https://npmjs.com/package/@aws-crypto/client-node) | Client SDK for Node.js client applications |

These client packages have everything you need to encrypt/decrypt.
They are the primary starting point.
The AWS Encryption SDK for Javascript is built from a group of modularized packages.
You can also compose the functional packages you need.

### Functional Packages

| Package | Description |
|:--------|:------------|
| [@aws-crypto/encrypt-browser](https://npmjs.com/package/@aws-crypto/encrypt-browser) | Encrypt function for **Web applications** |
| [@aws-crypto/encrypt-node](https://npmjs.com/package/@aws-crypto/encrypt-node) | Encrypt function for Node.js client applications |
| [@aws-crypto/decrypt-browser](https://npmjs.com/package/@aws-crypto/decrypt-browser) | Decrypt function for **Web applications** |
| [@aws-crypto/decrypt-node](https://npmjs.com/package/@aws-crypto/decrypt-node) | Decrypt function for Node.js client applications |
| [@aws-crypto/kms-keyring-browser](https://npmjs.com/package/@aws-crypto/kms-keyring-browser) | Kms keyring for **Web applications** |
| [@aws-crypto/kms-keyring-node](https://npmjs.com/package/@aws-crypto/kms-keyring-node) | Kms keyring for Node.js client applications |
| [@aws-crypto/raw-rsa-keyring-browser](https://npmjs.com/package/@aws-crypto/raw-rsa-keyring-browser) | Raw RSA keyring for **Web applications** |
| [@aws-crypto/raw-rsa-keyring-node](https://npmjs.com/package/@aws-crypto/raw-rsa-keyring-node) | Raw RSA keyring for Node.js client applications |
| [@aws-crypto/raw-aes-keyring-browser](https://npmjs.com/package/@aws-crypto/raw-aes-keyring-browser) | Raw AES keyring for **Web applications** |
| [@aws-crypto/raw-aes-keyring-node](https://npmjs.com/package/@aws-crypto/raw-aes-keyring-node) | Raw AES keyring for Node.js client applications |
| [@aws-crypto/caching-materials-manager-browser](https://npmjs.com/package/@aws-crypto/caching-materials-manager-browser) | Caching Materials Manager for **Web applications** |
| [@aws-crypto/caching-materials-manager-node](https://npmjs.com/package/@aws-crypto/caching-materials-manager-node) | Caching Materials Manager for Node.js client applications |

## Concepts

There are four main concepts that you need to understand to use this library:

### Cryptographic Materials Managers

Cryptographic materials managers (CMMs) are resources that collect cryptographic materials
and prepare them for use by the Encryption SDK core logic.

An example of a CMM is the default CMM,
which is automatically generated anywhere a caller provides a keyring.
The default CMM collects encrypted data keys from it's keyrings.

An example of a more advanced CMM is the caching CMM,
which caches cryptographic materials provided by another CMM.

### Keyrings

Keyrings use wrapping keys to to generate, encrypt, and decrypt data keys.
The keyring that you use determines the source of the unique data keys that protect each message,
and the wrapping keys that encrypt that data key.
An example of a keyring is the `KmsKeyringNode`.

An example of a more advanced keyring is the multi keyring.
A multi keyring can be used to compose keyrings together.

### Wrapping Keys

Wrapping keys are used to protect data keys.
An example of a wrapping key is a `KMS customer master key (CMK)`_.

### Data Keys

Data keys are the encryption keys that are used to encrypt your data.
If your algorithm suite uses a key derivation function,
the data key is used to generate the key that directly encrypts the data.

## test

```sh
npm test
```

## License

This SDK is distributed under the
[Apache License, Version 2.0](http://www.apache.org/licenses/LICENSE-2.0),
see LICENSE.txt and NOTICE.txt for more information.


text:
# Gallery pages
[![Netlify Status](https://api.netlify.com/api/v1/badges/0500c8ed-62f3-439a-a387-f116b61f7ed7/deploy-status)](https://app.netlify.com/sites/shixie-projects/deploys)

Simple pages with a responsive gallery component for showcasing projects. Credit goes to [simonyiszk](https://github.com/simonyiszk/schdesign-web) for original build.

## Setup

- Clone this project and install all the required dependencies using [`yarn`][yarn]
- Start a development server with `yarn develop`

- _Other scripts like `build`, [`format`](#automatic-code-formatting), [`type-check`](#static-type-checking) and [`lint`](#linting) are also available_

repository: shixiedesign/gallery-pages

gha_language: TypeScript

repo_name: shixiedesign/gallery-pages


text:
# useResponsive

---

**This hook has been migrated to `ahooks`: https://ahooks.js.org/hooks/dom/use-responsive**

---

React hook for responsive design.

## API

```typescript
  interface ResponsiveConfig {
    [key: string]: number;
  }
  interface ResponsiveInfo {
    [key: string]: boolean;
  }

  function configResponsive(config: ResponsiveConfig): void
  function useResponsive(): ResponsiveInfo
```

## The `useResponsive` Hook

Use `useResponsive` to get responsive info in components.

```javascript
function App() {
  const info = useResponsive() // info is {"xs":true,"sm":true,"md":true,"lg":false,"xl":false}
  return (
    <div>
      {info.md && <p>Window width >= middle</p>}
      {info.lg && <p>Window width >= large</p>}
    </div>
  )
}
```

## Config

The default responsive config is the same as Bootstrap's breakpoints:

```javascript
{
  'xs': 0,
  'sm': 576,
  'md': 768,
  'lg': 992,
  'xl': 1200,
}
```

If you wish to config your own breakpoints, you can use `configResponsive`:

(Attention: You only need to set your config once, so don't call this function in components.)

```javascript
configResponsive({
  'xs': 0,
  'sm': 576,
  'md': 768,
  'lg': 992,
  'xl': 1200,
})
```

repository: Esbruiz/use-responsive

repo_name: Esbruiz/use-responsive


text:
![Mollie](https://www.mollie.com/files/Mollie-Logo-Style-Small.png)

### Changelog

All notable changes to this project will be documented in this file.

#### v1.4.1 - 2016-06-20
  - Added `refunded?` method.

#### v1.4.0 - 2016-06-16
  - Added Subscriptions API.

#### v1.3.0 - 2016-04-14
  - Added Mandates API.
  - Added `recurringType` and `mandateId` to Payments API.

#### v1.2.2 - 2016-03-21
  - The members `customers` and `customers_payments` weren't made public within `attr_reader` ([#21](https://github.com/mollie/mollie-api-ruby/pull/21)).
  - Updated the bundled CA's cacert.pem file to date `Jan 20, 2016`.

repository: richardvenneman/mollie-api-ruby

repo_name: richardvenneman/mollie-api-ruby


text:
---
layout: post
title:  "Eat that cake!"
date:   2014-08-28 22:47:00
comments: true
categories: design
tags:
    - design pattern
    - dependency injection
    - scala
summary: "We know it, managing dependencies between software components within a project of magnitude
          rather than trivial could become a hell if not planned properly Scala provides a mechanism
          to resolve dependencies similar to DI, the *cake pattern*. Unlike DI pattern in other languages,
          such as Java, C++, etc ... the cake pattern can be used directly with language constructs
          available in Scala."
social-share: true
social-title: "Eat that cake! Cake pattern explained"
social-tags: "scala, programming"
---
## Introduction
We know it: managing dependencies between software components within a project of magnitude rather
than trivial could become a hell if not planned properly. Many of you will wonder why it is necessary
to dwell too much on this topic. Responses to this question are two, namely:

1. low grade of coupling;
2. high grade of cohesion.

The minimization of dependencies between software components enables greater maintainability, and
enhances software testability, cleanliness and elegance. A high cohesion allows us to join one of
the basic principles of software development, the **DRY** principle, "*Don't Repeat Yourself*". So, the
minimization of dependencies is a goal to be pursued at all levels.

For example, let’s analyze a classic software problem: a repository class (`UserRepository`), which
enables access to information stored on database, and a service class (`UserService`) that allows the
use of this information to other components.

![Class diagram showing a direct dependency between two classes](http://rcardin.github.io/assets/cp_1.png)

Clearly, the direct association between these two elements creates a high level of coupling between them.
In particular `UserService` is responsible for creating an instance of `UserRepository`, tying the evolution
of `UserService` in the evolution of the `UserRepository` code. In addition, it is complex to design any
unit tests, which would require the replacement of a `UserRepository` with a mock in `UserService` code.

You can improve the situation by abstracting the `UserRepository` component in an interface and changing
the dependency in the `UserService` towards this interface.

![Class diagram showing a better solution using an interface](http://rcardin.github.io/assets/cp_2.png)

This new configuration enables the use of **dependency injection** (DI) pattern. DI is a design pattern
that helps to separate the behavior of a component from the responsibility for resolution of its
dependencies.

For strongly typed object-oriented languages, you must use an external framework that handles and resolves
the dependencies. These dependencies are suitably configured by the developer and framework uses these
information to operate like a factory, managing the complete life cycle of software components.

In Java, for example, the most commonly used DI frameworks are [Spring](http://spring.io/) and
[Google Guice](https://github.com/google/guice).

## The Cake pattern

Scala provides a mechanism to resolve dependencies similar to DI, the *cake pattern*. Unlike DI pattern in
other languages, such as Java, C++, etc ... the cake pattern can be used directly with language constructs
available in Scala.

### Traits

In Scala the cake pattern is made possible by the presence of the construct of the trait. A trait is similar
to a Java interface, but unlike the latter it can have both attributes, both methods completely implemented.

A trait also has a key feature for the cake pattern: can be "added" to any component as a mixin. In this way
the attributes and methods of the trait are added to those of the component. This makes it possible to
implement some kind of multiple inheritance, that is not allowed in other programming languages.

![Class diagram showing the meaning of traits](http://rcardin.github.io/assets/cp_3.png)

### Self-type annotation

The Self-type annotation is a special construct that allows you to declare the dependencies of a component
from one or more trait, i.e. which trait should be associated with the component using mixin. For example,
consider the following class `BarAble` and trait `FooAble`.

{% highlight scala %}
trait FooAble {
  def foo() = "I am a foo!"
}
class BarAble { this: FooAble =>   // self-type annotation
  def bar = "I am a bar and " + foo()
}
{% endhighlight %}

Using the notation `this: => FooAble` we mean that an object of class `BarAble` must be associated with
trait `FooAble` using mixin.

{% highlight scala %}
object Main {
  def main(args: Array[String]) {
    val barWithFoo = new BarAble with FooAble   // mixin
    println(barWithFoo.bar())
  }
}
{% endhighlight %}

In order to instantiate the object `barWithFoo`, the compiler checks the dependencies declared through mixin
and if these are not met it will provide the following error:

> class BarAble cannot be instantiated because it does not conform to its self-type BarAble with FooAble

### The full recipe

Trait and self-type annotation are the foundation of the cake pattern. Return to the main problem, namely the
resolution of dependency between the class `UserService` and the class `UserRepository`. An elegant solution
to the problem must not modify these two classes, maintaining isolated and distinct business aspects from
more technical aspects. For this reason, we define two traits whose aim is to enclose the original classes
in components more malleable, like layers of a cake.

![A visual representation of the cake pattern](http://rcardin.github.io/assets/cp_4.jpg)

{% highlight scala %}
trait UserRepositoryComponent {
  val userRepository: UserRepository
  class UserRepository {
    def findAll() = Array[User]()   // fake implementation
    def save(user: User) = "Saving a user..."   // fake implementation
  }
}
trait UserServiceComponent { this: UserRepositoryComponent =>
  val userService: UserService
  class UserService {
    def findAll() = userRepository.findAll
    def save(user: User) = userRepository.save(user)
  }
}
{% endhighlight %}

In each of the new trait original classes are made available through an internal attribute. Note that an attribute
(`val`) has been preferred to a function (`def`) because both service and repository are *Singleton* classes: they
require no more than one active instance within an application. Also, note that both traits are abstract, since
internal class instances are not initialized.

Note in particular the latter trait, i.e. `UserServiceComponent`. Through the use of a self-type annotation is
declared the dependency upon trait `UserRepositoryComponent`, through which `UserService` class can access the
`userRepository`. In case you need to declare multiple dependencies this process can be repeated, using a notation
such as

{% highlight scala %}
this: A with B with C =>
{% endhighlight %}

which `A`, `B` and `C` are traits similar to `UserRepositoryComponent`.

So we have built a mechanism for declaring dependencies among components. We need to take the last few steps and
implement the real mechanism of dependency injection: the implementation of abstract trait; their composition, which
allows the resolution of dependencies.

Let’s build a configuration object, implementing concrete instances of classes `UserRepository` and `UserService`.

{% highlight scala %}
object ComponentRegistry extends
  UserServiceComponent with UserRepositoryComponent {
  // Dependency injection
  val userRepository = new UserRepository
  val userService = new UserService
}
{% endhighlight %}

The dependencies are resolved in one single place and can be modified using subclassing, simplifying unit testing
and mocking.

In case you want to replace the dependency to the `UserRepository` class with a pure interface, you can provide a
trait within `UserRepositoryComponent`, which will be concretely implemented later.

{% highlight scala %}
trait UserRepositoryComponent {
  def userRepository: UserRepository
  trait UserRepository {
    def findAll()
    def save(user: User)
  }
}
{% endhighlight %}

## References

- [Real-World Scala: Dependency Injection (DI)](http://jonasboner.com/2008/10/06/real-world-scala-dependency-injection-di/)
- [Cake pattern in depth](http://www.cakesolutions.net/teamblogs/2011/12/19/cake-pattern-in-depth)
- [Dependency Injection in Scala: Extending the Cake Pattern](http://www.warski.org/blog/2010/12/di-in-scala-cake-pattern/)
- [Dependency injection vs. Cake pattern](http://www.cakesolutions.net/teamblogs/2011/12/15/dependency-injection-vs-cake-pattern)
- [Cake Pattern Practices](http://vpatryshev.blogspot.com.au/2012/06/cake-pattern-practices.html?m=1)
- [Cake Pattern in Scala / Self type annotations / Explicitly Typed Self References – explained](https://coderwall.com/p/t_rapw)

extension: markdown


text:
# Change Log
All notable changes to this project will be documented in this file.
This project adheres to [Semantic Versioning](http://semver.org/).
Thanks to [keepachangelog.com](http://keepachangelog.com/) for this template.

## [Unreleased][unreleased]

## [0.0.4] - 2015-09-13
### Added
- This CHANGELOG file to track changes across versions.
- Example to submit Spark job to remote EC2 standalone cluster.

### Changed
- `sparkSubmitJar` now accepts `String` instead of `File` so arbitrary file path can be specified
such as `hdfs:///tmp/my-awesome-job.jar`

## [0.0.3] - 2015-08-26
### Changed
- Unsuccessful submission now emits error and shows on the screen
- Compile sources to Java 7 for compatability

### Fixed
- Fix a bug that prevents `settings(settings: Setting[_]*)` from working

## [0.0.2] - 2014-06-24
### Added
- `SparkSubmitYARN` plugin provides default settings suitable for YARN submission
- `SparkSubmitSetting` gains a few convenience constructors

## 0.0.1 - 2014-06-23
### Added
- Productivity and Awesomeness incoming...

[unreleased]: https://github.com/saurfang/sbt-spark-submit/compare/v0.0.4...HEAD
[0.0.4]: https://github.com/saurfang/sbt-spark-submit/compare/v0.0.3...v0.0.4
[0.0.3]: https://github.com/saurfang/sbt-spark-submit/compare/v0.0.2...v0.0.3
[0.0.2]: https://github.com/saurfang/sbt-spark-submit/compare/v0.0.1...v0.0.2

repository: izhangzhihao/sbt-spark-submit

repo_name: izhangzhihao/sbt-spark-submit


text:
# fluent-plugin-bwlist

fluent-plugin-bwlist is a record filter plugin using black/white list.

## Installation

### RubyGems

```
$ gem install fluent-plugin-bwlist
```

### Bundler

Add following line to your Gemfile:

```ruby
gem "fluent-plugin-bwlist"
```

And then execute:

```
$ bundle
```

## Example

conf:

```
<filter **>
  @type bwlist
  s3_bucket sample-bucket-name
  s3_key bwlist.txt
  key id
  mode blacklist
</filter>
```

list(Newline-delimited):

```txt
1
3
5
...
```

## Configuration

### s3_bucket (string) (required)

S3 bucket name

### s3_key (string) (required)

S3 path

### key (string) (required)

filtering target key name

### mode (enum) (optional)

Select black/white filtering mode

## Copyright

* Copyright(c) 2018- kei_q
* License
  * Apache License, Version 2.0

repository: kei-q/fluent-plugin-bwlist

gha_language: Ruby

repo_name: kei-q/fluent-plugin-bwlist


text:
# Homebrew spotinfo Tap

Homebrew tap for [spotinfo](https://github.com/alexei-led/spotinfo) 🍺

> The `spotinfo` is a command-line tool that helps you determine AWS Spot instance types with the least chance of interruption and provides the savings you get over on-demand rates.

## Install

```sh
brew tap alexei-led/spotinfo
brew install spotinfo
```

repository: alexei-led/homebrew-spotinfo

repo_name: alexei-led/homebrew-spotinfo


text:
# Painful-Minecraft

A Minecraft Mod, that will trigger a esp32 via wifi to turn on a 9V taser, that will hurt the player when steve's gets hurt

## Changelog

### The V1.0 is deprecated

The version 1.0 was working with a CV based method using image recognition from the game itself. it wasn't 100% accurate so is changed. all the data is in the folder PythonTest

### Mod Now Working

The mod is almost done, to make useful only requires to change the url from the request to match with the esp32 ip address
Also the Esp32 code is done, and the schematics will be uploaded soon


text:
### 3.1.0 (2018-03-22)

* **dependencies:** Update dependencies and js code style

#### 2.0.1 (2018-3-22)

##### Chores

* **dependencies:** Remove all gulp related packages and replace it with npm scripts
- Update code to support es6 features

##### Chores

* **plugins:** add accepting configurable plugins ([96ad4494](https://github.com/Do-IT-Programming-Solutions-LP/hapi-bookshelf-models2/commit/96ad449482ab67f4c765f9090c8b10b801103d6d))
* **errors:** report original error if models/collections fail ([222a658e](https://github.com/Do-IT-Programming-Solutions-LP/hapi-bookshelf-models2/commit/222a658e08a2c59085d9f60eee98b1cd1fd34bda))
* **lint:** replace jscs and jshint with eslint ([52357827](https://github.com/Do-IT-Programming-Solutions-LP/hapi-bookshelf-models2/commit/523578276a8a24431fa269ce9113594d5ace48f1))

##### Documentation Changes

* **contributing:** add CONTRIBUTING.md ([b8c0289a](https://github.com/Do-IT-Programming-Solutions-LP/hapi-bookshelf-models2/commit/b8c0289a1c2f1fd1e589b3201b6dc3092381e8be))

## 2.0.0 (2016-6-4)

##### Chores

* **dependencies:** make bookshelf and knex peer dependencies ([e39441c9](https://github.com/lob/hapi-bookshelf-models/commit/e39441c9a4ca394982158a34d8f65f671a2b18ab))

### 1.8.0 (2016-5-17)

##### New Features

* **collections:** add support for bookshelf collections ([4cae0206](https://github.com/lob/hapi-bookshelf-models/commit/4cae0206a35679f8b176d6d71f958b5949ab6611))

#### 1.7.1 (2016-2-11)

##### Bug Fixes

* **paths:** fixed error thrown by a configuration with a relative path to models ([efadcdd4](https://github.com/lob/hapi-bookshelf-models/commit/efadcdd4b6f36fa197b27577eeeb205cef12c710))

### 1.7.0 (2016-2-2)

##### New Features

* **paths:** added glob to support wildcard model folder structure ([68fc0f5a](https://github.com/lob/hapi-bookshelf-models/commit/68fc0f5a8e888204145d9ba4f1764b5efb7bdb5b))

##### Chores

* **release:** add release scripts with changelog generation ([63bbf46f](https://github.com/lob/hapi-bookshelf-models/commit/63bbf46f))

## 1.6.0 (2015-12-23)

- Update bookshelf to 0.9.1

## 1.5.0 (2015-09-29)

- Allow Hapi 9.x and 10.x

## 1.4.0 (2015-09-10)

- Ignore hidden files in models directory

## 1.3.0 (2015-03-01)

- Pass `bookshelf` instance to model register

## 1.2.0 (2015-01-29)

- Added ability to namespace models to support multiple connections

## 1.1.0 (2015-01-21)

- Made `base` optional in register

gha_language: JavaScript