text:
---
layout: post
title:  "Deep Work by Cal Newport"
date:   2019-06-07
desc: "Book Notes"
keywords: "Books"
categories: [Books]
tags: [Books, Notes]
icon: icon-html
---

Introduction:

	• Deep work is undistracted focus on important / valuable work
	• Isolation from everything is required
		• social media, electronics, people, everything...
	• work periods / time frames should be long and consecutive
	• a typical knowledge worker spends 30% time w/ emails and 60% time w/ electronic communication & internet browsing
	• Shallow work is cognitively undemanding and preformed while distracted
	• To learn complex subjects requires uninterrupted and intense concentration
	• To learn computer programming
		• e.g: a man locked himself in a room w/ no computer, just textbooks, note cards, and a highlighter
		• he used the technique of:
			• highlight -> notecards -> practice
			• made sure there was nothing in his room to distract him
	• to be valuable, you must be able to learn complicated things fast
	• deep work is becoming scarce and increasing in value
	• Cal Newport constructs his days with chosen deep work and shallow activities, unavoidable, that are batched into smaller bursts
	• lack of distraction helps remove nervous mental energy

Chapter 1: Deep Work is Valuable

	• Highly Skilled worker: In our economy, those good with working with intelligent machines will thrive
	• Superstars: As the talent market is more universal, those at the top will thrive
	• Owners: With invested capital in intelligent machines, those will thrive
	• Desired Attributes:
		• quickly master difficult things
		• produce with high quality and speed
	• experts reflect a pursuit of effort in a particular field
	• keep your attention focused on a specific field
	• receive feedback to improve / change your approach to:
	• keep your mindset where it has highest productivity
	• myelin: a layer of fatty tissue around neurons
		• allows cells to fire faster / cleaner
	• check out the book: Give & Take by Grant
	• High-Quality work Produces = (Time Spent) X (Intensity of Focus)
	• Residue: when switching tasks, you attention doesn't immediately follow, there is residue

Chapter 2: Deep Work is Rare

	• Other trends have become common place
		• Rapid communication
		• Active Social Media
		• Open Collaboration
	• Deep work's metrics are a " black hole"
		• Contribution and business behavior are hard to quantify
	• Principle of Least Resistance
		• Without clear feedback, we trend towards the easiest behaviors
	• Open-ended interrogative questions such as "Thoughts•" Is painfully frustrating
		• This is time-sucking, but is easier from the sender's perspective
	• Avoid administrative duties
		• Feymman, to avoid administrative duties, he made a myth that he is irresponsible and uses this as a reason to get out of doing extra things
	• Busyness is a proxy for productivity
		• Doing lots of stuff in a visible manner to demonstrate value is misguided
	• Trade-offs surrounding new tech is disregarded and assumed to be good

Chapter 3: Deep Work is Meaningful

	• Deep work and a good life has a strong connection for master craftsmen
	• Master craftsmen find great meaning in their work
	• Depth destroying behaviors are praised in society
		• Avoidance generates suspicion
	• What we choose to focus on and ignore can define our quality of life
		• Skillfully manage your attention!
	• 'The idle mind is the devil's workshop'
		• When focus is lost, your mind fixates on what is wrong instead of what is right
	• Best moments occur when:
		• Mind is stretched to its limits
		• To accomplish a difficult and meaningful task
	• Jobs are easier to enjoy than free time
		• Built in goals, feedback, challenges
	• Our minds like going deep regardless of the subject
	• Deep work generates flow state, which includes:
		• Stretching mind to its limits
		• Loosing yourself in the work
		• Flow generates happiness
	• Seek out opportunities for flow
	• Beautiful code is short and precise
	• Meaning is uncovered through effort due to skill and appreciation of the craftsmanship
		• Craftsman don't have rarified jobs
		• A sword doesn't have to be special, the effort and skill of the craftsman who made it is

Rule #1: Work Deeply

	• Dissertation: long essay on a particular subject
	• Example strategies of deep work:
		• Eliminate / minimize all other types of work
		• Rhythmic strategy, doing deep work for the same hours every weekday
	• Eudaimonia Machine by David Dewane
		• 5 Rooms
			• 1st Room: Gallery
				• Demonstrates examples of deep work
				• Designed to inspire users
			• 2nd Room: Salon
				• Coffee, Bar, couches, Wi-Fi
				• Designed for curiosity and argumentation
			• 3rd Room: library
				• The hard drive, archive, or vault of your information
				• Designed as a permanent record of all work produced
			• 4th Room: Office Space
				• Conferences, white boards, cubicles
				• Designed for shallow efforts
			• 5th Room: Deep Work Chambers
				• Soundproof, Isolation
				• Designed for total focus
				• Ideal scenario:
					• A process of spending 90 min inside, take a 90 min break, then repeat 2-3 times to reach peak of concentration for the day
	• Main obstacle for going deep:
		• Urge to turn your attention to something superficial
	• Finite amount of will power that depletes as you use it
		• Like a muscle that tires
	• Add routines and rituals to your life to minimize limited willpower needed to transition into a focused state
		• Setting a time, a quiet location, would allow you to use less willpower to start and keep going
	• Bimodal philosophy for  Deep Work
		• Dedicates stretches of time, min 1 day, to deep tasks
	• People will respect your inaccessibility if well-defined and advertised
	• Seinfeld uses a chain method
		• Marks calendar w/ a red 'X' when he writes a new joke for the day
		• Once he starts getting chains, he try's no to break them
		• Transforms this task into a habit
	• 90-min stretches doesn't yield much productivity in an ad-hoc style
	• Ad-hoc is something done for a particular purpose
	• Committing time for depth in a rhythm, in big chunks, e.g 2.5 -> 3 hrs is effective
	• It is a good idea to log hours in depth
	• Journalistic approach doesn't come easy and is a skill obtained by their deadline driven nature
	• Journalistic approach takes advantage of any free time that pops up
	• Cal Newport's strategy is to map out when he will work deeply at the beginning of each week
		• Then refines details later to save effort / energy
	• Waiting for inspiration is a bad plan
		• Instead ignoring is the better alternative
		• Great creative minds think like artists but work like accountants
	• Build rituals that are strict!
	• Questions to ask yourself when trying to setup a deep work routine
		• Where you will work, and for how long•
			• Specify location, ideally a location for depth only
			• Set a specific time frame, gives it a challenge
				• No open endings
		• How you'll work once you start to work•
			• Structure the work to avoid wasted will power deciding
		• How you'll support the work•
			• Exercise, diet, walks, find what works!
			• Structure this as well to avoid waste
	• Radical environment change w/ significant investment of effort / money increases perceived importance
		• Perceived importance can provide motivational energy and focus
		• Psychology of strong commitment through setting up a good environment can unlock mental resources
	• Architecture of isolated offices with common connected area
		• Supports deep thinking and serendipitous encounters
		• Expose oneself to new ideas / places / people, but have a place for your isolation
	• White board effect: Some problems can be worked on side by side with another
		• Each pushing each other to greater depths
		• Carefully choose this type of engagement, it only applies to specific situations
	• 4DX framework is to help get through the gap between what and how
		• Focus on the wildly important
			• "the more you try to do, the less you actually accomplish"
			• Add some ambitious outcomes to pursue w/ deep work hours
			• Focus on saying 'yes' to the subjects you are longing and let this crowd out everything else
		• Action on the Lead Measures
			• 2 metrics: lead and lag
			• Lag: measures what you are trying to improve (past)
				• e.g: getting a promotion, mastering a concept
			• Lead: measures effectiveness of new behaviors (present)
				• e.g: time spent in a state of deep work for you wildly important goal
			• I can improve lead by tracking hours spent in deep work
		• Keep a compelling Scoreboard
			• Tracks tasks completed but more importantly the effort of each day
		• Create a Cadence of Accountability
			• Regular meetings to confront scoreboards
			• Encourages execution
				• With independent work, this can be effective with weekly reviews
	• 4DX helps support a lifestyle of depth and its regularity, not necessarily the intensity
		• Regularity > intensity
	• "I am not busy.  I am the laziest ambitious person I know"
		• In this example, a man fled to a place w/ no Internet or TV
		• Going online required biking to the library
	• "Idleness is not just a vacation, an indulgence or a vice;  it is as indispensable to the brain as vitamin D is to the body... paradoxically, necessary to getting any work done."
		• To strategize and use this technique, it can be used as follows:
			• Shutdown all work issues at the end of your workday
				• No email, no mental replays, no scheming
			• If you need more time, extend your workday and shutdown later
	• Some decisions are better left for the unconscious mind to untangle
		• Move on and don't dwell on certain things too long
	• Decisions that require the application of strict rules are good conscious mind
		• e.g: calculator
	• Decisions that are vague, lots of info, conflictions, restraints, many variables... are well suited for the unconscious mind
	• Conscious mind is like a computer / calculator
	• Unconscious mind is like a data center, with complex algorithms to sift through unstructured data
		• This can output ideas and solutions to complex problems
	• Walking on a busy street, driving on a complex road, maneuvering around obstacles, drains will power / directed attention
	• Walking through nature exposes you to fascinating stimuli, freed from challenges which helps recover willpower / directed attention
	• The work that evening routines replaces is usually not that important
		• It is common to reach your deep work capacity by the end of the workday
		• Therefore, shallow work will take its place
	• Implement a strict shutdown ritual
		• Series of steps you always do, and a set phrase at the end
			• e.g "Shutdown Complete"
		• Without shutdown, issues will battle for you attention
	• Capture items/ tasks to do before shutdown
		• These items will be taken care of when appropriate and will not interfere w/ your ritual
	• Regularly resting improves the quality of deep work

Rule #2: Embrace Boredom

	• The ability to concentrate intensely is a skill that must be trained
	• To focus deeper, one must distance their dependence on distractions
	• Constant attention switching online has a lasting negative effect on the brain
	• When a brain is used to distraction, it's hard to shake the addiction
		• e.g. for every potential moment of boredom, you find yourself reaching for your phone
	• People who multitask all the time are mental wrecks
	• Deep work training consists of 2 main ideas
		• Improving your ability to concentrate intensely
		• Overcoming your desire for distraction
	• You can use the Internet in a focused and deep way, but for a distraction addict, it's not possible
	• Schedule in advance when you'll use the internet, and avoid it outside of these times
		• This minimizes, the number of distractions that will occur and let your attention selecting muscles strength
	• Schedule internet usage blocks at work to do the following things:
		• Check emails, research content, etc.
		• When blocked, you must be absolutely free from internet
			• turn off notifications!
			• Go airplane mode
			• Turn off computer
	• When stuck / blocked from lack of internet, switch to another offline activity or relax
		• Remember the idea, take a break from deep work, and don't fill time off with shallow work
	• If urgency is dire, at least add a gap (5 min) before you go online to separate the sensation and the reward
	• Scheduling internet use at home as well as at work can further improve your concentration training
	• Rewire your brain to eliminate the ability for distractions to hijack your attention
	• Another strategy to increase intensity of deep work, estimate the time it takes to do something, then give yourself an aggressive deadline to complete it
	• Try this experiment at most 1 time a week at first then increase
		• Keep deadlines right at the edge of feasibility
		• You should always beat the buzzer, but to do so will be very hard
		• Deep work requires levels of concentration well beyond what most knowledge workers would find  comfortable
		• Use artificial deadlines to systematically increase the level you can regularly achieve
	• Productive meditation: take a time frame where you are physically but not mentally occupied
		• Focus your attention on a single well-defined professional problem
		• Continue to bring your attention back to the problem at hand when it wanders or stalls
		• The purpose of these is to rapid improve your ability to think deeply
		• Your mind will rebel and try to switch topics to seemingly more interesting thoughts
		• Remember you can return to that thought later, then redirect your attention
		• Your mind will avoid diving deeper and begin to loop on information you already know
		• Identify looping and distractions!
	• Structure your deep thinking
		• Start with a careful review of what variables you want to focus on
		• Store these into your working memory
		• After solving / addressing these variables, consolidate your gains
		• Then repeat this process by identifying your new variables
	• Humans can't internalize abstract information quickly
	• Humans can remember scenes very well
	• Ron White's Card memorization technique
		• Cement the image of walking through 5 rooms in your home
		• In each room, conjure a clear image of what you see
		• Fix in your mind a collection of 10 items in each room
			• These items should be large, like a desk
		• Establish an order in which you look at each room
		• Combined this is 50 items, so add 2 more
		• Then associate a memorable person / thing with each card
		• Then walk through your house and deck of cards together assigning the mental images you have for the card w/ the place
		• e.g. king of diamonds = Donald trump, 1st item in 1st room is a mat, hence Donald trump wiping mud off his shoes on the map, is the image
		• After completing a room, walk through it a couples times to retain info
	• There is nothing special about this card memorization technique
	• Any thought process rewiring to this level of attention can deepen and strength your ability to concentrate
	• If card memorization is weird and not appropriate to you, then find a replacement

Rule #3: Quit Social Media

	• Accept that these tools aren't designed to be evil
		• Some can be vital to happiness and success
		• BUT, reject the state of hyper connectedness
		• Don't give these tools such regular access to your time and attention
			• Use less
	• Friendships online are lightweight, shouldn't be the center of the user's social life
	• Don't justify the use of social media or anything just because it might have some benefit or from fear of missing out
		• This would ignore all negatives that come along
		• When adopting tools, keep a skeptic mind & approach
			• This type of approach is obvious for craftsmen and skilled knowledge workers
	• Craftsman Approach:
		• Identify core factors that determine success and happiness
		• Adopt a tool only if positives outweigh negatives
	• Reduce complexity in deciding what matters to you
		• Keep a list limited to what's most important
		• Keep descriptions high-level
		• Ex: being an effective teacher in the classroom, effective mentor to graduate students, being effective researcher
		• Go through the list of tools you currently use, and identify its impact; substantially positive, substantially negative, or little impact
	• Ex: Professional Goal: To craft well-written, narrative driven stories that change the way people understand the world
		• Key activities supporting this goal:
			• Research patiently and deeply
			• Write carefully and with purpose
	• Then apply the tools in comparison with these ideas / goals
	• There are situations where Facebook and other tools should be used
		• e.g: new college students who goals are to establish new friends by attending lots of events and socialize with lots of people
	• Law of vital few: 80% of a given effect is due to 20% of the possible causes
		• e.g: Out of 10-15 different activities you do, it•s the top 2 or 3 activities that make the most difference
		• Don't service these lower impact activities just because they provide "some" benefit
		• e.g: Instead of finding old friends on FB, take a good friend out to lunch
		• Get more and do more of the activities you know will yield large benefits
	• "packing party": pack up all your possessions, then only take out what you used / needed, after a while throw / get rid of the stuff you haven't used
		• This can be used w/ tech too, ban everything and only use what gives you high/large benefits
		• Notice the impact over a period of time, e.g: 30 days
		• Don't announce this experiment
			• Part of the delusion is that people actually want to hear what you have to say
			• Gaining an audience is easier than ever before because of the idea of "you like my status update and I'll like yours"
		• This is underserved agreements of attention
	• For many activities, tools, commitments, you won't understand their value until you sample life without them
	• Put more thought into your leisure time
		• Don't default to whatever catches your attention in the moment
	• Your 16 hours of full energy will increase the value of the 8 business hours
		• Your mental reserves are capable of continuous hard activity and just require a change and of course sleep
		• Doing this will perhaps teach what it means to live, and not just exist

Rule #4 Drain the Shallows:

	• Most people don't work 8 hours a day
		• Too many interruptions, web surfing, office politics, personal agendas
		• With fewer hours allocated, you spend them more wisely
			• Therefore more shallow time would be given up for deep work
	• Import quote on productivity:
		• "How can we afford to put our business on hold for a month to 'mess around' with new ideas•" • "How can we afford not to•"
	• Don't pursue all your timed spent in depth
		• Shallow work is needed in most jobs
		• Tame shallow work's quantity and control over you
		• Shallow work is bad when it drowns out your ~4 hours of deep work
	• The most adept thinker can't spend more than 4 hours in depth
	• Most spend their day on auto pilot, not giving thought about what they are doing w/ their time
		• Adopt a habit of pursing before action and asking "what makes the most sense right now•"
	• Divide the hours into blocks and assign activities to these blocks
		• Not all of these blocks need to be work related
		• Min length should be 30 min
		• At first estimates will be wrong
		• Interruptions and obligations break your schedule
			• If your schedule is broken, pursue and resize your schedule
			• Cross out and add to the side of it
			• The goal isn't to stick to a given schedule no matter what, it's to have a thoughtful say in what you're doing and will be doing
	• Make a goal to accurately predict the time tasks require
		• Add overflow blocks incase previous time blocks get blocked
			• e.g: If an import insight is discovered, it may be worth looking into
			• Afterwards, rebuild schedule and dive back into it
	• Remember the goal of a schedule isn't to force your behaviors into a rigid plan
	• Continually ask yourself "What makes sense for me to do with the time that remains"
	• Without structure, it's easy for your time to devolve into shallow
	• When determining whether a task is deep or not, ask "how long would it take (in months) to train a smart recent college graduate w/ no specialized training in my field to complete this task"
	• Tasks that tend to lever your expertise are deep
		• Return more value per time spent and stretch your abilities, and enhance improvement
	• It•s a waste of money to pay professionals to send emails and attend meetings all day
		• Need to say no to somethings things, and streamline others
		• Perhaps get an explicit approval ration on time spent on shallow work from boss
			• Perhaps your boss will say yes to shallow work, which is useful because it highlights your job doesn't support deep work
	• Deliberately... do specific things to preserve happiness
	• Limit shallow endeavors
	• For excuses, avoid providing enough details that allows the requester to defuse it
		• Ex: "Sounds interesting, but I have schedule conflicts"
			• Also resist the urge to offer consolation prizes
			• A clean break is the best
	• Choose behaviors that reorient your focus towards the deep
	• Make people who send you email do more work
		• Cal Newport uses a sender filter
			• E.g<EMAIL_ADDRESS>			• Only responds if they match schedule and interests
		• You have the right to control your own incoming communication
			• Expectation resets might earn you even more credit when you do respond
	• The idea of all messages arriving in the same inbox, regardless of purpose and sender, is ridiculously unproductive
	• Answer this question before answering vague, time consuming, and interrogative questions:
		• "What is the project represented by this message, and what is the most efficient (in terms of messages generated) process for brining that project to a successful conclusion•"
		• Minimize messages back and forth, and be very process oriented with your replies
	• Many academics just don't respond
		• They believe it's the sender's responsibility to convince the receiver that a reply is worth while
	• Don't reply to an email if these rules apply to it
		• Ambiguous, a reasonable response would be difficult
		• It doesn't interest you
		• No value
	• People adjust to your communications, don't fret

Conclusion:

	• Setting up early commitments and adding perceived value such as an expensive notebook can induce more careful thinking
	• Obsessive focus can be sparked by:
		• Feeling not good enough
		• Having limited time
		• Understand the value of the focus
	• Work on problems in your head when the time is presented

---


text:
---
layout: post
title:  "Framework Design Notes"
date:   2015-06-24
categories: Framework
excerpt:
comments: true
---

### CAP
* Consistency (all nodes see the same data at the same time)
* Availability (a guarantee that every request receives a response about whether it succeeded or failed)
* Partition tolerance (the system continues to operate despite arbitrary partitioning due to network failures)

1. Stering and communication. Open-ended. Consider the following points. And be creative.

* Concurrency. Do you understand threads, deadlock, and starvation? Do you know how to parallelize algorithms? Do you understand consistency and coherence?
* Networking. Do you roughly understand IPC and TCP/IP? Do you know the difference between throughput and latency, and when each is the relevant factor?
* Abstraction. You should understand the systems you’re building upon. Do you know roughly how an OS, file system, and database work? Do you know about the various levels of caching in a modern OS?
* Real-World Performance. You should be familiar with the speed of everything your computer can do, including the relative performance of RAM, disk, SSD and your network.
* Estimation. Estimation, especially in the form of a back-of-the-envelope calculation, is important because it helps you narrow down the list of possible solutions to only the ones that are feasible. Then you have only a few prototypes or micro-benchmarks to write.
* Availability and Reliability. Are you thinking about how things can fail, especially in adistributed environment? Do know how to design a system to cope with network failures? Do you understand durability

### system design

0. Sterring and commucation.

1. constraints and use cases
2. Abstract design: draw diagrams and components and connections. And provide application layer, basic services, backend db design.
3. understand bottleneck:  Perhaps your system needs a load balancer and many machines behind it to handle the user requests. Or maybe the data is so huge that you need to distribute your database on multiple machines. What are some of the downsides that occur from doing that? Is the database too slow and does it need some in-memory caching?
4. scalable abstract design: solve the bottle neck.
Be prepared for discussions about tradeoffs, about pros and cons. Be prepared to give alternatives, to ask questions, to identify and solve bottlenecks, to go broad or deep depending on your interviewer's preferences.
Don't get defensive: whenever your interviewer challenges your architectural choices, acknowledge that rarely an idea is perfect, and outline the advantages and disadvantages of your choice. Be open to new constraints to pop up during the discussion and to adjust your architecture on the fly.

### scalability basics
1. clone: Capistrano, AWS calls this AMI - Amazon Machine Image
2. db: Path #2 means to denormalize right from the beginning and include no more Joins in any database query. You can stay with MySQL, and use it like a NoSQL database, or you can switch to a better and easier to scale NoSQL database like MongoDB or CouchDB. Joins will now need to be done in your application code. The sooner you do this step the less code you will have to change in the future. But even if you successfully switch to the latest and greatest NoSQL database and let your app do the dataset-joins, soon your database requests will again be slower and slower. You will need to introduce a cache.
3. cache: With “cache” I always mean in-memory caches like Memcached or Redis. Please never do file-based caching, it makes cloning and auto-scaling of your servers just a pain.
choice 1: cache db queries, hashed key is the query. Problem: updated data. 2. cache objects, better.
And the best part: it makes asynchronous processing possible! Just imagine an army of worker servers who assemble your objects for you! The application just consumes the latest cached object and nearly never touches the databases anymore!
4. asynchronism case 1: cron job case 2: mq.If you now want to dive more into the details and actual technical design, RabbitMQ is one of many systems which help to implement async processing. You could also use ActiveMQ or a simple Redis list. The basic idea is to have a queue of tasks or jobs that a worker can process. Asynchronism seems complicated, but it is definitely worth your time to learn about it and implement it yourself. Backends become nearly infinitely scalable and frontends become snappy which is good for the overall user experience.

scale horizontally: add nodes. vertically: add cpus or memory in one node.

### web distributed system design

key principles: availability, performance, reliability, scalability, manageability, cost. (rapsmc)

partition data problem: data locality, need to fetch data. Inconsistency

global cache: to overcome local cache missing by load balancer

proxy once the cache is not found: collaborate similar request One way to use a proxy to speed up data access is to collapse the same (or similar) requests together into one request, and then return the single result to the requesting clients. This is known as collapsed forwarding. but generally it is best to put the cache in front of the proxy, for the same reason that it is best to let the faster runners start first in a crowded marathon race.

index:

load balancers: issue: user-session-specific data. managing user-session-specific data. In an e-commerce site, when you only have one client it is very easy to allow users to put things in their shopping cart and persist those contents between visits (which is important, because it is much more likely you will sell the product if it is still in the user's cart when they return).

queue:

### scalability trade-offs

1. performance vs scalability
2. latency vs throughput: maximum throughput with acceptable latency
3. availability vs. consistency
availability pattern: fail-over, replication
scalability pattern: state (partitioning, http caching, rdbms sharding, nosql, distributed caching, data grids, concurrency)

### Concurrency:

1. shared-state: indeterministic, introduce locks(problem)
2. message-passing: actors: has msg queues, communicate with msgs, dataflow,
software transactional memory: transaction can nest and compose

![alt text](https://cloud.githubusercontent.com/assets/5607138/8345396/3dd6f8b2-1aa3-11e5-88b9-22b55219fa89.png)

scalability patterns: behavior

ESB: enterprise service bus

stability patterns: timeouts, circuit breaker, let it crash, fail fast, bulkheads, steady state (logging), throttling (queue requests in diff steps)

### google online document

other methods:
1. event update: not reliable , not robust
2. 3 way merge, (C,S, version 1) -> version 2. problem: no realtime, on feedback from server, need to stop typing and cannot update live.
3. Solution:

### two algo: diff and fuzzy patch.

shadow and backup for each client and server. version number n (client) and m(server), when send out an update edits, n++, m remain the same. And server receives the edits and compare the version m of server, if m is smaller in server, it will use backup to rollback server version. Patch the updates, diff the shadow with server, update the backup/shadow/server, m++, send edits back to client. And the client will do the similar thing.

Note: only client will send update to servers, servers are not positive.

For each client, server will have one to one shadow for them.

Limitation: attributions(cannot figure out who did what in server), (one package only in the route) might be big latency if c,s has long distance.

### redis

key --> data structure, provide list, set, etc and its operations. provide score(weight) to the the value.

has publish/subscribe pattern.

For example if there are more querying required, it mostly means more work in Redis, where you can use different data structures to suit your queries. Same is easier in MongoDB. On other hand this is often extra work in Redis would most likely to pay off with sheer speed.

MongoDB offers simplicity, much smaller learning for guys with SQL experience. Whereas Redis offers non-traditional approach hence more learning but huge flexibility.

Eg. A cache layer can probably be better implemented in Redis, and as for a more schema-able data MongoDB might be better. [Note: mongodb is schemaless]
 Programming with Redis is just like doing everything with Lists and Hashes inside memory with your favorite dynamic programming language, but the dataset is persistent and of course not as fast as accessing directly to your PC's memory (there is a networking layer in the middle).

### memcached

* storage cost
* retrieval cost
* invalidation
* replacement policy
* cold/warm cache

typical used in session, user and homepage data.

### twitter trending topics

* bad method: parametric: detect a jump with a theadhold . Problem: gradual rise and no clear jump, diff model.
* good method: data-driven: compute the distance between vote and no-vote to fit one module.

### [mapreduce vs spark](http://blog.cloudera.com/blog/2014/09/how-to-translate-from-mapreduce-to-apache-spark/)

### Bloom filter

A Bloom filter is a space-efficient probabilistic data structure, conceived by Burton Howard Bloom in 1970, that is used to test whether an element is a member of a set. False positive matches are possible, but false negatives are not, thus a Bloom filter has a 100% recall rate. In other words, a query returns either "possibly in set" or "definitely not in set". Elements can be added to the set, but not removed (though this can be addressed with a "counting" filter). The more elements that are added to the set, the larger the probability of false positives.

Bloom filters have a strong space advantage over other data structures for representing sets, such as self-balancing binary search trees, tries, hash tables, or simple arrays or linked lists of the entries. Most of these require storing at least the data items themselves, which can require anywhere from a small number of bits, for small integers, to an arbitrary number of bits, such as for strings (tries are an exception, since they can share storage between elements with equal prefixes). Linked structures incur an additional linear space overhead for pointers. A Bloom filter with 1% error and an optimal value of k, on the other hand, requires only about 9.6 bits per element — regardless of the size of the elements. This advantage comes partly from its compactness, inherited from arrays, and partly from its probabilistic nature. If a 1% false positive rate seems too high, each time we add about 4.8 bits per element we decrease it by ten times.

license: MIT

branch_name: refs/heads/master

extension: md

license_type: permissive


text:
# DTarray_pro
## Combine DTASelect-filter files.
`DTarray_pro` extracts Uniprot ID numbers, molecular weights, and spectral counts from .dtafilter files  stored  in  the  working directory.  Protein  data  is  combined  into  one  dataset and written to the working directory as a tab delimitated text file (.tsv). At runtime, a file list named `dtarray_pro_flist.txt`, containing all the valid DTASelect-filter files  found  in  the  working directory,  is generated. If a file list already exists, the existing file list is used. Various options are available to, modify how files are treated in the working directory, extract additional data from the DTASelect-filter files, and to change the  output file format.

## Installation

`DTarray_pro` is written in c++ 11 and can be compiled on linux or OS-X with gcc.

`DTarray_pro` expects to be installed in `~/local`.  The program needs data stored in text files in `~local/<program_dir>/db` for some features to work.  First make `~/local` if it doesn't exist.
```bash
mkdir -p ~/local
```
If you do not wish to build the program in `~/local`, you can run the configure script from the desired location with the `--progdir ./` option.

`DTarray_pro` depends on the [utils](https://github.com/ajmaurais/utils) c++ library which is included as a submodule in this repository. To clone this repository and the `utils` submodule, run:

```bash
git clone --recurse-submodules --single-branch --branch stable https://github.com/ajmaurais/DTarray_pro
```

To build the executable, run:
```bash
cd ~/local/DTarray_pro
./configure
make
```

After compilation, the `bin` directory should contain two files:

* `DTarray` - the `DTarray_pro` executable.
* `DTSetup` - a bash script which can optionally be used to automatically setup the input directory for `DTarray`

## Usage
From a folder containing DTASelect-filter files with the format `<sample_name>.dtafilter`.
```bash
DTarray
```

See `~/local/<program_directory>/helpFile.pdf` for a detailed explanation of optional arguments or use `DTarray -h` to see the help file from the terminal.


text:
# Butterfly Client

[![Circle CI](https://circleci.com/gh/appynest/butterfly.svg?style=svg&circle-token=ff4017292ad10dc0c64d3bec61c3c1007c865440)](https://circleci.com/gh/appynest/butterfly)

It builds src/, uploads files to S3, and serves via CloudFlare.

## Development

- `npm install` - to install dependencies
- `npm test` - run tests
- `npm run dev` - start development server
- `npm run publish` - build src/apps & upload changes to S3
- `npm run stat` - get code stats with [cloc](http://cloc.sourceforge.net/)
- `aws s3 sync s3://assets.butterflyhacks.com/build ./public/build/` - sync bucket

## Setup /etc/hosts

```
<IP_ADDRESS> butterfly.dev
<IP_ADDRESS> www.butterfly.dev
<IP_ADDRESS> api.butterfly.dev
<IP_ADDRESS> anonymessenger.butterfly.dev
<IP_ADDRESS> social-lab.butterfly.dev
<IP_ADDRESS> foo-bar.butterfly.dev
<IP_ADDRESS> foobar.butterfly.dev
<IP_ADDRESS> back-office.butterfly.dev
```

repository: khsr/BTFly-React-Redux

repo_name: khsr/BTFly-React-Redux


text:
## Predictive power for thrombus detection after atrial appendage closure: machine learning vs. classical methods

[![DOI](https://zenodo.org/badge/480768705.svg)](https://zenodo.org/badge/latestdoi/480768705)

This repo provides code for the research article *Predictive power for thrombus detection after atrial appendage closure: machine learning vs. classical methods*

The aim of this article is to determine whether any differences exist with respect to the predictive power between the classical multivariable methodology and the machine learning one to predict thrombus after atrial appendage closure. Secondarily, we extract the predictor variables of both methodologies and analyze their differences.

### Notebooks:

- [Reported experiments](https://github.com/IA-Cardiologia-husa/LAAC_Thrombus_detection_MLvsClassical/blob/main/All_Experiments.ipynb)

![Experiment workflow](WorkFlowNotebook.png "Experiments workflow")

- [Aditional analisys](https://github.com/IA-Cardiologia-husa/LAAC_Thrombus_detection_MLvsClassical/blob/main/Aditional_analisys.ipynb)

---
# Requirements

Programming language: [Python 3](https://www.python.org/).

Dependencies:
- numpy
- pandas
- scikit-learn
- scipy
- plotnine
- warnings

A suitable [conda](https://docs.conda.io/projects/conda/en/latest/index.html) environment named LAAC_Thrombus_detection can be created and activated with:

    conda env create -f environment.yml
    conda activate LAAC_Thrombus_detection

otherwise

    pip install numpy pandas scipy scikit-learn plotnine warnings

branch_name: refs/heads/main


text:
# fedora-easyfix
A collection of self-contained and well-documented issues for newcomers to start contributing with

## How to set up?

1. `git clone https://github.com/t0xic0der/fedora-easyfix.git`
2. `cd fedora-easyfix/`
3. `virtualenv venv`
4. `source venv/bin/activate`
5. `pip3 install -r requirements.txt`
6. `python3 main.py --help`
7. `python3 main.py -p 9696 -4`

Detailed documentation would arrive shortly. Inconvenience is regretted.

repository: onuralpszr/fedora-easyfix

repo_name: onuralpszr/fedora-easyfix


text:
ofxSatellite
============

A thin wrapper for the [libsgp4](http://www.danrw.com/sgp4/) library.

Track all of your favorite satellites in real-time.

![Screenshot](https://github.com/bakercp/ofxSatellite/raw/master/screen.png)

repository: antoniohof/android-orbital-tracker

repo_name: antoniohof/android-orbital-tracker


text:
# Data-project-hub

Some of the interesting projects that i've worked on and working on ,can be found here

 Contents :
 1. Mining GitHub: Inspecting Software Collaboration Habits, Building Interest .
 2. Mining Web Pages:Using NLP to Understand Human Language, Summarize Blog Posts .
 3. Sentiment classification and visualization of product review .
 4. Temporally aware online news mining and visualization .
 5. Understanding Wikipedia with Latent  Semantic Analysis .
 6. Recipe recommendation engine .
 7. Disease prediction .
 8. Detecting fradulent transactions .
 9. [Predicting stock market returns](https://github.com/bharathgs/project-hub/tree/master/Predicting%20stock%20market%20returns)
 10. Geospatial and Temporal Data Analysis .
 11. Analyzing Neuro imaging Data.
 12. Estimating Financial Risk through Monte Carlo Simulation .
 13. Classifying microarray samples .

repository: akshayjh/project-hub

repo_name: akshayjh/project-hub


text:
[![Nuget](https://img.shields.io/nuget/v/aspnetcore.hashids?label=AspNetCore.Hashids)](https://www.nuget.org/packages/AspNetCore.Hashids/)

# AspNetCore Hashids: Avoid predictable ids in your APIs

Sometimes developers don’t take care about security concerns when they develop a REST API. I know this scope is not straightforward but at least we should comply with a minimum of basic security rules in order to avoid the most common attacks.

The first thing we must take into account is to avoid using predictable ids like identity fields. The last vulnerability I’ve seen was a simple bash script which using a for statement it was making calls to the api passing an autoincremental id and retrieving all the confidentials documents from the system. It was an error configuring the authorization of the system, but the use of predictable ids facilitated the attack.

There are some ways to fix this kind of vulnerability, for example using GUIDs to represent this ids but from my side and for many DBA’s there are some caveats on the usage of this approach:

- 4 times larger than the traditional 4-byte index value
- Cumbersome to debug
- They are not monotonically increasing

Looking for a better alternative to avoid exposing our database ids to the clients is to use hashing ids (it creates short, unique, decryptable hashes from unsigned integers). There is a library written in many languages to generate short unique ids from integers hashids and of course there is an available version for .NET [hashids.net](https://hashids.org/net/)

## Getting started

You should install the package

```PowerShell
Install-Package AspNetCore.Hashids
```

Or via the .NET Core CLI

```csharp
dotnet add package AspNetCore.Hashids
```

In your **ConfigureServices** in the **Startup** class

```csharp
public void ConfigureServices(IServiceCollection services)
{
    services
        .AddHashids(setup => setup.Salt = "your_salt");
}
```

In your DTOs, decorate the properties that you want to be hash with the **JsonConverter** attribute **HashidsJsonConverter**

```csharp
public class CustomerDto
{
    [JsonConverter(typeof(HashidsJsonConverter))]
    public int Id { get; set; }
    public int NonHashid { get; set; }
    public string FirstName { get; set; }
    public string LastName { get; set; }
}
```

It will be hashed:

```json
[
  {
    "id": "rA3d",
    "nonHashid": 10000,
    "firstName": "Luis",
    "lastName": "Ruiz"
  },
  {
    "id": "1vzX",
    "nonHashid": 20000,
    "firstName": "Unai",
    "lastName": "Zorrilla"
  }
]
```

You can encoding the ids specifying **minimum hash lenght**:

```csharp
public void ConfigureServices(IServiceCollection services)
{
    services
        .AddHashids(setup =>
        {
            setup.Salt = "your_salt";
            setup.MinHashLength = 8;
        });
}
```

and the output will be something like this:

```json
[
  {
    "id": "omrA3dl2",
    "nonHashid": 10000,
    "firstName": "Luis",
    "lastName": "Ruiz"
  },
  {
    "id": "gj1vzXlz",
    "nonHashid": 20000,
    "firstName": "Unai",
    "lastName": "Zorrilla"
  }
]
```

Also you can use the [HashidsRouteConstarint](https://github.com/Xabaril/AspNetCore.Hashids/blob/master/src/AspNetCore.Hashids/Mvc/HashidsRouteConstraint.cs) and the [HashidsModelBinder](https://github.com/Xabaril/AspNetCore.Hashids/blob/master/src/AspNetCore.Hashids/Mvc/HashidsModelBinder.cs) to convert the hashid generated in the original integer value:

```csharp
[HttpGet]
[Route("{id:hashids}")]
[Produces(MediaTypeNames.Application.Json)]
public ActionResult<CustomerDto> Get(
    [ModelBinder(typeof(HashidsModelBinder))] int id)
{
    return Ok(customers.SingleOrDefault(c => c.Id == id));
}
```

You see a full example [here](https://github.com/Xabaril/AspNetCore.Hashids/samples/WebApi) and how to modify our swagger-ui to change the type of the hashids from integers to strings.

... and last but not least a big thanks to all our [contributors](https://github.com/Xabaril/AspNetCore.Hashids/graphs/contributors)!

## Code of conduct

This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [opencode<EMAIL_ADDRESS>with any additional questions or comments.


text:
# Unit 19 PWA Homework: Text Editor

![Screenshot from 2021-10-16 19-22-30](https://user-images.githubusercontent.com/81760763/137604476-07d8e7ce-555c-417f-a742-f13f6d8997cb.png)
![Screenshot from 2021-10-16 19-23-54](https://user-images.githubusercontent.com/81760763/137604477-1f6650f2-4837-4138-8328-bf45c9a360f1.png)
![Screenshot from 2021-10-16 19-23-58](https://user-images.githubusercontent.com/81760763/137604480-4fac397c-fbbf-4768-aa51-51a09f97204e.png)

# Description

This application is a text editor with PWA packages allowing it's use online and executes IndexDB.   It will also save entry data when used within a browser.

# Deployment

You can find the  Heroku deployed version of this app here:
https://dry-sierra-05056.herokuapp.com/

# License

MIT License

Copyright (c) 2021 Lucas DiFalco

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

 IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

# Questions

If there are any further questions, please contact me via email at<EMAIL_ADDRESS>

repository: ldifalco/DiFalco-Text-Editor

repo_name: ldifalco/DiFalco-Text-Editor


text:
# Securing Electron Apps with OpenID Connect and OAuth 2.0

**TL;DR:** In this article, you will learn how to properly secure Electron applications with OpenID Connect and OAuth 2.0. To keep things as secure as possible, you will learn how to keep sensitive data away from the renderer processes and how to make these processes communicate with the main process to issue secure requests.

## Electron Overview

> Write one or two paragraphs about Electron, why its a good choice, and about its popularity.

> Add a quick paragraph mentioning that there are two types of processes: main and renderers. Then, describe these types on their own sub sections.

### Electron Main Process

> At least three sentences about this type of process. If possible, writing more sentences into two or more paragraphs would be better.

### Electron Renderer Processes

> At least three sentences about this type of process. If possible, writing more sentences into two or more paragraphs would be better.

### Why the Main Process is Securer

> Write about the main process being more secure than the renderer process. Tip: keeping sensitive data in the main process enables developers to load remote resources without worrying about leaking tokens.

## OAuth 2.0 Overview

> At least three sentences about OAuth 2.0. If possible, writing more sentences into two or more paragraphs would be better.

## OpenID Connect Overview

> At least three sentences about OpenID Connect. If possible, writing more sentences into two or more paragraphs would be better.

## Auth0 Introduction

> Write about: what Auth0 is; some of its great features; that [Auth0 is OpenID Connect certified](https://auth0.com/blog/we-are-now-open-id-certified/); and about [other certifications held by Auth0](https://auth0.com/docs/compliance).

> Mention that users will have to register to a free Auth0 account.

### Defining an Auth0 API

> Instruct readers how to create an Auth0 API to represent their backend API.

### Defining an Auth0 Application

> Instruct readers how to create an Auth0 Application to represent their native application.

## Creating the Backend API

> Quickly instruct readers how to build the `backend` project.

## Creating the Electron Application

> Instruct readers how to create the `frontend` project.

### Installing the Electron App Dependencies

- installing `devDependencies`
- installing `dependencies`

### Managing Different Environments on the Electron App

- creating `env-variables.json` file

### Persisting Data with the Electron App

- creating the `./service/store-service.js` file

### Connecting the Electron App to Auth0

- creating the `./service/auth-service.js` file
- creating the `./main/auth-process.js` file

### Creating the Electron App Main Process

- creating the `./main/app-process.js` file
- creating the `./main.js` file

> Mention why using `custom-scheme` and `custom-domain`

### Creating the Electron App Renderer Process

- creating the `./renderer/home.html` file
- creating the `./renderer/home.js` file
- creating the `./renderer/home.css` file

## Conclusion


text:
# evernote-url2clip
Clip webpages to Evernote from a text file. Using this program I successfully clipped hundreds of web pages from Chrome and Pocket automatically. The only thing I couldn't automate is enabling shortcuts for Web Clipper extension (see [demo](#demo)). Tested on Ubuntu 16.04 and on Windows 8.1.

## Demo
![](img/demo.gif)

## How to install
### Linux (tested on Ubuntu 16.04, Python 2.7.12)
* Download source
```bash
git clone https://github.com/yell/evernote-url2clip.git
cd evernote-url2clip
```
* Install dependencies
```bash
pip install -r requirements.txt
```
* Download driver for Chrome [here](https://sites.google.com/a/chromium.org/chromedriver/downloads)
and make sure its location is added to PATH (or copy to `/usr/bin/` or `/usr/local/bin/`)

* I tested program using extension (`.crx`) provided in the current directory. Also you can download newer one from https://chrome-extension-downloader.com by typing its ID (go `chrome://extensions/`, check Developer mode, and find out ID) and put in the root directory of this project.

* You can also add the script to PATH by running
```bash
make install
```

### Windows (tested on Windows 8.1, Anaconda 4.4.11)
* Download source (e.g. as zip and unzip somewhere)
* Download and install [Anaconda](https://www.anaconda.com/download/#windows)
* Version can be checked from Anaconda prompt:
```bash
conda -V
# conda 4.4.11
```
* Run Anaconda prompt and explicitly install requirements (note that tildes such as `selenium~=3.9.0` does not work with `conda install`):
```bash
conda install selenium=3.9.0
conda install tqdm=4.14.0
```
* Download [chrome driver](https://sites.google.com/a/chromium.org/chromedriver/downloads) and unzip it to `X:/Windows/System32`
* Run script in conda prompt by navigating to root folder of the project and use directions from the next section

## How to run
* Copy and fill `config.yaml.example` file with Evernote credentials.
* Run program:
```bash
usage: evernote_url2clip.py [-h] [--clip-type TYPE] [--clip-timeout SEC]
                            FILEPATH

positional arguments:
  FILEPATH            file path with urls

optional arguments:
  -h, --help          show this help message and exit
  --clip-type TYPE    how to clip content: 'A' - article, 'B' - bookmark, 'C'
                      - simplified article, 'E' - email, 'F' - full page, 'M'
                      - screenshot, 'P' - pdf, ... [see extension for more]
                      (default: A)
  --clip-timeout SEC  default timeout (in seconds) to clip pages (default:
                      120)
```

## How to migrate from Chrome (bookmarks)
Simply select and copy-paste links from Chrome bookmark manager to a text file.

## How to migrate from Pocket
Go to
```
Settings > Export > Export to HTML file
```
download HTML file (`fil_export.html` by default) and run (install `lynx`)
```bash
lynx -dump fil_export.html | awk '/http/{print $2}' > links.txt
```
to create desired text file.


text:
---
layout: post
title:      "Cloud Infrastructure and uses in Data Science"
date:       2020-09-30 08:09:02 -0400
permalink:  cloud_infrastructure_and_uses_in_data_science
---

## What is Cloud Computing?

Cloud computing is the act of running workloads within clouds—which are IT environments that abstract, pool, and share scalable resources across a network. Neither cloud computing nor clouds are technologies unto themselves.

* Cloud computing is an act—the function of running a workload in a cloud.
* Clouds are environments—places where applications run.
* Technologies are things—software and hardware used to build and use clouds.

## Whats is Cloud infrastructure?

Cloud infrastructure is a term used to describe the components needed for cloud computing, which includes hardware, abstracted resources, storage, and network resources. Think of cloud infrastructure as the tools needed to build a cloud. In order to host services and applications in the cloud, you need cloud infrastructure.

## What's included in cloud infrastructure?

Cloud infrastructure is made up of several components, each integrated with one another into a single architecture supporting business operations. A typical solution may be composed of hardware, virtualization, storage, and networking components.

As a term, cloud infrastructure can be used to describe a complete cloud computing system—once all the pieces are put together—as well as the individual technologies themselves.

## Components of cloud infrastructure

### Hardware

Although you probably think of clouds as being virtual, they require hardware as part of the infrastructure.

A cloud network is made up of a variety of physical hardware that can be located at multiple geographical locations.

The hardware includes networking equipment, like switches, routers, firewalls, and load balancers, storage arrays, backup devices, and servers.

Virtualization connects the servers together, dividing and abstracting resources to make them accessible to users.

### Virtualization

Virtualization is technology that separates IT services and functions from hardware.

Software called a hypervisor sits on top of physical hardware and abstracts the machine's resources, such as memory, computing power, and storage.

Once these virtual resources are allocated into centralized pools they’re considered clouds.

With clouds, you get the benefits of self-service access, automated infrastructure scaling, and dynamic resource pools.

### Storage

Within a single datacenter, data may be stored across many disks in a single storage array. Storage management ensures data is correctly being backed up, that outdated backups are removed regularly, and that data is indexed for retrieval in case any storage component fails.

Virtualization abstracts storage space from hardware systems so that it can be accessed by users as cloud storage.

When storage is turned into a cloud resource, you can add or remove drives, repurpose hardware, and respond to change without manually provisioning separate storage servers for every new initiative.

### Network

The network is composed of physical wires, switches, routers, and other equipment. Virtual networks are created on top of these physical resources.

A typical cloud network configuration is composed of multiple subnetworks, each with varying levels of visibility. The cloud permits the creation of virtual local area networks (VLANs) and assigns static and/or dynamic addresses as needed for all network resources.

The cloud resources are delivered to users over a network, such as the internet or an intranet, so you can access cloud services or apps remotely on demand.

## Type of Cloud

### Public Cloud

A public cloud environment is owned by an outsourced cloud provider and is accessible to many businesses through the internet on a pay-per-use model. This deployment model provides services and infrastructure to businesses who want to save money on IT operational costs, but it’s the cloud provider who is responsible for the creation and maintenance of the resources.

Public clouds are ideal for small and medium sized businesses with a tight budget requiring a quick and easy platform in which to deploy IT resources.

#### Pros of a public cloud

* Easy scalability
* No geographical restrictions
* Cost effective
* Highly reliable
* Easy to manage

#### Cons of a public cloud

* Not considered the safest option for sensitive data

### Private Cloud

This cloud deployment model is a bespoke infrastructure owned by a single business. It offers a more controlled environment in which access to IT resources is more centralised within the business. This model can be externally hosted or can be managed in-house. Although private cloud hosting can be expensive, for larger businesses it can offer a higher level of security and more autonomy to customise the storage, networking and compute components to suit their IT requirements.

#### Pros of a private cloud

* Improved level of security
* Greater control over the server
* Customisable

#### Cons of a private cloud

* Harder to access data from remote locations
* Requires IT expertise

### Hybrid Cloud

For businesses seeking the benefits of both private and public cloud deployment models, a hybrid cloud environment is a good option. By combining the two models, a hybrid cloud model provides a more tailored IT solution that meets specific business requirements.

#### Pros of a hybrid cloud

* Highly flexible and scalable
* Cost effective
* Enhanced security

#### Cons of a hybrid cloud

* Communication in network level may be conflicted as it’s used in both private and public clouds.

## Cloud Services

There are three main service models of cloud computing – Infrastructure as a Service (IaaS), Platform as a Service (PaaS) and Software as a Service (SaaS). There are clear differences between the three and what they can offer a business in terms of storage and resource pooling, but they can also interact with each other to form one comprehensive model of cloud computing.

#### IaaS (Infrastructure as Service)

This is the most common service model of cloud computing as it offers the fundamental infrastructure of virtual servers, network, operating systems and data storage drives. It allows for the flexibility, reliability and scalability that many businesses seek with the cloud, and removes the need for hardware in the office. This makes it ideal for small and medium sized organisations looking for a cost-effective IT solution to support business growth. IaaS is a fully outsourced pay-for-use service and is available as a public, private or hybrid infrastructure.

#### PaaS (Platform-as-a-Service)

This is where cloud computing providers deploy the infrastructure and software framework, but businesses can develop and run their own applications. Web applications can be created quickly and easily via PaaS, and the service is flexible and robust enough to support them. PaaS solutions are scalable and ideal for business environments where multiple developers are working on a single project. It is also handy for situations where an existing data source (such as CRM tool) needs to be leveraged.

#### SaaS (Software as a Service)

This cloud computing solution involves the deployment of software over the internet to variousbusinesses who pay via subscription or a pay-per-use model. It is a valuable tool for CRM and for applications that need a lot of web or mobile access – such as mobile sales management software. SaaS is managed from a central location so businesses don’t have to worry about maintaining it themselves, and is ideal for short-term projects.

### Platforms of Cloud Infrastructure

![](https://859766.smushcdn.com/1876234/wp-content/uploads/2020/04/Cloud-provider.jpg?lossy=1&strip=1&webp=1)

#### 1. Amazon Web Services:

![](https://859766.smushcdn.com/1876234/wp-content/uploads/2020/04/AWS.png?lossy=1&strip=1&webp=1)

Amazon Web Services (AWS) is a subsidiary of Amazon (a leading company in eCommerce). Under the umbrella term of AWS, Amazon provides on-demand cloud computing platforms like storage, data analysis, etc. With a whopping 35% market share, Amazon lends its services to individuals, companies, and governments. Amazon Web Services allow their subscribers to enjoy a full-fledged virtual cluster of computers, at any time, based on their requirements. The entire service is enabled through the internet.

Among all the cloud service providers Amazon is considered is the most powerful and flexible solution. AWS’s virtual cloud platform comes with most of the attributes of an actual computer including hardware (CPU(s) & GPU(s) for processing, hard-disk/SSD for storage & local/RAM for memory); an operating system to choose from and pre-loaded apps like web servers, databases, CRM, etc.

#### 2. Google Cloud Platform:

![](https://859766.smushcdn.com/1876234/wp-content/uploads/2020/04/google-cloud-platform-300x191.png?lossy=1&strip=1&webp=1)

Google offers its public cloud computing solutions with the name of Google Cloud Platform, or GCP. It offers services in all major spheres including compute, networking, storage, machine learning (ML) and the internet of things (IoT). It also includes tools for cloud management, security, and development. The Google Cloud Storage is a highly dynamic storage solution that supports both SQL (Cloud SQL) and NoSQL (Cloud Datastore) database storage.

The Google Compute Engine (Infrastructure as a Service, or IaaS) provides users with virtual machine instances for workload hosting. The Google App Engine (Platform as a Service, or PaaS) offers software developers access to Google’s on-demand hosting and a software development kit (SDK) to develop applications that run on app engine. All these services can be accessed using a public internet or via dedicated networks.

Google Cloud Platform or GCP is the home of Kubernetes. Kubernetes was developed by Google in 2014 as open source. Google Kubernetes Engine or GKE provides users with a managed environment for the deployment, management, and scaling of containerized applications with the help of Google infrastructure.

#### 3. Microsoft Azure:

![](https://859766.smushcdn.com/1876234/wp-content/uploads/2020/04/Windows_Azure_logo-300x69.png?lossy=1&strip=1&webp=1)

Microsoft Azure (formerly Windows Azure) is the cloud computing service from Microsoft. Enabled primarily through Microsoft-managed data centers, this service proves to be a reliable solution, especially for Microsoft evangelists. Like the above solutions, it supports the development, test, deployment, and management of applications and services. For web development, it offers support for PHP, ASP.net, and Node.js.

The Windows Azure is used to deploy code on Microsoft’s servers. This code holds access to local storage resources (blobs, queues, and tables). While the SQL Azure it is not a full SQL Server instance it can be integrated with SQL Server. The security features like authentication, security, etc. are supported using Azure AppFabric that allows applications within your LAN to communicate with Azure cloud. Overall it is a complete package that supports development, management as well security of applications.

#### 4. IBM Bluemix:

![](https://859766.smushcdn.com/1876234/wp-content/uploads/2020/04/IBM_Bluemix_logo.svg_-300x297.png?lossy=1&strip=1&webp=1)

IBM Bluemix is the cloud computing solution from IBM that comes with the platform (PaaS) as well as infrastructure as a service (IaaS) offerings. With Bluemix IaaS, users can deploy and access virtualized compute power, storage and networking using the internet. The IBM service offerings can be used as a public, private or hybrid model whichever suits an organization’s need.

The IBM Bluemix PaaS is based on an open source cloud platform – Cloud Foundry. IBM services can be used by developers to create, manage, run and deploy scalable applications for both public cloud and on-premises environments. The programming languages that are supported by IBM Bluemix include Java, Node.js, PHP, and Python. The solution can even be extended to support other languages.

#### 5. Alibaba:

![](https://859766.smushcdn.com/1876234/wp-content/uploads/2020/04/alibaba-cloud-300x51.png?lossy=1&strip=1&webp=1)

Alibaba Cloud is the cloud initiative of the Chinese eCommerce giant – Alibaba Group. The Alibaba services dominate the Chinese market and have roots all across the globe with headquarters in Hangzhou, China. It was established in 2009, just three years after the incorporation of Amazon Web Services. Being in the same business as Amazon it is often considered the toughest competitor of AWS.

Alibaba Cloud began as a private cloud which was meant to support the internal infrastructural needs of the Alibaba Group but it soon scaled to a public cloud solution provider. The Alibaba Cloud computing services cover all the major cloud services like elastic computing, hosting, object storage, relational database (SQL), NoSQL database (using the Table Store), big data analytics (Hadoop), Machine Learning (ML) and Artificial Intelligence (AI).

If you are looking for a suitable multi cloud solutions for your enterprise needs then feel free to contact us. With a deep expertise in cloud management and development, big data and machine learning, we can suggest an appropriate solution for your needs and help you securely move to cloud.

extension: markdown


text:
# National Basketball Association (NBA) Game Outcome Projections

![NBA Logo](https://github.com/abewoycke/NBA-Projections/blob/master/5_Documentation/nba_logo.jpg)

_Who will win each NBA game? This classifier takes inputs of both team's various team statistics entering a game (i.e. 2-pt FG %, ASTs/game, winning %) and outputs the probability of each team winning._

__1. Data__

A wonderful CSV file containing game data from 2004-2020 was available at: https://www.kaggle.com/nathanlauga/nba-games

__2. Data Wrangling/Cleaning__

[Data Wrangling Notebook](https://github.com/abewoycke/NBA-Projections/blob/master/2_Data_Wrangling/NBA%20Projections%20Data%20Wrangling.ipynb)

I dealt with a handful of null/missing values. I verified some individual player stats that had been missing to make sure that missing values should be filled with 0. This was accurate.

I made several functions to transform the data into the form of observations that would be most useful. I wrote a function that compiles team's individual games in a season up to the game in question and returns their stats coming into that game. The resultant data frame has rows that are both the home and away team's stats entering the game, along with the game result.

__3. EDA__

[EDA Notebook](https://github.com/abewoycke/NBA-Projections/blob/master/3_EDA/NBA%20Projections%20EDA.ipynb)

I did some manual feature transformations based on my domain knowledge. My main goal was to remove redundancies/collinearity in the predictor variables (i.e. transforming 2-pt FGM and 2-pt FGA into 2-pt FG%). I also used a Bayesian prior for each team's win percentage so that early season win percentages (that would normally be at 0% or 100%) wouldn't be outliers.  The hope is that this would make for more meaningful predictor features, and that I wouldn't have to limit the range of available classifier models to those that deal with multicollinearity well.

![Variable Heatmap](https://github.com/abewoycke/NBA-Projections/blob/master/3_EDA/heatmap.png)

The resultant heat map showed a lot of data relationships that were predictable in their direction, although interesting in their magnitude. The top 5 variables correlated with the home team winning, in order of correlation, were:

1. Winning %
2. 2-point FG%
3. Assists per Game
4. Defensive Rebounds per Game
5. 3-point FG%

![Point Plot of Variable Relationships](https://github.com/abewoycke/NBA-Projections/blob/master/3_EDA/Normalized_Pointplot.png)

The above plot also shows the variable relationships. The more the orange line is to the right of the blue line, the more the home team being better in that statistic entering the game impacts their expected win %. These results mostly align with a normal basketball viewer's expectations. For example, if the home team has more assists per game than the away team, the home team is more likely to win the game.

Conversely, if the blue line is to the right of the orange line, the home team having a higher value in this category is negatively correlated with the home team's probability of winning the game. These statistics are things normally thought to be bad for a team. including turnovers and personal fouls.

One variable I was surprised by was teams having higher offensive rebounds are more likely to _lose_. This goes against my intuition. But perhaps you only have more chances for offensive rebounds if you miss shots!

__4. Modeling and Tuning__

[Modeling Notebook](https://github.com/abewoycke/NBA-Projections/blob/master/4_Preprocessing_Modeling/NBA%20Projections%20Preprocessing%20Modeling%20Clean.ipynb)

I focused on three models: GradientBoostingClassifier, a Neural Network, and a Logistic Regression. I also built hard and soft voting classifiers from those three. I used Area Under the Receiver Operator Curve (AUROC) as my primary metric for performance, because:

a) in this use case, I don't place a stronger value on either Type 1 or Type 2 errors

b) AUROC is better than pure accuracy in this case when the data is imbalanced (home teams win 59.6% of the time)

After tuning all three of the models with a combination of HyperOpt, GridSearch, and RandomSearch, the tuned __GradientBoostingClassifier__ performed the best with an AUROC of 0.697 and a classification accuracy of 66.2%.

__5. Model Predictions__

[Documentation Notebook](https://github.com/abewoycke/NBA-Projections/blob/master/5_Documentation/NBA%20Projections%20Documentation.ipynb)

The model performed fairly well. The accuracy being about 66% is good when compared with some other classifications (expert consensus picks come in at ~59%). This represents an ~11% increase in relative performance, or a 7% increase in classification accuracy overall.

![Roc Curve](https://github.com/abewoycke/NBA-Projections/blob/master/5_Documentation/roc.png)

The overall AUROC score is alright. I think that NBA games are inherently hard to predict at too high a clip. An AUROC in the area of 0.7 represents that the model makes somewhat more meaningful predictions than just randomly guessing or always guessing the dominant class.

![Precision Recall Curve](https://github.com/abewoycke/NBA-Projections/blob/master/5_Documentation/precision_recall.png)

In this case, I don't place a different value on Type 1 or Type 2 errors, so don't have a strong opinion on maximizing precision or recall.

![Confusion Matrix](https://github.com/abewoycke/NBA-Projections/blob/master/5_Documentation/confusion_matrix.png)

I would be interested in seeing if there are tweaks that could be made that would decrease the false positive rate. I attribute the high false positives to class imbalance (home teams win 59% of the time).

__6. Future Improvements__

If the data is available, I would be interested in expanding the dataset and looking at data before 2004. I would also be interested in building a pipeline to simulate full seasons using the dataset. Also, I would be interested in seeing how the model performs against truly unseen data with pre-2004 backtesting, and how it will perform in the upcoming 2020-2021 season (although that might be a weird sample due to limited home fan attendance).

__7. Project Writeup:__

[Final Project PDF](https://github.com/abewoycke/NBA-Projections/blob/master/5_Documentation/Final%20Project%20Deliverables/NBA%20Game%20Outcome%20Classifier%20Final%20Project%20Report.pdf)

__8. Credits__

Scikit-learn: Machine Learning in Python, Pedregosa et al., JMLR 12, pp. 2825-2830, 2011.


text:
ncr-bundle-php
=============

Symfony bundle that integrates [gdbots/ncr](https://github.com/gdbots/ncr-php) library.

# Configuration
Follow the standard [bundle install](http://symfony.com/doc/current/bundles/installation.html) using __gdbots/ncr-bundle__ as the composer package name.

> The examples below assume you're running the DynamoDb Ncr and Elastica NcrSearch.

__config/packages/ncr.yml:__

```yaml
# many of the configurations below are defaults so you can remove them
# from your configuration, added here for reference

parameters:
  cloud_region: 'us-west-2'
  es_clusters:
    default:
      debug: '%kernel.debug%'
      timeout: 300 # default
      persistent: true # default
      round_robin: true # default
      servers:
        - {host: '<IP_ADDRESS>', port: 9200} # default

gdbots_ncr:
  ncr:
    provider: dynamodb
    memoizing:
      enabled: true # default
      #
      # IMPORTANT read_through notes for the memoizer
      #
      # If true, the NcrCache will be updated when a cache miss occurs.
      # When the Pbjx request bus is in memory, then you'd want this
      # to be false so there aren't two processes updating the NcrCache.
      # One is the memoizer and the other are event listeners which are
      # updating cache after successful get node requests.
      read_through: false # default
    psr6:
      enabled: true # default
      # by default this cache is the same cache as your application
      # it is recommend to configure this separately because Ncr items
      # can be in cache much longer than app cache, for example, they
      # don't need to be cleared on app deployments.
      provider: cache.app # default
      read_through: true # default
      # to customize the cache keys or tweak cache times
      # you must provide your own class and override those methods.
      # see Psr6Ncr::getCacheKey and Psr6Ncr::beforeSaveCacheItem
      #class: Acme\Ncr\Repository\Psr6Ncr
    ncr_cache:
      max_items: 500 # default
    dynamodb:
      config:
        # these apply to batch get operations
        batch_size: 100 # default
        concurrency: 25 # default
      table_manager:
        # multi-tenant applications will likely need to provide a custom
        # table manager so node tables can be derived at runtime.
        # class: Acme\Ncr\Repository\DynamoDb\TableManager
        table_name_prefix: my-ncr
        node_tables:
          # any SchemaCurie not defined here will end up using the default
          # the entire default key is not needed unless you're changing it
          default:
            class: Gdbots\Ncr\Repository\DynamoDb\NodeTable # default
            table_name: multi # default
          'acme:article':
            table_name: article
          'acme:user':
            # class is optional but needed if you want to have custom
            # global secondary indexes or customize items before they
            # are put into the table.
            class: Acme\Ncr\Repository\DynamoDb\UserTable
            table_name: user
  ncr_search:
    provider: elastica
    elastica:
      # your app will at some point need to customize the queries
      # override the class so you can provide these customizations.
      class: Acme\Ncr\Search\Elastica\ElasticaNcrSearch
      query_timeout: '500ms' # default
      clusters: '%es_clusters%'
      index_manager:
        # multi-tenant apps will probably need to use a custom class
        #class: Acme\Ncr\Search\Elastica\IndexManager
        index_prefix: my-ncr
        indexes:
          default:
            number_of_shards: 5 # default
            number_of_replicas: 1 # default
        types:
          # types do not need to be configured unless there is a custom
          # mapping or they must be put into different indexes or types
          # than the automatic handling would provide
          'acme:user':
            index_name: members # default is "default"
            mapper_class: Acme\Ncr\Search\Elastica\UserMapper

# typically these would be in services.yml file.
services:
  # If you are using AWS ElasticSearch service, use AwsAuthV4ClientManager
  gdbots_ncr.ncr_search.elastica.client_manager:
    class: Gdbots\Ncr\Search\Elastica\AwsAuthV4ClientManager
    arguments:
      - '@aws_credentials'
      - '%cloud_region%'
      - '%es_clusters%'
      - '@logger'
    tags:
      - {name: monolog.logger, channel: ncr}

  # recommended, create your own lazy loading handler
  # to optimize batch requests.
  acme_ncr.ncr_lazy_loading_handler:
    class: Acme\Ncr\NcrLazyLoadingHandler
    public: false
    arguments: ['@ncr_lazy_loader']
    tags:
      - {name: pbjx.event_subscriber}

```

# Controllers
It is recommended to have data retrieval be the responsibility of Pbjx requests, however, that strategy doesn't work for all uses cases.  Use Symfony autowiring and typehint the interface in your constructor or setter methods to get key Ncr services.

Autowiring supported for these interfaces:

* `Gdbots\Ncr\Ncr`
* `Gdbots\Ncr\NcrCache`
* `Gdbots\Ncr\NcrLazyLoader`
* `Gdbots\Ncr\NcrSearch`

# Twig Extension
The `NcrExtension` provides a function called `ncr_get_node`.  It is important to note that this does __NOT__ make a query to get a node, instead it pulls from `NcrCache`.

> This might change in the future, but this strategy eliminates horribly performing twig templates that make Ncr queries.

The function will accept a `MessageRef`, `NodeRef` or a string version of a NodeRef.

__Example use of ncr_get_node:__

```txt
{% set created_by = ncr_get_node(pbj.get('created_by_ref')) %}
Here is the creator:
{{ created_by }}
```

__Other twig functions (documentation wip):__

+ ncr_deref_nodes
+ ncr_get_preloaded_nodes
+ ncr_get_preloaded_published_nodes
+ ncr_is_node_published
+ ncr_preload_node
+ ncr_preload_nodes
+ ncr_preload_embedded_nodes
+ ncr_to_node_ref

# Console Commands
This library provides the basics for creating and extracting data from the Ncr services. Run the Symfony console and look for __ncr__ commands.

```txt
ncr:create-search-storage    Creates the NcrSearch storage.
ncr:create-storage           Creates the Ncr storage.
ncr:describe-search-storage  Describes the NcrSearch storage.
ncr:describe-storage         Describes the Ncr storage.
ncr:export-nodes             Pipes nodes from the Ncr to STDOUT.
ncr:get-node                 Fetches a single node by its NodeRef and writes to STDOUT.
ncr:reindex-nodes            Pipes nodes from the Ncr and reindexes them.
ncr:sync-node                Syncs a single node from the Ncr with the EventStore.
ncr:sync-nodes               Syncs nodes from the Ncr with the EventStore.
```

Review the `--help` on the ncr commands for more details.

repository: gdbots/ncr-bundle-php

gha_language: PHP

repo_name: gdbots/ncr-bundle-php


text:
ptv2gtfs
========

This is a conversion script to translate from the standard Metlink/PTV SQLite3
timetable database format used by PTV applications (e.g. iPhone app) to GTFS.

Requirements
------------

   * Python v2.7
   * [Google Transitfeed] [transitfeed]
   * A MetLink/PTV database (SQLite3 DB)

This has only been tested on Linux and Mac OS X using Python 2.7.
It should probably work on any platform.

Obtaining the data
------------------

Currently, obtaining and using this database is up to you, and may or may not
be legal. I believe the Department of Transport may not like it.

You may be better off to use the [PTV Timetable API] [timetable_api].

Usage
-----

Command line help looks like this:

```
Usage: ptv2gtfs.py --file <input db> --service <service> --output <output file>

Options:
  -h, --help                     show this help message and exit
  -f INPUTDB, --file=INPUTDB     PTV SQLite3 database file
  -s SERVICE, --service=SERVICE  Should be train, tram or bus
  -o OUTPUT, --output=OUTPUT     Path of output file (should end in .zip)
```

An example of the output used for tram timetable data looks like this:

```
$ ./ptv2gtfs.py --file Metlink-Tram.sql --service tram --output tram_gtfs.zip
Processing timetable: fri
Processing timetable: monfri
Processing timetable: monthur
Processing timetable: sat
Processing timetable: sun
All services are defined on a weekly basis from 2014-01-01 to 2014-12-31 with
no single day variations. If there are exceptions such as holiday service
dates please ensure they are listed in calendar_dates.txt
The stops "Batmans Hill Dr/Collins St" (ID 2701) and "Batman's Hill/700
Collins St" (ID 2489) are 0.00m apart and probably represent the same
location.
The stops "Cotham Rd/Burke Rd" (ID 2695) and "1219 Burke Rd" (ID 2436) are
0.00m apart and probably represent the same location.
```

The output zip file (tram_gtfs.zip) in this case is our resulting GTFS data
file.

Performance: This script can take up to 10 minutes to run on a large DB, such
as Victoria's bus network.
It will also require over 100MB of RAM (e.g., between 800MB - 1GB for the bus
network).

Once the data has been processed, you can use the schedule_viewer.py script
provided with Google's Transitfeed package to visualise the data:

```
$ schedule_viewer.py train_gtfs.zip
Loading data from feed "train_gtfs.zip"...
(this may take a few minutes for larger cities)
routes.txt:1 column route_short_name
Missing column route_short_name in file routes.txt
To view, point your browser at http://localhost:8765/
```

Limitations
------------

Currently this script does NOT deal with special timetable days, like public
holidays.

You may also find inconsistancies within the data provided. This is up to you
to resolve.

Contact
-------

Feel free to contact me at<EMAIL_ADDRESS>for any questions.

If you find this useful, please let me know. Pull requests appreciated.

[timetable_api]: https://www.data.vic.gov.au/raw_data/ptv-timetable-api/6056
[transitfeed]: http://code.google.com/p/googletransitdatafeed


text:
# jnaOSearch

Find buffer color theme for Sublime text plugin [BetterFindBuffer](https://github.com/aziz/BetterFindBuffer). The base is originally from [solarized](http://ethanschoonover.com/solarized)

## Installation

* Install [BetterFindBuffer](https://github.com/aziz/BetterFindBuffer)
* Go to sublime packages folder (`~/Library/Application Support/Sublime Text 3/Packages`)
* Clone this repo
* In `~/Library/Application Support/Sublime Text 3/Packages/User`, add a file called `Find Results.sublime-settings` and paste the code below:

``` json
{
  "color_scheme": "Path to your custom color scheme file. e.g. Packages/jnaOSearch/jnaOSearch.tmTheme",
}
```

### Other things of interest...

...if this is:
* [jnaOSol](https://github.com/jnaO/jnaOSol)
* [jnaOTheme](https://github.com/jnaO/jnaOTheme)

## Preview

Displayed below:

![jnaOTheme](screenshot.png)

# License

The MIT License (MIT)

Copyright (c) 2012 jnaO

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

 IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

repository: jnaO/jnaOSearch

repo_name: jnaO/jnaOSearch


text:
> This project has two homes.
> It is ok to work in github, still, for a better decentralized web
> please consider contributing (issues, PR, etc...) throught:
>
> https://gitlab.esy.fun/yogsototh/ylog

---

# `ylog`

`ylog` is a simple command line note taking tool.

Example:

~~~
> cd project
> ylog init # <-- create a log file (.log)
> cd subdir/of/project
> ylog It took me very long to verify the file foo.hs
> ylog todo rename foo.hs to bar.hs
> cd ../../../another/subdir
> ylog There is another foo.hs here
> ylog show
[13-11-2013] (subdir/of/project) It took me very long to verify the file foo.hs
[13-11-2013] (subdir/of/project) [ ] rename foo.hs to bar.hs
[13-11-2013] (another/subdir) There is another foo.hs here
> ylog edit <-- manually edit the file ; use EDITOR environment variable
~~~

Of course you could always add aliases to make it easier to use:

~~~
> alias n=ylog
> alias t="ylog todo"
~~~

## Install

~~~
git clone https://github.com/yogsototh/ylog.git
~~~

Then copy the `ylog` file into one of the directory of your $PATH.
To get the list simply type:

~~~
> echo $PATH | sed 's/:/\n/g'
~~~

It uses zsh which is pretty common in UNIX world.

repository: yogsototh/ylog

repo_name: yogsototh/ylog


text:
onyx
====

miRNA Differential Expression Analysis

![](mi_RNA_pipeline_summary.jpg)

Onyx allows the annotation and calculus of know miRNA differential expression levels from High-Throughput sequencing data.This Pipeline consist in Quality Analysis, Trimming, Mapping, Annotation and Expression Analysis.According with the studies puublished It is compatible with match-pair and non match-pair experimental design. The output is a summary of the normalized levels (TMM), the log Fold Change (logFC), the P Value, and the False Discovery Rate (FDR).

repository: Katospiegel/onyx

repo_name: Katospiegel/onyx


text:
# AlBeeChatbot - ArtificialIntelligenceBot
## Working Toward to a BotOFBots with implementations of DialogFlow (Google) and LEX (AWS)

```__For derivatives or redistribution of this opensource work, please include the message to support the awareness of People with Special Needs (SUPPORT.md) below.__```

![alt text](https://github.com/ArtificialIntelligenceBot/AlBeeChatbot/blob/master/config/artwork/AlBeeChatAngelBot680.png?raw=true "AlBee Love Angels Chatbot Logo")

## What is AlBeeChatbot
AlBee Chatbot is a feature-rich Customer Engagement platform designed to deliver great customer experience through its optimized Conversational UX (User Experience) and Interact Analysis — gaining genuine insight into the mind-set of their target audience. AlBee Chatbot further connects customers with its integrated Branding and Marketing Messaging tools.

AlBee Chatbot intent to provide continue integration with Commercial and Opensource Artificial Intelligence and Machine Learning platforms, i.e. DialogFlow or RASA.

 **Al** = Albert (my son) with his favor (**Bee**) costume:
![alt text](https://github.com/ArtificialIntelligenceBot/AlBeeChatbot/blob/master/public/images/albee128.png?raw=true?raw=true "Logo AlBee Chatbot App")

---
**We support children and adults with disabilities and special needs.**

May I invite you to learn more about **Angelman syndrome**?
```Angelman syndrome (AS) is a rare neuro-genetic disorder that occurs in one in 15,000 live births. Angelman syndrome is often misdiagnosed as cerebral palsy or autism due to lack of awareness. Characteristics of the disorder include developmental delay, lack of speech, seizures, and walking and balance disorders. Individuals with Angelman syndrome will require life-long care.```

![alt text](https://www.angelman.org/wp-content/uploads/2015/10/Normal_15_Chromosome.jpg "") We each have two number 15 chromosomes, one inherited from our mother (M.) and one inherited from our father (P, paternal). The Angelman syndrome gene (UBE3A) is located at chromosome 15, band q12, as depicted. In the brain, the Angelman gene is primarily expressed from the maternally inherited chromosome 15. The diagrams below illustrate the four known genetic mechanisms that cause Angelman syndrome

Ready to make an [impact](https://www.angelman.org/make-an-impact/)?

##

repository: ArtificialIntelligenceBot/AlBeeChatbot

repo_name: ArtificialIntelligenceBot/AlBeeChatbot


text:
# v2

- `createLogger(<config-and-fields>, <fields>)` changes (#460)
    - see section below
- the dtrace-provider thing (#487)
    TODO: answer Cody email
- use package.json version for VERSION
- use deps
    - dashdash
    - assert-plus?
    - verror?
- break out to multiple files
    - want to work through PRs before that, so don't just break them all
- TODO: a quick pass through tickets and pulls for other things to include
- get ticket refs for the above, if any
- formatters: read up again on `glp master..1.x`
- support for customer formatters
    - for the CLI as well? How? ~/.bunyanrc?
- if doing ~/.bunyanrc, then consider color schemes

# changes to ctor and log.child to separate fields from config

<https://github.com/trentm/node-bunyan/issues/460>

Current:

    createLogger(<config-and-fields>)
    log.child(<config-and-fields>, <just-fields-bool>)

Could be:

    createLogger(<config-and-fields>, <fields>)
    log.child(<config-and-fields>, <fields>)
        # Still support: log.child(<config-and-fields>, <just-fields-bool>)

Pros: Compat issues are minimal: a change is only required if there is a
collision with used field and a new config var name.
Cons: A *slight* con is that my guess is the common usage of child is
`log.child(<fields>)`, so the more future-proof common usage becomes:

    log.child(null, <fields>)

That's not too bad. It is clearer at least than:

    log.child(<fields>, true)

TODO:

- is there a ticket for this work already?  #460
- make the change
- do a migration guide? i.e. provide the grep commands to find all
  possible calls to inspect. E.g. if don't have `rg logUndefined` in your
  code, then you are fine. And one time future-proofing via changing
  to fields in the *second* arg.
- list of issues/pulls that wanted to add new config fields

# higher prio

- published organized advice for
  https://github.com/trentm/node-bunyan/issues/37 so can close that out.
  Perhaps a wiki page with examples and strategies.
- man page for the bunyan CLI (refer to it in the readme)
    - perhaps wait for a bunyan new version with deps, and use dashdash
      with a (vapour) man page generator

# docs

- document log.addStream() and log.addSerializers()

# someday/maybe

- 2.0 (?) with `v: 1` in log records. Fwd/bwd compat in `bunyan` CLI
- `tail -f`-like support
- full-on docs
- better examples/
- better coloring
- look at pino (bunyan style, perf benefits), also logpp (https://github.com/mrkmarron/logpp)
- would be exciting to have bunyan support in http://lnav.org/ if that
  made sense
- "template" support for 'rotating-file' stream to get dated rolled files
- "all" or "off" levels? log4j? logging.py?
  logging.py has NOTSET === 0. I think that is only needed/used for
  multi-level hierarchical effective level.
- buffered writes to increase speed:
    - I'd start with a tools/timeoutput.js for some numbers to compare
      before/after. Sustained high output to a file.
    - perhaps this would be a "buffered: true" option on the stream object
    - then wrap the "stream" with a local class that handles the buffering
    - to finish this, need the 'log.close' and `process.on('exit', ...)`
      work that Trent has started.
- "canWrite" handling for full streams. Need to buffer a la log4js
- test file log with logadm rotation: does it handle that?
- test suite:
    - test for a cloned logger double-`stream.end()` causing problems.
      Perhaps the "closeOnExit" for existing streams should be false for
      clones.
    - test that a `log.clone(...)` adding a new field matching a serializer
      works *and* that an existing field in the parent is not *re-serialized*.
- split out `bunyan` cli to a "bunyan" or "bunyan-reader" or "node-bunyan-reader"
  as the basis for tools to consume bunyan logs. It can grow indep of node-bunyan
  for generating the logs.
  It would take a Bunyan log record object and be expected to emit it.

        node-bunyan-reader
            .createReadStream(path, [options]) ?

- coloring bug: in less the indented extra info lines only have the first
  line colored. Do we need the ANSI char on *each* line? That'll be
  slower.
- document "well-known" keys from bunyan CLI p.o.v.. Add "client_req".
- More `bunyan` output formats and filtering features.
- Think about a bunyan dashboard that supports organizing and viewing logs
  from multiple hosts and services.
- doc the restify RequestCaptureStream usage of RingBuffer. Great example.
- A vim plugin (a la http://vim.cybermirror.org/runtime/autoload/zip.vim ?) to
  allow browsing (read-only) a bunyan log in rendered form.
- Some speed comparisons with others to get a feel for Bunyan's speed.
- what about promoting 'latency' field and making that easier?
- `log.close` to close streams and shutdown and `this.closed`
  process.on('exit', log.close)
  -> 'end' for the name
- bunyan cli: more layouts (http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/EnhancedPatternLayout.html)
  Custom log formats (in config file? in '-f' arg) using printf or hogan.js
  or whatever. Dap wants field width control for lining up. Hogan.js is
  probably overkill for this.
- loggly example using raw streams, hook.io?, whatever.
- serializer support:
    - restify-server.js example -> restifyReq ? or have `req` detect that.
      That is nicer for the "use all standard ones". *Does* restify req
      have anything special?
    - differential HTTP *client* req/res with *server* req/res.
- statsd stream? http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/
  Think about it.
- web ui. Ideas: http://googlecloudplatform.blogspot.ca/2014/04/a-new-logs-viewer-for-google-cloud.html

repository: trentm/node-bunyan

gha_language: JavaScript

gha_license_id: NOASSERTION

repo_name: trentm/node-bunyan


text:
### Contributing to python-demandapi-client

This is an InnerSource python project. It is the work of someone who thought it might benefit someone else in the company as well.

### Maintainers

This repository is maintained by

1. [Ridley Larsen](https://github.com/RidleyLarsen)
1. [Uthayakumar Kumarasamy](https://github.com/ukumark)
1. [Bradley Wogsland](https://github.com/wogsland)
1. [Nathan Workman](https://github.com/nathanworkman)

### Community Guidelines

This project follows the [Github Community Guidlines](https://help.github.com/en/github/site-policy/github-community-guidelines). Please feel free to contact the maintainers with any issues.

### Contributing Code

The best place to start is by looking at the [GitHub Issues](https://github.com/dynata/python-demandapi-client/issues) to see what needs to be done. If you decide to take on a task make sure to comment on it so work isn't duplicated. A good PR completely resolves the associated issue, passes python linting, and includes test coverage for your new code. This Github repository is integrated with CircleCI, so a PR cannot be accepted that has merge conflicts, fails to pass linting or tests, or lowers the repository's test coverage. Additionally your PR should include a high level description of your work or reviewers will be peppering you with questions. Approval of the maintainer is required merge a PR into `dev`, which is where all PRs go.

### Linting

Linting software is strongly recommended to improve code quality and maintain readability in Python projects. Python's official linting package is called pycodestyle, but another useful linting package is called flake8. Flake8 runs three different linters on your code, including pycodestyle, and a package called PyFlakes that checks for things like unused imports.

Read more [here](http://flake8.pycqa.org/en/latest/)

To lint the files,

    virtualenv venv
    . venv/bin/activate
    pip install flake8
    flake8 . --count --max-complexity=10 --max-line-length=127 --statistics
    deactivate

### Testing

To run the tests,

    virtualenv venv
    . venv/bin/activate
    pip install -r requirements.txt
    pytest tests
    deactivate

to run the tests for this project.

### Filing Issues

Please use the [GitHub Issues](https://github.com/dynata/python-demandapi-client/issues/new) to file an issue.

Thats it.

repository: dynata/python-demandapi-client

branch_name: refs/heads/dev

gha_language: Python

gha_license_id: MIT

repo_name: dynata/python-demandapi-client