text:
# Soozie [![Build Status](https://travis-ci.org/izhangzhihao/soozie.svg?branch=master)](https://travis-ci.org/izhangzhihao/soozie) [![](https://img.shields.io/nexus/s/https/oss.sonatype.org/com.github.izhangzhihao/soozie_2.12.svg)](https://oss.sonatype.org/content/repositories/snapshots/com/github/izhangzhihao/soozie_2.12/0.6-SNAPSHOT/)

## Getting Soozie

The current *SNAPSHOT* version is 0.6-SNAPSHOT, which is built against 2.12.x.

If you're using SBT, add the following line to your build file:

```
libraryDependencies += "com.github.izhangzhihao" %% "soozie" % "0.6-SNAPSHOT"
```

and add sonatype snapshots repo:

```
resolvers += "Sonatype OSS Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots"
```

For Gradle / Maven and other build tools, you can visit [Sonatype OSS Snapshots Repo](https://oss.sonatype.org/content/repositories/snapshots/com/github/izhangzhihao/soozie_2.12/0.6-SNAPSHOT/).

## Why?

Soozie is designed to solve problems concerning developer productivity when creating and running oozie workflow jobs.

### Problems that Soozie solves:

1. Job hierarchy and re-use: No more copy/pasting XML to create a new job
2. Type and syntax checking: No more errors from typos in XML
3. Safety / Verification: Scala provides a type-safe environment.
   Various other verifications and sanity checks are made. For example,
   it is impossible to create a cyclic graph or provide a fork/join
   structure that is not supported by oozie.
4. Developer Overhead: The Soozie DSL code makes it easy to read and
   write workflows - no more messy XML.
5. Multiple sources of truth: The Soozie specification of a job will
   be the source of truth for any workflow

## How it works:

There are three main layers to Soozie:

1. Specification of job in DSL. This encapsulates the syntactic sugar
   that makes it easy to read and write Soozie workflows. The general
   format for a Soozie workflow is as follows:

    ```scala
    val first = MapReduceJob("first") dependsOn Start
    val second = MapReduceJob("second") dependsOn first
    val third = MapReduceJob("third") dependsOn second
    val fourth = MapReduceJob("fourth") dependsOn third
    val end = End dependsOn fourth
    ```

    This is the code that a workflow developer will actually be writing.
    The intent is to make it easy to read and to see the dependency path
    between jobs. Soozie will automatically figure out forks/joins, etc.
    Specific samples of this can be found in [samples.scala](/src/test/scala/com/github/izhangzhihao/soozie/examples/samples.scala)

2. Logic / conversion from DSL spec to workflow graph. This is the real
   workhorse of Soozie, as it converts the programmer-specified workflow
   into an intermediate graph of workflow nodes. Forks, joins, and
   decisions are figured out and made ready for the final step - conversion
   to XML. This step also includes verification on the result graph.

3. Conversion from intermediate graph to XML. This step makes use of
   [scalaxb](https://github.com/eed3si9n/scalaxb), an XML data-binding tool for scala. In this step each node in
   the intermediate graph is converted to an oozie-specific type. Finally,
   these nodes are converted to XML. This XML can be run in oozie by a
   scala command.

## DSL Class Structure:

Node: Contains work and a list of dependencies. A node is also a dependency.

Dependency: The parent of everything that a workflow job could depend on. (nodes or decisions)

Work: Any work that a workflow node performs. This could be End, Kill, Jobs, sub workflows, etc.

Job: Extends Work. This is the parent of the typical oozie jobs. (`HiveJob`, `MapReduceJob`,
	 `DistCpJob`, etc). Specific jobs are easy to extend and customize in order to fit the client's
	 needs.

In practice, each line of developer-specified Soozie code will return a Node, which
other Nodes will depend on (Decisions are a special case of this, which will simply
return a Dependency).

## Tutorial:

1. **Hello World!**
	The following workflow will create an Oozie workflow that will create a Hello World directory on HDFS.

	```scala
	def HelloWorldWorkflow = {
	    val mkHelloWorld = FsJob(
	        name = "make-hello-world",
	        tasks = List(
	            MkDir("${nameNode}/users/tmp/oozie-fun/hello-world_${wf:id()}")
	        )
	    ) dependsOn Start
	    val done = End dependsOn mkHelloWorld
	    Workflow("hello-world-wf", done)
	}
	```

    This is an example of the basic structure of every Soozie workflow. Likely every Soozie workflow you will write will begin with a job that depends on Start and ends with an "End" job. The dependency chain makes it possible to specify the entire workflow by only referencing the end node in the Workflow object creation.

2. **Getting more complicated - using forks & joins.**
	Soozie is smart enough to figure out fork and join structures by looking at the dependencies between workflow nodes. This reduces cognitive load as *for each node the developer only needs to think about what that node depends on.*
For example, consider the following workflow:

    ```scala
    def SimpleForkJoin = {
        val first = MapReduceJob("first") dependsOn Start
        val secondA = MapReduceJob("secondA") dependsOn first
        val secondB = MapReduceJob("secondB") dependsOn first
        val third = MapReduceJob("third") dependsOn (secondA, secondB)
        Workflow("simple-fork-join", third)
    }
    ```

	Soozie will automatically create a workflow structured as
	first -> fork -> secondA/secondB -> join -> third

	In addition, Soozie will alert you by throwing an error if the workflow you have specified is not allowed by Oozie (for example, two nodes from different threads being joined)

3. **Making things interesting - Decisions**
	Often a developer needs to specify variable paths in a workflow that are run conditionally. This is allowed in Soozie via `Decision` and `OneOf`.

	```scala
	def DecisionExample = {
		val first = MapReduceJob("first") dependsOn Start
		val decision = Decision(
		    "route1" -> Predicates.BooleanProperty("${doRoute1}")
		) dependsOn first
		val route1Start = MapReduceJob("r1Start") dependsOn (decision option "route1")
		val route1End = MapReduceJob("r1End") dependsOn route1Start
		val route2Start = MapReduceJob("r2Start") dependsOn (decision default)
		val route2End = MapReduceJob("r2End") dependsOn route2Start
		val last = MapReduceJob("last") dependsOn OneOf(route1End, route2End)
		val done = End dependsOn last
		Workflow("simple-decision", done)
	}
	```
	In this example, route1 (route1Start -> route1End) will be run if ${doRoute1} is evaluated to true. Otherwise, route2 (route2Start -> route2End) will be run. Again, this reduces developer cognitive overhead as the developer only needs to consider the dependencies for one node at a time. Clearly "last" depends on only *one* of route1 and route2.
	More complex decision structures are supported by Oozie as well. Verification will be automatically performed on the decisions to make sure that the developer has included all routes and default routes for each decision.

	1. **Pour some sugar on me - "doIf" and "Optional"**
		A common design pattern for decisions is to have an "optional" route that may or may not be inserted into the workflow. Soozie has provided some sugar for making this special case easy to think about and specify in code.

		```scala
		def SugarOption = {
		    val first = MapReduceJob("first") dependsOn Start
		    val option = MapReduceJob("option") dependsOn first doIf "doOption"
		    val second = MapReduceJob("second") dependsOn Optional(option)
		    val done = End dependsOn second
		    Workflow("sugar-option-decision", done)
		}
		```
	This syntax makes the semantics behind the optional node clearer than thinking about a separate decision. In this example, "option" is run if the "${doOption}" argument is evaluated to true (Soozie will figure out the brackets), and "second" must come after "first" and optionally "option".

4. **Scalability, Modularity - Sub Workflows**
	Soozie allows the developer to control the granularity of his project by using and defining sub-workflows at will. Using the previous `SugarOption` workflow as an example, we can easily create a new, more complex workflow without worrying about its specifics.

	```scala
	def SubWfExample = {
		val begin = MapReduceJob("begin") dependsOn Start
		val someWork = MapReduceJob("someWork") dependsOn begin
		val subwf = SugarOption dependsOn someWork
		val end = End dependsOn subwf
		Workflow("sub-wf-example", end)
	}
	```

5. **Misc. Cool Things**
	1. Specifying custom error-to paths on nodes.
		Soozie also makes it easy to provide customizations such as error-to paths on nodes (rather than erroring to "kill" or "fail")

		```scala
		def CustomErrorTo = {
		    val first = MapReduceJob("first") dependsOn Start
		    val errorPath = MapReduceJob("error") dependsOn (first error)
		    val second = MapReduceJob("second") dependsOn first
		    val end = End dependsOn OneOf(second, errorPath)
		}
		```

	This Soozie code will create a workflow containing three nodes. The "first" node will proceed to "second" in the optimal case, but will now proceed to "errorPath" in the case of an error. Additionally, the `OneOf` clause is used again here and has an intuitive meaning.

6. **Creating and Running Workflows in Soozie**
	Soozie makes running workflows easy. Simply create an object that extends `SoozieApp`, pass in your workflow and any parameters, and you've got a runnable main class that will generate and run an oozie workflow.

	```scala
	object RunExample extends SoozieApp(SubWfExample)
	```

	Soozie uses sbt-assembly in order to create an executable jar. Simply type

		> assembly

	from sbt and all of the necessary dependencies and code will be assembled into a single jar: "Soozie-assembly-0.x.jar"
	You can run your workflow by typing

	```
	java -cp Soozie-assembly-0.x.jar RunExample -arg1=foo -arg2=bar ...
	```

    More examples of creating runnable Soozie objects can be found in runSamples.scala

## Additional Help

More example Soozie code can be found in [samples.scala](/src/test/scala/com/github/izhangzhihao/soozie/examples/samples.scala). Available job definitions can be found in [jobs](/src/main/scala/com/github/izhangzhihao/soozie/jobs). Those jobs can be easily extended to fit your specific use cases.

## Acknowledgements

This project is heavily inspired by [scoozie](https://github.com/klout/scoozie) and [scoozie](https://github.com/QuantiumTechnology/scoozie)

repository: izhangzhihao/soozie

repo_name: izhangzhihao/soozie


text:
# Zendesk Ticket

A widget to send tickets to your companies zendesk. No OAuth, login once and send additional metadata in the ticket like the user agent, url and resolution.

Use this on for instance an acceptance server. A client (which you registered) can leave feedback without letting him go through the trouble of authenticating with oauth. The developer doesn't have to register a new app or app domain, he only have to set up zendesk and the token once.

Because you use the api token to authenticate the user, there is a minor security issue, where client of product A could leave feedback on product B if he knows the acceptance url of product B.

## Installation
Include the gem in your Gemfile

```ruby
gem 'zendesk_ticket', git: 'https://github.com/wirelab/zendesk_ticket.git'
```

Bundle the Gemfile

```ruby
bundle install
```

## Setup Zendesk

Create an account at zendesk and visit https://YOUR_COMPANY.zendesk.com/agent/#/admin/api

Check 'access with token' and write down the API-token

## Configuring

To generate an intializer and translation files, run

```ruby
rails generate zendesk_ticket:install
```

Add the following to your environment:
* ZENDESK_TOKEN The token you just got from zendesk
* ZENDESK_URL https://YOUR_COMPANY.zendesk.com/

Add to your application.css:
```css
/*
 * require zendesk_ticket
 */
```

To your application.js:
```javascript
//= require zendesk_ticket
```

In your html
```
  <%= render "zendesk_ticket/popup" %>
  <%= render "zendesk_ticket/button" %>
```

If you want to use a custom button to call the popup, you can leave out the zendesk_ticket/button partial and add data-zendesk-ticket-button to your button.

Then somewhere in a js file or view you call the app
```
$(document).ready(function(){
  app = new ZendeskTicketApp;
  app.initialize();
});
```

Optional:

Instead of posting the form your app, you can post it to a remote website (if you for instance want a centralized server for handeling the data). If this is the case, set the following ENV variable:

* ZENDESK_REMOTE_FORM_URL 'https://YOUR_COMPANY.com/'

Make sure you set up the CORS right. In a future milestone, I'll look into JSONP, making the CORS not needed.

## License
This project rocks and uses MIT-LICENSE.

license: MIT

branch_name: refs/heads/master

extension: md

gha_language: Ruby

license_type: permissive


text:
# 560. Subarray Sum Equals K

Difficulty: Medium

https://leetcode.com/problems/subarray-sum-equals-k/

Given an array of integers and an integer k, you need to find the total number of continuous subarrays whose sum equals to k.

**Example 1:**
```
Input:nums = [1,1,1], k = 2
Output: 2
```
**Note:**
1. The length of the array is in range [1, 20,000].
2. The range of numbers in the array is [-1000, 1000] and the range of the integer k is [-1e7, 1e7].

Companies: Google

Related Topics: Array, Map

Similar Questions: Two Sum, Continuous Subarray Sum

repository: jiadaizhao/LeetCode

path: /0501-0600/0560-Subarray Sum Equals K/README.md

gha_license_id: MIT

repo_name: jiadaizhao/LeetCode


text:
# NestTweaks

Homemade tweaks through Google Assistant (as Nest API for Thermostat E is not available in EU)

## Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. You can also deploy the examples on any compatible device (such as rpi) and program its execution using crontabs.

### Prerequisites

* [Python](http://www.dropwizard.io/1.0.2/docs/) (>= 3.4 recommended)
* An [Actions Console Project](https://maven.apache.org/)
* A [Google Account](https://rometools.github.io/rome/)

### Setup

Install Python 3 (Ubuntu/Debian GNU/Linux)

```
sudo apt-get update
sudo apt-get install python3 python3-venv
```

Or [MacOSX, Windows, Others](https://www.python.org/downloads/).

Create a new virtual environment:

```
python3 -m venv env
env/bin/python -m pip install --upgrade pip setuptools wheel
source env/bin/activate
```

Install the required dependencies:

```
sudo apt-get install portaudio19-dev libffi-dev libssl-dev
pip install --upgrade -r requirements.txt
```

## Authorization

* Follow the steps to configure the Actions Console project and the Google account.
* Follow the steps to [register a new device model and download the client secrets file](https://developers.google.com/assistant/sdk/guides/service/python/embed/register-device).
* Generate device credentials using google-oauthlib-tool:

```
pip install --upgrade google-auth-oauthlib[tool] google-oauthlib-tool --client-secrets path/to/client_secret_<client-id>.json --scope https://www.googleapis.com/auth/assistant-sdk-prototype --save --headless
```

## Run the samples

You will find two different samples:

* Nest Thermostat History Data Logger (thermostat-monitor.py): get all the data from your Thermostat (temperature, humidity and working mode), the current outside weather and push the data to your Google SpreadSheet.

```
python /path/to/thermostat-monitor.py --device-model-id <device-model-id> --device-id <device-id> --display
```

* Pre-warm your home after work (thermostat-eco.py): if you enable the ECO mode (either manually or automatically when you are away), it won't change to HEAT mode until you are back home (and probably, your home will be terribly cold). With this script, the mode will be changed to HEAT if the thermostat is currently in ECO mode.

```
python /path/to/thermostat-eco.py --device-model-id <device-model-id> --device-id <device-id> --display
```

In order to make thermostat-monitor work, don't forget to replace both the OWM (OpenWeatherMap) API Key to get the outside weather data and the GS_ID (GoogleSpreadsheet ID) if you want to push the data to your Google SpreadSheet as explained [here](https://www.instructables.com/id/Nest-Thermostat-Data-Logger/). You can find my own google script within gs folder.

## Deployment

Add additional notes about how to deploy this on a live system

## Built With

* [Google Assistant SDK for devices](https://github.com/googlesamples/assistant-sdk-python) - Used gRPC API
* [pyowm](https://github.com/csparpa/pyowm) - A Python wrapper around the OpenWeatherMap web API to get current weather
* [lxml](https://lxml.de/) - Used to parse Google Assistant HTML replies

## Contributing

Please, feel free to submit any pull request. I've created this project to meet my needs, due the lack of any Nest official API on Thermostat E (atleast, here in Europe) and I'm sure we can build something better!

## Authors

* **Roberto Bermejo** - *Initial work* - [Android in Spain](http://main.androidinspain.com/)

See also the list of [contributors](https://github.com/Batlin/NestTweaks/contributors) who participated in this project.

## License

This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details

repository: Batlin/NestTweaks

repo_name: Batlin/NestTweaks


text:
# Chicago COVID Vaccine Information

A Python 3 powered [Pelican](https://docs.getpelican.com/en/3.6.3/quickstart.html) static site built via GitHub Action and hosted on GitHub Pages info.

## Have information to add?

Great! We'd love your contribution! You can modify the markdown article [here](https://github.com/lorenanicole/chicago-covid-vaccine-311/blob/main/content/content/pages/vaccine-info.md) and help us keep the information up to date!

If you want to confirm the build locally:

1. Clone the code
2. `pip install -r requirements.txt`
3. Modify the [markdown article](https://github.com/lorenanicole/chicago-covid-vaccine-311/blob/main/content/content/pages/vaccine-info.md)
4. From project root run `pelican content`
5. `cd` to `output` and `python -m pelican.server 4000`
6. Open in browser at `http://localhost:4000`

## Contributors

- [Gabriel Zurita](https://github.com/gabezurita) who penned the article originally too!
- [Lorena Mesa](https://github.com/lorenanicole)

branch_name: refs/heads/main

gha_language: HTML


text:
---
layout: page
title: About
---

<!-- <p class="message"> -->
Welcome!

This is the story of my journey into the world of AI.

I am data scientist with a background in chemistry research. As a chemist, I employed both experimental and computational techniques to solve protein structures and infer their functions. As an experimentalist, I studied protein structures using solid-state nuclear magnetic resonance spectroscopy (a branch of physics & chemistry that deals with deciphering select electromagnetic signals). As a computational chemist, I built molecular models of proteins and simulated their behaviors using supercomputers (high-performance computing centers). These data-driven research eventually led me to the world of data science (DS) and machine learning (ML).

After my post-doctoral work, I decided to hone my skills in coding & data analysis by attending a (fulltime) 12-week DS bootcamp in Seattle, [METIS](https://www.thisismetis.com/). At the bootcamp, I completed multiple end-to-end data science projects, focusing on business problems that intersect various industries, e.g., healthcare, marketing, sports analytics.

I love building and productionizing machine learning and AI. This interest led me to taking up a role as a Conversational Data Specialist with the Semantic Machines team at Microsoft,
 [![Microsoft's newest Coversational AI](http://img.youtube.com/vi/G_v5B_gYceM/0.jpg)](http://www.youtube.com/watch?v=G_v5B_gYceM "Microsoft's Conversational AI")

_Excited and looking forward to whatever comes next!_

<!-- </p> -->

Let's keep in touch:
- Connect with me on [LinkedIn](https://www.linkedin.com/in/jhonsen/) or [Twitter](https://twitter.com/Jhonsen_dj)
- Checkout my projects in [GitHub](https://github.com/jhonsen).

Thanks for visiting!


text:
# EasierCrafting

This makes crafting easier by displaying a list of items you can craft using your current inventory. Clicking one of these items will craft this item once, while Shift-Clicking an item will fill the crafting grid once and shift-click the output field to craft as many items as possible.

Also, if you're not sure about a specific recipe, you can enter the (partial) name of the output item in a text box above the recipe list; getting a list of all craftable items that match that name, no matter if you have the ingredients. Moving your mouse to one of these items will show you the recipe.

This mod will not fill the crafting grid more than once per click, so you can't craft your whole inventory at once. This is for the benefit of servers that allow mods, but don't want mods to be able to do "more" than vanilla minecraft - if you want to convert your whole inventory of sugar cane to paper, you'll have to keep shift-clicking the paper icon, converting one crafting grid (3 stacks) at a time.

Version 1.6.7 has a new feature that requests an inventory resync after each
craft. It seems like this is a problem when you're playing on servers that
are not Spigot/Paper servers. There's a new config option that you can turn
on. It's not turned on by default as what it's doing is a bit hacky, and I
don't know if there are server plugins (like NoCheatPlus) that don't like it
and kick you, or get you banned. So please only turn it on if you have problems.

To make sure the mod doesn't slow down your minecraft, it has been optimized using [![JProfiler Logo](https://www.ej-technologies.com/images/product_banners/jprofiler_small.png "Logo")](https://www.ej-technologies.com/products/jprofiler/overview.html).

repository: gbl/EasierCrafting

gha_language: Java

repo_name: gbl/EasierCrafting


text:
# Print records app

This example app shows artists and their artworks, grouped by collection.

The code shows how to create a printable record layout. It features:

-   Many records shown on the same page.

-   Records printed alongside their linked records.

-   Content areas that resize dynamically to the amount of content in a field.

-   A button to print the layout.

-   The use of `CellRenderer` to display rich base content with very little code.

-   Styling that changes, based on the base content.

## How to run this app

1. Create a new base using the
   [Art Gallery Management template](https://airtable.com/templates/creative/expAZgezgpfCF8wVH/art-gallery-management).

2. Create a new app in your new base (see
   [Create a new app](https://airtable.com/developers/blocks/guides/hello-world-tutorial#create-a-new-app)),
   selecting "Print records" as your template.

3. From the root of your new app, run `block run`.

## See the app running

![Seeing a layout of artists and their artworks grouped by collection, printing the layout](media/block.gif)

repository: jacksonrodri/airtable-lob

repo_name: jacksonrodri/airtable-lob


text:
---
hide:
  - navigation
---

# Changelog

## **Version 0.15.0**
*Release date: 29 May, 2023*

Highlights:

* [**Multimodal**](https://maartengr.github.io/BERTopic/getting_started/multimodal/multimodal.html) Topic Modeling
    * Train your topic modeling on text, images, or images and text!
    * Use the `bertopic.backend.MultiModalBackend` to embed images, text, both or even caption images!
* [**Multi-Aspect**](https://maartengr.github.io/BERTopic/getting_started/multiaspect/multiaspect.html) Topic Modeling
    * Create multiple topic representations simultaneously
* Improved [**Serialization**](https://maartengr.github.io/BERTopic/getting_started/serialization/serialization.html) options
    * Push your model to the HuggingFace Hub with `.push_to_hf_hub`
    * Safer, smaller and more flexible serialization options with `safetensors`
    * Thanks to a great collaboration with HuggingFace and the authors of [BERTransfer](https://github.com/opinionscience/BERTransfer)!
* Added new embedding models
    * OpenAI: `bertopic.backend.OpenAIBackend`
    * Cohere: `bertopic.backend.CohereBackend`
* Added example of [summarizing topics](https://maartengr.github.io/BERTopic/getting_started/representation/representation.html#summarization) with OpenAI's GPT-models
* Added `nr_docs` and `diversity` parameters to OpenAI and Cohere representation models
* Use `custom_labels="Aspect1"` to use the aspect labels for visualizations instead
* Added cuML support for probability calculation in `.transform`
* Updated **topic embeddings**
    * Centroids by default and c-TF-IDF weighted embeddings for `partial_fit` and `.update_topics`
* Added `exponential_backoff` parameter to `OpenAI` model

Fixes:

* Fixed custom prompt not working in `TextGeneration`
* Fixed [#1142](https://github.com/MaartenGr/BERTopic/pull/1142)
* Add additional logic to handle cupy arrays by [@metasyn](https://github.com/metasyn) in [#1179](https://github.com/MaartenGr/BERTopic/pull/1179)
* Fix hierarchy viz and handle any form of distance matrix by [@elashrry](https://github.com/elashrry) in [#1173](https://github.com/MaartenGr/BERTopic/pull/1173)
* Updated languages list by [@sam9111](https://github.com/sam9111) in [#1099](https://github.com/MaartenGr/BERTopic/pull/1099)
* Added level_scale argument to visualize_hierarchical_documents by [@zilch42](https://github.com/zilch42) in [#1106](https://github.com/MaartenGr/BERTopic/pull/1106)
* Fix inconsistent naming by [@rolanderdei](https://github.com/rolanderdei) in [#1073](https://github.com/MaartenGr/BERTopic/pull/1073)

Multimodal Topic Modeling

With v0.15, we can now perform multimodal topic modeling in BERTopic! The most basic example of multimodal topic modeling in BERTopic is when you have images that accompany your documents. This means that it is expected that each document has an image and vice versa. Instagram pictures, for example, almost always have some descriptions to them.

  ![Image title](getting_started/multimodal/images_and_text.svg)

In this example, we are going to use images from `flickr` that each have a caption accociated to it:

```python
# NOTE: This requires the `datasets` package which you can
# install with `pip install datasets`
from datasets import load_dataset

ds = load_dataset("maderix/flickr_bw_rgb")
images = ds["train"]["image"]
docs = ds["train"]["caption"]
```

The `docs` variable contains the captions for each image in `images`. We can now use these variables to run our multimodal example:

```python
from bertopic import BERTopic
from bertopic.representation import VisualRepresentation

# Additional ways of representing a topic
visual_model = VisualRepresentation()

# Make sure to add the `visual_model` to a dictionary
representation_model = {
   "Visual_Aspect":  visual_model,
}
topic_model = BERTopic(representation_model=representation_model, verbose=True)
```

We can now access our image representations for each topic with `topic_model.topic_aspects_["Visual_Aspect"]`.
If you want an overview of the topic images together with their textual representations in jupyter, you can run the following:

```python
import base64
from io import BytesIO
from IPython.display import HTML

def image_base64(im):
    if isinstance(im, str):
        im = get_thumbnail(im)
    with BytesIO() as buffer:
        im.save(buffer, 'jpeg')
        return base64.b64encode(buffer.getvalue()).decode()

def image_formatter(im):
    return f'<img src="data:image/jpeg;base64,{image_base64(im)}">'

# Extract dataframe
df = topic_model.get_topic_info().drop("Representative_Docs", 1).drop("Name", 1)

# Visualize the images
HTML(df.to_html(formatters={'Visual_Aspect': image_formatter}, escape=False))
```

![images_and_text](https://github.com/MaartenGr/BERTopic/assets/25746895/3a741e2b-5810-4865-9664-0c6bb24ca3f9)

Multi-aspect Topic Modeling

In this new release, we introduce `multi-aspect topic modeling`! During the `.fit` or `.fit_transform` stages, you can now get multiple representations of a single topic. In practice, it works by generating and storing all kinds of different topic representations (see image below).

  ![Image title](getting_started/multiaspect/multiaspect.svg)

The approach is rather straightforward. We might want to represent our topics using a `PartOfSpeech` representation model but we might also want to try out `KeyBERTInspired` and compare those representation models. We can do this as follows:

```python
from bertopic.representation import KeyBERTInspired
from bertopic.representation import PartOfSpeech
from bertopic.representation import MaximalMarginalRelevance
from sklearn.datasets import fetch_20newsgroups

# Documents to train on
docs = fetch_20newsgroups(subset='all',  remove=('headers', 'footers', 'quotes'))['data']

# The main representation of a topic
main_representation = KeyBERTInspired()

# Additional ways of representing a topic
aspect_model1 = PartOfSpeech("en_core_web_sm")
aspect_model2 = [KeyBERTInspired(top_n_words=30), MaximalMarginalRelevance(diversity=.5)]

# Add all models together to be run in a single `fit`
representation_model = {
   "Main": main_representation,
   "Aspect1":  aspect_model1,
   "Aspect2":  aspect_model2
}
topic_model = BERTopic(representation_model=representation_model).fit(docs)
```

As show above, to perform multi-aspect topic modeling, we make sure that `representation_model` is a dictionary where each representation model pipeline is defined.
The main pipeline, that is used in most visualization options, is defined with the `"Main"` key. All other aspects can be defined however you want. In the example above, the two additional aspects that we are interested in are defined as `"Aspect1"` and `"Aspect2"`.

After we have fitted our model, we can access all representations with `topic_model.get_topic_info()`:

As you can see, there are a number of different representations for our topics that we can inspect. All aspects are found in `topic_model.topic_aspects_`.

Serialization

Saving, loading, and sharing a BERTopic model can be done in several ways. With this new release, it is now  advised to go with `.safetensors` as that allows for a small, safe, and fast method for saving your BERTopic model. However, other formats, such as `.pickle` and pytorch `.bin` are also possible.

The methods are used as follows:

```python
topic_model = BERTopic().fit(my_docs)

# Method 1 - safetensors
embedding_model = "sentence-transformers/all-MiniLM-L6-v2"
topic_model.save("path/to/my/model_dir", serialization="safetensors", save_ctfidf=True, save_embedding_model=embedding_model)

# Method 2 - pytorch
embedding_model = "sentence-transformers/all-MiniLM-L6-v2"
topic_model.save("path/to/my/model_dir", serialization="pytorch", save_ctfidf=True, save_embedding_model=embedding_model)

# Method 3 - pickle
topic_model.save("my_model", serialization="pickle")
```

Saving the topic modeling with `.safetensors` or `pytorch` has a number of advantages:

* `.safetensors` is a relatively **safe format**
* The resulting model can be **very small** (often < 20MB>) since no sub-models need to be saved
* Although version control is important, there is a bit more **flexibility** with respect to specific versions of packages
* More easily used in **production**
* **Share** models with the HuggingFace Hub

The above image, a model trained on 100,000 documents, demonstrates the differences in sizes comparing `safetensors`, `pytorch`, and `pickle`. The difference in sizes can mostly be explained due to the efficient saving procedure and that the clustering and dimensionality reductions are not saved in safetensors/pytorch since inference can be done based on the topic embeddings.

HuggingFace Hub

When you have created a BERTopic model, you can easily share it with other through the HuggingFace Hub. First, you need to log in to your HuggingFace account:

```python
from huggingface_hub import login
login()
```

When you have logged in to your HuggingFace account, you can save and upload the model as follows:

```python
from bertopic import BERTopic

# Train model
topic_model = BERTopic().fit(my_docs)

# Push to HuggingFace Hub
topic_model.push_to_hf_hub(
    repo_id="MaartenGr/BERTopic_ArXiv",
    save_ctfidf=True
)

# Load from HuggingFace
loaded_model = BERTopic.load("MaartenGr/BERTopic_ArXiv")
```

## **Version 0.14.1**
*Release date: 2 March, 2023*

Highlights:

* Use [**ChatGPT**](https://maartengr.github.io/BERTopic/getting_started/representation/representation.html#chatgpt) to create topic representations!:
* Added `delay_in_seconds` parameter to OpenAI and Cohere representation models for throttling the API
    * Setting this between 5 and 10 allows for trial users now to use more easily without hitting RateLimitErrors
* Fixed missing `title` param to visualization methods
* Fixed probabilities not correctly aligning ([#1024](https://github.com/MaartenGr/BERTopic/issues/1024))
* Fix typo in textgenerator  [@dkopljar27](https://github.com/dkopljar27) in [#1002](https://github.com/MaartenGr/BERTopic/pull/1002)

ChatGPT

Within OpenAI's API, the ChatGPT models use a different API structure compared to the GPT-3 models.
In order to use ChatGPT with BERTopic, we need to define the model and make sure to set `chat=True`:

```python
import openai
from bertopic import BERTopic
from bertopic.representation import OpenAI

# Create your representation model
openai.api_key = MY_API_KEY
representation_model = OpenAI(model="gpt-3.5-turbo", delay_in_seconds=10, chat=True)

# Use the representation model in BERTopic on top of the default pipeline
topic_model = BERTopic(representation_model=representation_model)
```

Prompting with ChatGPT is very satisfying and can be customized in BERTopic by using certain tags.
There are currently two tags, namely `"[KEYWORDS]"` and `"[DOCUMENTS]"`.
These tags indicate where in the prompt they are to be replaced with a topics keywords and top 4 most representative documents respectively.
For example, if we have the following prompt:

```python
prompt = """
I have topic that contains the following documents: \n[DOCUMENTS]
The topic is described by the following keywords: [KEYWORDS]

Based on the information above, extract a short topic label in the following format:
topic: <topic label>
"""
```

then that will be rendered as follows and passed to OpenAI's API:

```python
"""
I have a topic that contains the following documents:
- Our videos are also made possible by your support on patreon.co.
- If you want to help us make more videos, you can do so on patreon.com or get one of our posters from our shop.
- If you want to help us make more videos, you can do so there.
- And if you want to support us in our endeavor to survive in the world of online video, and make more videos, you can do so on patreon.com.

The topic is described by the following keywords: videos video you our support want this us channel patreon make on we if facebook to patreoncom can for and more watch

Based on the information above, extract a short topic label in the following format:
topic: <topic label>
"""
```

!!! note
    Whenever you create a custom prompt, it is important to add
    ```
    Based on the information above, extract a short topic label in the following format:
    topic: <topic label>
    ```
    at the end of your prompt as BERTopic extracts everything that comes after `topic: `. Having
    said that, if `topic: ` is not in the output, then it will simply extract the entire response, so
    feel free to experiment with the prompts.

## **Version 0.14.0**
*Release date: 14 February, 2023*

Highlights:

* Fine-tune [topic representations](https://maartengr.github.io/BERTopic/getting_started/representation/representation.html) with `bertopic.representation`
    * Diverse range of models, including KeyBERT, MMR, POS, Transformers, OpenAI, and more!'
    * Create your own prompts for text generation models, like GPT3:
        * Use `"[KEYWORDS]"` and `"[DOCUMENTS]"` in the prompt to decide where the keywords and set of representative documents need to be inserted.
    * Chain models to perform fine-grained fine-tuning
    * Create and customize your represention model
* Improved the topic reduction technique when using `nr_topics=int`
* Added `title` parameters for all graphs ([#800](https://github.com/MaartenGr/BERTopic/issues/800))

Fixes:

* Improve documentation ([#837](https://github.com/MaartenGr/BERTopic/issues/837), [#769](https://github.com/MaartenGr/BERTopic/issues/769), [#954](https://github.com/MaartenGr/BERTopic/issues/954), [#912](https://github.com/MaartenGr/BERTopic/issues/912), [#911](https://github.com/MaartenGr/BERTopic/issues/911))
* Bump pyyaml ([#903](https://github.com/MaartenGr/BERTopic/issues/903))
* Fix large number of representative docs ([#965](https://github.com/MaartenGr/BERTopic/issues/965))
* Prevent stochastisch behavior in `.visualize_topics` ([#952](https://github.com/MaartenGr/BERTopic/issues/952))
* Add custom labels parameter to `.visualize_topics` ([#976](https://github.com/MaartenGr/BERTopic/issues/976))
* Fix cuML HDBSCAN type checks by [@FelSiq](https://github.com/FelSiq) in [#981](https://github.com/MaartenGr/BERTopic/pull/981)

API Changes:

* The `diversity` parameter was removed in favor of `bertopic.representation.MaximalMarginalRelevance`
* The `representation_model` parameter was added to `bertopic.BERTopic`

Representation Models

Fine-tune the c-TF-IDF representation with a variety of models. Whether that is through a KeyBERT-Inspired model or GPT-3, the choice is up to you!

KeyBERTInspired

The algorithm follows some principles of [KeyBERT](https://github.com/MaartenGr/KeyBERT) but does some optimization in order to speed up inference. Usage is straightforward:

![keybertinspired](https://user-images.githubusercontent.com/25746895/216336376-d2c4e5d6-6cf7-435c-904c-fc195aae7dcd.svg)

```python
from bertopic.representation import KeyBERTInspired
from bertopic import BERTopic

# Create your representation model
representation_model = KeyBERTInspired()

# Use the representation model in BERTopic on top of the default pipeline
topic_model = BERTopic(representation_model=representation_model)
```

![keybert](https://user-images.githubusercontent.com/25746895/218417161-bfd5980e-43c7-498a-904a-b6018ba58d45.svg)

PartOfSpeech

Our candidate topics, as extracted with c-TF-IDF, do not take into account a keyword's part of speech as extracting noun-phrases from all documents can be computationally quite expensive. Instead, we can leverage c-TF-IDF to perform part of speech on a subset of keywords and documents that best represent a topic.

![partofspeech](https://user-images.githubusercontent.com/25746895/216336534-48ff400e-72e1-4c50-9030-414576bac01e.svg)

```python
from bertopic.representation import PartOfSpeech
from bertopic import BERTopic

# Create your representation model
representation_model = PartOfSpeech("en_core_web_sm")

# Use the representation model in BERTopic on top of the default pipeline
topic_model = BERTopic(representation_model=representation_model)
```

![pos](https://user-images.githubusercontent.com/25746895/218417198-41c19b5c-251f-43c1-bfe2-0a480731565a.svg)

MaximalMarginalRelevance

When we calculate the weights of keywords, we typically do not consider whether we already have similar keywords in our topic. Words like "car" and "cars"
essentially represent the same information and often redundant. We can use `MaximalMarginalRelevance` to improve diversity of our candidate topics:

![mmr](https://user-images.githubusercontent.com/25746895/216336697-558f1409-8da3-4076-a21b-d87eec583ac7.svg)

```python
from bertopic.representation import MaximalMarginalRelevance
from bertopic import BERTopic

# Create your representation model
representation_model = MaximalMarginalRelevance(diversity=0.3)

# Use the representation model in BERTopic on top of the default pipeline
topic_model = BERTopic(representation_model=representation_model)
```

![mmr (1)](https://user-images.githubusercontent.com/25746895/218417234-88b145e2-7293-43c0-888c-36abe469a48a.svg)

Zero-Shot Classification

To perform zero-shot classification, we feed the model with the keywords as generated through c-TF-IDF and a set of candidate labels. If, for a certain topic, we find a similar enough label, then it is assigned. If not, then we keep the original c-TF-IDF keywords.

We use it in BERTopic as follows:

```python
from bertopic.representation import ZeroShotClassification
from bertopic import BERTopic

# Create your representation model
candidate_topics = ["space and nasa", "bicycles", "sports"]
representation_model = ZeroShotClassification(candidate_topics, model="facebook/bart-large-mnli")

# Use the representation model in BERTopic on top of the default pipeline
topic_model = BERTopic(representation_model=representation_model)
```

![zero](https://user-images.githubusercontent.com/25746895/218417276-dcef3519-acba-4792-8601-45dc7ed39488.svg)

Text Generation: 🤗 Transformers

Nearly every week, there are new and improved models released on the 🤗 [Model Hub](https://huggingface.co/models) that, with some creativity, allow for
further fine-tuning of our c-TF-IDF based topics. These models range from text generation to zero-classification. In BERTopic, wrappers around these
methods are created as a way to support whatever might be released in the future.

Using a GPT-like model from the huggingface hub is rather straightforward:

```python
from bertopic.representation import TextGeneration
from bertopic import BERTopic

# Create your representation model
representation_model = TextGeneration('gpt2')

# Use the representation model in BERTopic on top of the default pipeline
topic_model = BERTopic(representation_model=representation_model)
```

![hf](https://user-images.githubusercontent.com/25746895/218417310-2b0eabc7-296d-499d-888b-0ab48a65a2fb.svg)

Text Generation: Cohere

Instead of using a language model from 🤗 transformers, we can use external APIs instead that
do the work for you. Here, we can use [Cohere](https://docs.cohere.ai/) to extract our topic labels from the candidate documents and keywords.
To use this, you will need to install cohere first:

```bash
pip install cohere
```

Then, get yourself an API key and use Cohere's API as follows:

```python
import cohere
from bertopic.representation import Cohere
from bertopic import BERTopic

# Create your representation model
co = cohere.Client(my_api_key)
representation_model = Cohere(co)

# Use the representation model in BERTopic on top of the default pipeline
topic_model = BERTopic(representation_model=representation_model)
```

![cohere](https://user-images.githubusercontent.com/25746895/218417337-294cb52a-93c9-4fd5-b981-29b40e4f0c1e.svg)

Text Generation: OpenAI

Instead of using a language model from 🤗 transformers, we can use external APIs instead that
do the work for you. Here, we can use [OpenAI](https://openai.com/api/) to extract our topic labels from the candidate documents and keywords.
To use this, you will need to install openai first:

```
pip install openai
```

Then, get yourself an API key and use OpenAI's API as follows:

```python
import openai
from bertopic.representation import OpenAI
from bertopic import BERTopic

# Create your representation model
openai.api_key = MY_API_KEY
representation_model = OpenAI()

# Use the representation model in BERTopic on top of the default pipeline
topic_model = BERTopic(representation_model=representation_model)
```

![openai](https://user-images.githubusercontent.com/25746895/218417357-cf8c0fab-4450-43d3-b4fd-219ed276d870.svg)

Text Generation: LangChain

[Langchain](https://github.com/hwchase17/langchain) is a package that helps users with chaining large language models.
In BERTopic, we can leverage this package in order to more efficiently combine external knowledge. Here, this
external knowledge are the most representative documents in each topic.

To use langchain, you will need to install the langchain package first. Additionally, you will need an underlying LLM to support langchain,
like openai:

```bash
pip install langchain, openai
```

Then, you can create your chain as follows:

```python
from langchain.chains.question_answering import load_qa_chain
from langchain.llms import OpenAI
chain = load_qa_chain(OpenAI(temperature=0, openai_api_key=MY_API_KEY), chain_type="stuff")
```

Finally, you can pass the chain to BERTopic as follows:

```python
from bertopic.representation import LangChain

# Create your representation model
representation_model = LangChain(chain)

# Use the representation model in BERTopic on top of the default pipeline
topic_model = BERTopic(representation_model=representation_model)
```

## **Version 0.13.0**
*Release date: 4 January, 2023*

Highlights:

* Calculate [topic distributions](https://maartengr.github.io/BERTopic/getting_started/distribution/distribution.html) with `.approximate_distribution` regardless of the cluster model used
    * Generates topic distributions on a document- and token-levels
    * Can be used for any document regardless of its size!
* [Fully supervised BERTopic](https://maartengr.github.io/BERTopic/getting_started/supervised/supervised.html)
    * You can now use a classification model for the clustering step instead to create a fully supervised topic model
* [Manual topic modeling](https://maartengr.github.io/BERTopic/getting_started/manual/manual.html)
    * Generate topic representations from labels directly
    * Allows for skipping the embedding and clustering steps in order to go directly to the topic representation step
* [Reduce outliers](https://maartengr.github.io/BERTopic/getting_started/outlier_reduction/outlier_reduction.html) with 4 different strategies using `.reduce_outliers`
* Install BERTopic without `SentenceTransformers` for a [lightweight package](https://maartengr.github.io/BERTopic/getting_started/tips_and_tricks/tips_and_tricks.html#lightweight-installation):
    * `pip install --no-deps bertopic`
    * `pip install --upgrade numpy hdbscan umap-learn pandas scikit-learn tqdm plotly pyyaml`
* Get meta data of trained documents such as topics and probabilities using `.get_document_info(docs)`
* Added more support for cuML's HDBSCAN
    * Calculate and predict probabilities during `fit_transform`  and `transform` respectively
    * This should give a major speed-up when setting `calculate_probabilities=True`
* More images to the documentation and a lot of changes/updates/clarifications
* Get representative documents for non-HDBSCAN models by comparing document and topic c-TF-IDF representations
* Sklearn Pipeline [Embedder](https://maartengr.github.io/BERTopic/getting_started/embeddings/embeddings.html#scikit-learn-embeddings) by [@koaning](https://github.com/koaning) in [#791](https://github.com/MaartenGr/BERTopic/pull/791)

Fixes:

* Improve `.partial_fit` documentation ([#837](https://github.com/MaartenGr/BERTopic/issues/837))
* Fixed scipy linkage usage ([#807](https://github.com/MaartenGr/BERTopic/issues/807))
* Fixed shifted heatmap ([#782](https://github.com/MaartenGr/BERTopic/issues/782))
* Fixed SpaCy backend ([#744](https://github.com/MaartenGr/BERTopic/issues/744))
* Fixed representative docs with small clusters (<3) ([#703](https://github.com/MaartenGr/BERTopic/issues/703))
* Typo fixed by [@timpal0l](https://github.com/timpal0l) in [#734](https://github.com/MaartenGr/BERTopic/pull/734)
* Typo fixed by [@srulikbd](https://github.com/timpal0l) in [#842](https://github.com/MaartenGr/BERTopic/pull/842)
* Correcting iframe urls by [@Mustapha-AJEGHRIR](https://github.com/Mustapha-AJEGHRIR) in [#798](https://github.com/MaartenGr/BERTopic/pull/798)
* Refactor embedding methods by [@zachschillaci27](https://github.com/zachschillaci27) in [#855](https://github.com/MaartenGr/BERTopic/pull/855)
* Added diversity parameter to update_topics() function by [@anubhabdaserrr](https://github.com/anubhabdaserrr) in [#887](https://github.com/MaartenGr/BERTopic/pull/887)

Documentation

Personally, I believe that documentation can be seen as a feature and is an often underestimated aspect of open-source. So I went a bit overboard😅... and created an animation about the three pillars of BERTopic using Manim. There are many other visualizations added, one of each variation of BERTopic, and many smaller changes.

Topic Distributions

The difficulty with a cluster-based topic modeling technique is that it does not directly consider that documents may contain multiple topics. With the new release, we can now model the distributions of topics! We even consider that a single word might be related to multiple topics. If a document is a mixture of topics, what is preventing a single word to be the same?

To do so, we approximate the distribution of topics in a document by calculating and summing the similarities of tokensets (achieved by applying a sliding window) with the topics:

```python
# After fitting your model run the following for either your trained documents or even unseen documents
topic_distr, _ = topic_model.approximate_distribution(docs)
```

To calculate and visualize the topic distributions in a document on a token-level, we can run the following:

```python
# We need to calculate the topic distributions on a token level
topic_distr, topic_token_distr = topic_model.approximate_distribution(docs, calculate_tokens=True)

# Create a visualization using a styled dataframe if Jinja2 is installed
df = topic_model.visualize_approximate_distribution(docs[0], topic_token_distr[0]); df
```

Supervised Topic Modeling

BERTopic now supports fully-supervised classification! Instead of using a clustering algorithm, like HDBSCAN, we can replace it with a classifier, like Logistic Regression:

```python
from bertopic import BERTopic
from bertopic.dimensionality import BaseDimensionalityReduction
from sklearn.datasets import fetch_20newsgroups
from sklearn.linear_model import LogisticRegression

# Get labeled data
data= fetch_20newsgroups(subset='all',  remove=('headers', 'footers', 'quotes'))
docs = data['data']
y = data['target']

# Allows us to skip over the dimensionality reduction step
empty_dimensionality_model = BaseDimensionalityReduction()

# Create a classifier to be used instead of the cluster model
clf= LogisticRegression()

# Create a fully supervised BERTopic instance
topic_model= BERTopic(
        umap_model=empty_dimensionality_model,
        hdbscan_model=clf
)
topics, probs = topic_model.fit_transform(docs, y=y)
```

Manual Topic Modeling

When you already have a bunch of labels and simply want to extract topic representations from them, you might not need to actually learn how those can predicted. We can bypass the `embeddings -> dimensionality reduction -> clustering` steps and go straight to the c-TF-IDF representation of our labels:

```python
from bertopic import BERTopic
from bertopic.backend import BaseEmbedder
from bertopic.cluster import BaseCluster
from bertopic.dimensionality import BaseDimensionalityReduction

# Prepare our empty sub-models and reduce frequent words while we are at it.
empty_embedding_model = BaseEmbedder()
empty_dimensionality_model = BaseDimensionalityReduction()
empty_cluster_model = BaseCluster()

# Fit BERTopic without actually performing any clustering
topic_model= BERTopic(
        embedding_model=empty_embedding_model,
        umap_model=empty_dimensionality_model,
        hdbscan_model=empty_cluster_model,
)
topics, probs = topic_model.fit_transform(docs, y=y)
```

Outlier Reduction

Outlier reduction is an frequently-discussed topic in BERTopic as its default cluster model, HDBSCAN, has a tendency to generate many outliers. This often helps in the topic representation steps, as we do not consider documents that are less relevant, but you might want to still assign those outliers to actual topics. In the modular philosophy of BERTopic, keeping training times in mind, it is now possible to perform outlier reduction **after** having trained your topic model. This allows for ease of iteration and prevents having to train BERTopic many times to find the parameters you are searching for. There are 4 different strategies that you can use, so make sure to check out the [documentation](https://maartengr.github.io/BERTopic/getting_started/outlier_reduction/outlier_reduction.html)!

Using it is rather straightforward:

```python
new_topics = topic_model.reduce_outliers(docs, topics)
```

Lightweight BERTopic

The default embedding model in BERTopic is one of the amazing sentence-transformers models, namely `"all-MiniLM-L6-v2"`. Although this model performs well out of the box, it typically needs a GPU to transform the documents into embeddings in a reasonable time. Moreover, the installation requires `pytorch` which often results in a rather large environment, memory-wise.

Fortunately, it is possible to install BERTopic without `sentence-transformers` and use it as a lightweight solution instead. The installation can be done as follows:

```bash
pip install --no-deps bertopic
pip install --upgrade numpy hdbscan umap-learn pandas scikit-learn tqdm plotly pyyaml
```

Then, we can use BERTopic without `sentence-transformers` as follows using a CPU-based embedding technique:

```python
from sklearn.pipeline import make_pipeline
from sklearn.decomposition import TruncatedSVD
from sklearn.feature_extraction.text import TfidfVectorizer

pipe = make_pipeline(
    TfidfVectorizer(),
    TruncatedSVD(100)
)

topic_model = BERTopic(embedding_model=pipe)
```

As a result, the entire package and resulting model can be run quickly on the CPU and no GPU is necessary!

Document Information

Get information about the documents on which the topic was trained including the documents themselves, their respective topics, the name of each topic, the top n words of each topic, whether it is a representative document, and the probability of the clustering if the cluster model supports it. There are also options to include other metadata, such as the topic distributions or the x and y coordinates of the reduced embeddings that you can learn more about here.

To get the document info, you will only need to pass the documents on which the topic model was trained:

```python
>>> topic_model.get_document_info(docs)

Document                               Topic	Name	                    Top_n_words                     Probability    ...
I am sure some bashers of Pens...	    0	    0_game_team_games_season	game - team - games...	        0.200010       ...
My brother is in the market for...      -1     -1_can_your_will_any	        can - your - will...	        0.420668       ...
Finally you said what you dream...	    -1     -1_can_your_will_any	        can - your - will...            0.807259       ...
Think! It is the SCSI card doing...	    49     49_windows_drive_dos_file	windows - drive - docs...	    0.071746       ...
1) I have an old Jasmine drive...	    49     49_windows_drive_dos_file	windows - drive - docs...	    0.038983       ...
```

## **Version 0.12.0**
*Release date: 5 September, 2022*

**Highlights**:

* Perform [online/incremental topic modeling](https://maartengr.github.io/BERTopic/getting_started/online/online.html) with `.partial_fit`
* Expose [c-TF-IDF model](https://maartengr.github.io/BERTopic/getting_started/ctfidf/ctfidf.html) for customization with `bertopic.vectorizers.ClassTfidfTransformer`
    * The parameters `bm25_weighting` and `reduce_frequent_words` were added to potentially improve representations:
* Expose attributes for easier access to internal data
* Major changes to the [Algorithm](https://maartengr.github.io/BERTopic/algorithm/algorithm.html) page of the documentation, which now contains three overviews of the algorithm:
    *  [Visualize Overview](https://maartengr.github.io/BERTopic/algorithm/algorithm.html#visual-overview)
    *  [Code Overview](https://maartengr.github.io/BERTopic/algorithm/algorithm.html#code-overview)
    *  [Detailed Overview](https://maartengr.github.io/BERTopic/algorithm/algorithm.html#detailed-overview)
* Added an [example](https://maartengr.github.io/BERTopic/getting_started/tips_and_tricks/tips_and_tricks.html#keybert-bertopic) of combining BERTopic with KeyBERT
* Added many tests with the intention of making development a bit more stable

**Fixes**:

* Fixed iteratively merging topics ([#632](https://github.com/MaartenGr/BERTopic/issues/632) and ([#648](https://github.com/MaartenGr/BERTopic/issues/648))
* Fixed 0th topic not showing up in visualizations ([#667](https://github.com/MaartenGr/BERTopic/issues/667))
* Fixed lowercasing not being optional ([#682](https://github.com/MaartenGr/BERTopic/issues/682))
* Fixed spelling ([#664](https://github.com/MaartenGr/BERTopic/issues/664) and ([#673](https://github.com/MaartenGr/BERTopic/issues/673))
* Fixed 0th topic not shown in `.get_topic_info` by [@oxymor0n](https://github.com/oxymor0n) in [#660](https://github.com/MaartenGr/BERTopic/pull/660)
* Fixed spelling by [@domenicrosati](https://github.com/domenicrosati) in [#674](https://github.com/MaartenGr/BERTopic/pull/674)
* Add custom labels and title options to barchart [@leloykun](https://github.com/leloykun) in [#694](https://github.com/MaartenGr/BERTopic/pull/694)

**Online/incremental topic modeling**:

Online topic modeling (sometimes called "incremental topic modeling") is the ability to learn incrementally from a mini-batch of instances. Essentially, it is a way to update your topic model with data on which it was not trained on before. In Scikit-Learn, this technique is often modeled through a `.partial_fit` function, which is also used in BERTopic.

At a minimum, the cluster model needs to support a `.partial_fit` function in order to use this feature. The default HDBSCAN model will not work as it does not support online updating.

```python
from sklearn.datasets import fetch_20newsgroups
from sklearn.cluster import MiniBatchKMeans
from sklearn.decomposition import IncrementalPCA
from bertopic.vectorizers import OnlineCountVectorizer
from bertopic import BERTopic

# Prepare documents
all_docs = fetch_20newsgroups(subset=subset,  remove=('headers', 'footers', 'quotes'))["data"]
doc_chunks = [all_docs[i:i+1000] for i in range(0, len(all_docs), 1000)]

# Prepare sub-models that support online learning
umap_model = IncrementalPCA(n_components=5)
cluster_model = MiniBatchKMeans(n_clusters=50, random_state=0)
vectorizer_model = OnlineCountVectorizer(stop_words="english", decay=.01)

topic_model = BERTopic(umap_model=umap_model,
                       hdbscan_model=cluster_model,
                       vectorizer_model=vectorizer_model)

# Incrementally fit the topic model by training on 1000 documents at a time
for docs in doc_chunks:
    topic_model.partial_fit(docs)
```

Only the topics for the most recent batch of documents are tracked. If you want to be using online topic modeling, not for a streaming setting but merely for low-memory use cases, then it is advised to also update the `.topics_` attribute as variations such as hierarchical topic modeling will not work afterward:

```python
# Incrementally fit the topic model by training on 1000 documents at a time and track the topics in each iteration
topics = []
for docs in doc_chunks:
    topic_model.partial_fit(docs)
    topics.extend(topic_model.topics_)

topic_model.topics_ = topics
```

**c-TF-IDF**:

Explicitly define, use, and adjust the `ClassTfidfTransformer` with new parameters, `bm25_weighting` and `reduce_frequent_words`, to potentially improve the topic representation:

```python
from bertopic import BERTopic
from bertopic.vectorizers import ClassTfidfTransformer

ctfidf_model = ClassTfidfTransformer(bm25_weighting=True)
topic_model = BERTopic(ctfidf_model=ctfidf_model)
```

**Attributes**:

After having fitted your BERTopic instance, you can use the following attributes to have quick access to certain information, such as the topic assignment for each document in `topic_model.topics_`.

| Attribute | Type | Description |
|--------------------|----|---------------------------------------------------------------------------------------------|
| topics_            | List[int]   | The topics that are generated for each document after training or updating the topic model. The most recent topics are tracked.  |
| probabilities_ | List[float] | The probability of the assigned topic per document. These are only calculated if a HDBSCAN model is used for the clustering step. When `calculate_probabilities=True`, then it is the probabilities of all topics per document. |
| topic_sizes_          | Mapping[int, int] | The size of each topic.                                                                   |
| topic_mapper_         | TopicMapper | A class for tracking topics and their mappings anytime they are merged, reduced, added, or removed.             |
| topic_representations_  | Mapping[int, Tuple[int, float]] | The top *n* terms per topic and their respective c-TF-IDF values.                            |
| c_tf_idf_             | csr_matrix | The topic-term matrix as calculated through c-TF-IDF. To access its respective words, run `.vectorizer_model.get_feature_names()` or `.vectorizer_model.get_feature_names_out()`                                      |
| topic_labels_         | Mapping[int, str] |  The default labels for each topic.                                                          |
| custom_labels_        | List[str] | Custom labels for each topic as generated through `.set_topic_labels`.                                                               |
| topic_embeddings_     | np.ndarray | The embeddings for each topic. It is calculated by taking the weighted average of word embeddings in a topic based on their c-TF-IDF values.                                                  |
| representative_docs_  | Mapping[int, str] | The representative documents for each topic if HDBSCAN is used.                                                |

## **Version 0.11.0**
*Release date: 11 July, 2022*

**Highlights**:

* Perform [hierarchical topic modeling](https://maartengr.github.io/BERTopic/getting_started/hierarchicaltopics/hierarchicaltopics.html) with `.hierarchical_topics`

```python
hierarchical_topics = topic_model.hierarchical_topics(docs, topics)
```

* Visualize [hierarchical topic representations](https://maartengr.github.io/BERTopic/getting_started/hierarchicaltopics/hierarchicaltopics.html#visualizations) with `.visualize_hierarchy`

```python
topic_model.visualize_hierarchy(hierarchical_topics=hierarchical_topics)
```

* Extract a [text-based hierarchical topic representation](https://maartengr.github.io/BERTopic/getting_started/hierarchicaltopics/hierarchicaltopics.html#visualizations) with `.get_topic_tree`

```python
tree = topic_model.get_topic_tree(hierarchical_topics)
```

* Visualize [2D documents](https://maartengr.github.io/BERTopic/getting_started/visualization/visualization.html#visualize-documents) with `.visualize_documents()`

```python
# Use input embeddings
topic_model.visualize_documents(docs, embeddings=embeddings)

# or use 2D reduced embeddings through a method of your own (e.g., PCA, t-SNE, UMAP, etc.)
reduced_embeddings = UMAP(n_neighbors=10, n_components=2, min_dist=0.0, metric='cosine').fit_transform(embeddings)
topic_model.visualize_documents(docs, reduced_embeddings=reduced_embeddings)
```

* Visualize [2D hierarchical documents](https://maartengr.github.io/BERTopic/getting_started/visualization/visualization.html#visualize-hierarchical-documents) with `.visualize_hierarchical_documents()`

```python
# Run the visualization with the original embeddings
topic_model.visualize_hierarchical_documents(docs, hierarchical_topics, embeddings=embeddings)

# Or, if you have reduced the original embeddings already which speed things up quite a bit:
reduced_embeddings = UMAP(n_neighbors=10, n_components=2, min_dist=0.0, metric='cosine').fit_transform(embeddings)
topic_model.visualize_hierarchical_documents(docs, hierarchical_topics, reduced_embeddings=reduced_embeddings)
```

* Create [custom labels](https://maartengr.github.io/BERTopic/getting_started/topicrepresentation/topicrepresentation.html#custom-labels) to the topics throughout most visualizations

```python
# Generate topic labels
topic_labels = topic_model.generate_topic_labels(nr_words=3, topic_prefix=False, word_length=10, separator=", ")

# Set them internally in BERTopic
topic_model.set_topic_labels(topics_labels)
```

* Manually [merge topics](https://maartengr.github.io/BERTopic/getting_started/hierarchicaltopics/hierarchicaltopics.html#merge-topics) with `.merge_topics()`

```python
# Merge topics 1, 2, and 3
topics_to_merge = [1, 2, 3]
topic_model.merge_topics(docs, topics, topics_to_merge)

# Merge topics 1 and 2, and separately merge topics 3 and 4
topics_to_merge = [[1, 2], [3, 4]]
topic_model.merge_topics(docs, topics, topics_to_merge)
```

* Added example for finding similar topics between two models in the [tips & tricks](https://maartengr.github.io/BERTopic/getting_started/tips_and_tricks/tips_and_tricks.html) page
* Add multi-modal example in the [tips & tricks](https://maartengr.github.io/BERTopic/getting_started/tips_and_tricks/tips_and_tricks.html) page
* Added native [Hugging Face transformers](https://maartengr.github.io/BERTopic/getting_started/embeddings/embeddings.html#hugging-face-transformers) support

**Fixes**:

* Fix support for k-Means in `.visualize_heatmap` ([#532](https://github.com/MaartenGr/BERTopic/issues/532))
* Fix missing topic 0 in `.visualize_topics` ([#533](https://github.com/MaartenGr/BERTopic/issues/533))
* Fix inconsistencies in `.get_topic_info` ([#572](https://github.com/MaartenGr/BERTopic/issues/572)) and ([#581](https://github.com/MaartenGr/BERTopic/issues/581))
* Add `optimal_ordering` parameter to `.visualize_hierarchy` by [@rafaelvalero](https://github.com/rafaelvalero) in [#390](https://github.com/MaartenGr/BERTopic/pull/390)
* Fix RuntimeError when used as sklearn estimator by [@simonfelding](https://github.com/simonfelding) in [#448](https://github.com/MaartenGr/BERTopic/pull/448)
* Fix typo in visualization documentation by [@dwhdai](https://github.com/dwhdai) in [#475](https://github.com/MaartenGr/BERTopic/pull/475)
* Fix typo in docstrings by [@xwwwwww](https://github.com/xwwwwww) in [#549](https://github.com/MaartenGr/BERTopic/pull/549)
* Support higher Flair versions

## **Version 0.10.0**
*Release date: 30 April, 2022*

**Highlights**:

* Use any dimensionality reduction technique instead of UMAP:

```python
from bertopic import BERTopic
from sklearn.decomposition import PCA

dim_model = PCA(n_components=5)
topic_model = BERTopic(umap_model=dim_model)
```

* Use any clustering technique instead of HDBSCAN:

```python
from bertopic import BERTopic
from sklearn.cluster import KMeans

cluster_model = KMeans(n_clusters=50)
topic_model = BERTopic(hdbscan_model=cluster_model)
```

**Documentation**:

* Add a CountVectorizer page with tips and tricks on how to create topic representations that fit your use case
* Added pages on how to use other dimensionality reduction and clustering algorithms
 * Additional instructions on how to reduce outliers in the FAQ:

```python
import numpy as np
probability_threshold = 0.01
new_topics = [np.argmax(prob) if max(prob) >= probability_threshold else -1 for prob in probs]
```

**Fixes**:

* Fixed `None` being returned for probabilities when transforming unseen documents
* Replaced all instances of `arg:` with `Arguments:` for consistency
* Before saving a fitted BERTopic instance, we remove the stopwords in the fitted CountVectorizer model as it can get quite large due to the number of words that end in stopwords if `min_df` is set to a value larger than 1
* Set `"hdbscan>=0.8.28"` to prevent numpy issues
  * Although this was already fixed by the new release of HDBSCAN, it is technically still possible to install 0.8.27 with BERTopic which leads to these numpy issues
* Update gensim dependency to `>=4.0.0` ([#371](https://github.com/MaartenGr/BERTopic/issues/371))
* Fix topic 0 not appearing in visualizations ([#472](https://github.com/MaartenGr/BERTopic/issues/472))
* Fix ([#506](https://github.com/MaartenGr/BERTopic/issues/506))
* Fix ([#429](https://github.com/MaartenGr/BERTopic/issues/429))
* Fix typo in DTM documentation by [@hp0404](https://github.com/hp0404) in [#386](https://github.com/MaartenGr/BERTopic/pull/386)

## **Version 0.9.4**
*Release date: 14 December, 2021*

A number of fixes, documentation updates, and small features:

* Expose diversity parameter
    * Use `BERTopic(diversity=0.1)` to change how diverse the words in a topic representation are (ranges from 0 to 1)
* Improve stability of topic reduction by only computing the cosine similarity within c-TF-IDF and not the topic embeddings
* Added property to c-TF-IDF that all IDF values should be positive ([#351](https://github.com/MaartenGr/BERTopic/issues/351))
* Improve stability of `.visualize_barchart()` and `.visualize_hierarchy()`
* Major [documentation](https://maartengr.github.io/BERTopic/) overhaul (mkdocs, tutorials, FAQ, images, etc. ) ([#330](https://github.com/MaartenGr/BERTopic/issues/330))
* Drop python 3.6 ([#333](https://github.com/MaartenGr/BERTopic/issues/333))
* Relax plotly dependency ([#88](https://github.com/MaartenGr/BERTopic/issues/88))
* Additional logging for `.transform` ([#356](https://github.com/MaartenGr/BERTopic/issues/356))

## **Version 0.9.3**
*Release date:  17 October, 2021*

* Fix [#282](https://github.com/MaartenGr/BERTopic/issues/282)
    * As it turns out the old implementation of topic mapping was still found in the `transform` function
* Fix [#285](https://github.com/MaartenGr/BERTopic/issues/285)
    * Fix getting all representative docs
* Fix [#288](https://github.com/MaartenGr/BERTopic/issues/288)
    * A recent issue with the package `pyyaml` that can be found in Google Colab

## **Version 0.9.2**
*Release date:  12 October, 2021*

A release focused on algorithmic optimization and fixing several issues:

**Highlights**:

* Update the non-multilingual paraphrase-* models to the all-* models due to improved [performance](https://www.sbert.net/docs/pretrained_models.html)
* Reduce necessary RAM in c-TF-IDF top 30 word [extraction](https://stackoverflow.com/questions/49207275/finding-the-top-n-values-in-a-row-of-a-scipy-sparse-matrix)

**Fixes**:

* Fix topic mapping
    * When reducing the number of topics, these need to be mapped to the correct input/output which had some issues in the previous version
    * A new class was created as a way to track these mappings regardless of how many times they were executed
    * In other words, you can iteratively reduce the number of topics after training the model without the need to continuously train the model
* Fix typo in embeddings page ([#200](https://github.com/MaartenGr/BERTopic/issues/200))
* Fix link in README ([#233](https://github.com/MaartenGr/BERTopic/issues/233))
* Fix documentation `.visualize_term_rank()` ([#253](https://github.com/MaartenGr/BERTopic/issues/253))
* Fix getting correct representative docs ([#258](https://github.com/MaartenGr/BERTopic/issues/258))
* Update [memory FAQ](https://maartengr.github.io/BERTopic/faq.html#i-am-facing-memory-issues-help) with [HDBSCAN pr](https://github.com/MaartenGr/BERTopic/issues/151)

## **Version 0.9.1**
*Release date:  1 September, 2021*

A release focused on fixing several issues:

**Fixes**:

* Fix TypeError when auto-reducing topics ([#210](https://github.com/MaartenGr/BERTopic/issues/210))
* Fix mapping representative docs when reducing topics ([#208](https://github.com/MaartenGr/BERTopic/issues/208))
* Fix visualization issues with probabilities ([#205](https://github.com/MaartenGr/BERTopic/issues/205))
* Fix missing `normalize_frequency` param in plots ([#213](https://github.com/MaartenGr/BERTopic/issues/208))

## **Version 0.9.0**
*Release date:  9 August, 2021*

**Highlights**:

* Implemented a [**Guided BERTopic**](https://maartengr.github.io/BERTopic/getting_started/guided/guided.html) -> Use seeds to steer the Topic Modeling
* Get the most representative documents per topic: `topic_model.get_representative_docs(topic=1)`
    * This allows users to see which documents are good representations of a topic and better understand the topics that were created
* Added `normalize_frequency` parameter to `visualize_topics_per_class` and `visualize_topics_over_time` in order to better compare the relative topic frequencies between topics
* Return flat probabilities as default, only calculate the probabilities of all topics per document if `calculate_probabilities` is True
* Added several FAQs

**Fixes**:

* Fix loading pre-trained BERTopic model
* Fix mapping of probabilities
* Fix [#190](https://github.com/MaartenGr/BERTopic/issues/190)

**Guided BERTopic**:

Guided BERTopic works in two ways:

First, we create embeddings for each seeded topics by joining them and passing them through the document embedder.
These embeddings will be compared with the existing document embeddings through cosine similarity and assigned a label.
If the document is most similar to a seeded topic, then it will get that topic's label.
If it is most similar to the average document embedding, it will get the -1 label.
These labels are then passed through UMAP to create a semi-supervised approach that should nudge the topic creation to the seeded topics.

Second, we take all words in `seed_topic_list` and assign them a multiplier larger than 1.
Those multipliers will be used to increase the IDF values of the words across all topics thereby increasing
the likelihood that a seeded topic word will appear in a topic. This does, however, also increase the chance of an
irrelevant topic having unrelated words. In practice, this should not be an issue since the IDF value is likely to
remain low regardless of the multiplier. The multiplier is now a fixed value but may change to something more elegant,
like taking the distribution of IDF values and its position into account when defining the multiplier.

```python
seed_topic_list = [["company", "billion", "quarter", "shrs", "earnings"],
                   ["acquisition", "procurement", "merge"],
                   ["exchange", "currency", "trading", "rate", "euro"],
                   ["grain", "wheat", "corn"],
                   ["coffee", "cocoa"],
                   ["natural", "gas", "oil", "fuel", "products", "petrol"]]

topic_model = BERTopic(seed_topic_list=seed_topic_list)
topics, probs = topic_model.fit_transform(docs)
```

## **Version 0.8.1**
*Release date:  8 June, 2021*

**Highlights**:

* Improved models:
    * For English documents the default is now: `"paraphrase-MiniLM-L6-v2"`
    * For Non-English or multi-lingual documents the default is now: `"paraphrase-multilingual-MiniLM-L12-v2"`
    * Both models show not only great performance but are much faster!
* Add interactive visualizations to the `plotting` API documentation

For better performance, please use the following models:

* English: `"paraphrase-mpnet-base-v2"`
* Non-English or multi-lingual: `"paraphrase-multilingual-mpnet-base-v2"`

**Fixes**:

* Improved unit testing for more stability
* Set transformers version for Flair

## **Version 0.8.0**
*Release date:  31 May, 2021*

**Highlights**:

* Additional visualizations:
    * Topic Hierarchy: `topic_model.visualize_hierarchy()`
    * Topic Similarity Heatmap: `topic_model.visualize_heatmap()`
    * Topic Representation Barchart: `topic_model.visualize_barchart()`
    * Term Score Decline: `topic_model.visualize_term_rank()`
* Created `bertopic.plotting` library to easily extend visualizations
* Improved automatic topic reduction by using HDBSCAN to detect similar topics
* Sort topic ids by their frequency. -1 is the outlier class and contains typically the most documents. After that 0 is the largest  topic, 1 the second largest, etc.

**Fixes**:

* Fix typo [#113](https://github.com/MaartenGr/BERTopic/pull/113), [#117](https://github.com/MaartenGr/BERTopic/pull/117)
* Fix [#121](https://github.com/MaartenGr/BERTopic/issues/121) by removing [these](https://github.com/MaartenGr/BERTopic/blob/5c6cf22776fafaaff728370781a5d33727d3dc8f/bertopic/_bertopic.py#L359-L360) two lines
* Fix mapping of topics after reduction (it now excludes 0) ([#103](https://github.com/MaartenGr/BERTopic/issues/103))

## **Version 0.7.0**
*Release date:  26 April, 2021*

The two main features are **(semi-)supervised topic modeling**
and several **backends** to use instead of Flair and SentenceTransformers!

**Highlights**:

* (semi-)supervised topic modeling by leveraging supervised options in UMAP
    * `model.fit(docs, y=target_classes)`
* Backends:
    * Added Spacy, Gensim, USE (TFHub)
    * Use a different backend for document embeddings and word embeddings
    * Create your own backends with `bertopic.backend.BaseEmbedder`
    * Click [here](https://maartengr.github.io/BERTopic/getting_started/embeddings/embeddings.html) for an overview of all new backends
* Calculate and visualize topics per class
    * Calculate: `topics_per_class = topic_model.topics_per_class(docs, topics, classes)`
    * Visualize: `topic_model.visualize_topics_per_class(topics_per_class)`
* Several tutorials were updated and added:

| Name  | Link  |
|---|---|
| Topic Modeling with BERTopic  | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1FieRA9fLdkQEGDIMYl0I3MCjSUKVF8C-?usp=sharing)  |
| (Custom) Embedding Models in BERTopic  | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/18arPPe50szvcCp_Y6xS56H2tY0m-RLqv?usp=sharing) |
| Advanced Customization in BERTopic  |  [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1ClTYut039t-LDtlcd-oQAdXWgcsSGTw9?usp=sharing) |
| (semi-)Supervised Topic Modeling with BERTopic  |  [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1bxizKzv5vfxJEB29sntU__ZC7PBSIPaQ?usp=sharing)  |
| Dynamic Topic Modeling with Trump's Tweets  | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1un8ooI-7ZNlRoK0maVkYhmNRl0XGK88f?usp=sharing)  |

**Fixes**:

* Fixed issues with Torch req
* Prevent saving term frequency matrix in CTFIDF class
* Fixed DTM not working when reducing topics ([#96](https://github.com/MaartenGr/BERTopic/issues/96))
* Moved visualization dependencies to base BERTopic
    * `pip install bertopic[visualization]` becomes `pip install bertopic`
* Allow precomputed embeddings in bertopic.find_topics() ([#79](https://github.com/MaartenGr/BERTopic/issues/79)):

```python
model = BERTopic(embedding_model=my_embedding_model)
model.fit(docs, my_precomputed_embeddings)
model.find_topics(search_term)
```

## **Version 0.6.0**
*Release date:  1 March, 2021*

**Highlights**:

* DTM: Added a basic dynamic topic modeling technique based on the global c-TF-IDF representation
    * `model.topics_over_time(docs, timestamps, global_tuning=True)`
* DTM: Option to evolve topics based on t-1 c-TF-IDF representation which results in evolving topics over time
    * Only uses topics at t-1 and skips evolution if there is a gap
    * `model.topics_over_time(docs, timestamps, evolution_tuning=True)`
* DTM: Function to visualize topics over time
    * `model.visualize_topics_over_time(topics_over_time)`
* DTM: Add binning of timestamps
    * `model.topics_over_time(docs, timestamps, nr_bins=10)`
* Add function get general information about topics (id, frequency, name, etc.)
    *  `get_topic_info()`
* Improved stability of c-TF-IDF by taking the average number of words across all topics instead of the number of documents

**Fixes**:

*  `_map_probabilities()` does not take into account that there is no probability of the outlier class and the probabilities are mutated instead of copied (#63, #64)

## **Version 0.5.0**
*Release date:  8 Februari, 2021*

**Highlights**:

* Add `Flair` to allow for more (custom) token/document embeddings, including 🤗 transformers
* Option to use custom UMAP, HDBSCAN, and CountVectorizer
* Added `low_memory` parameter to reduce memory during computation
* Improved verbosity (shows progress bar)
* Return the figure of `visualize_topics()`
* Expose all parameters with a single function: `get_params()`

**Fixes**:

* To simplify the API, the parameters stop_words and n_neighbors were removed. These can still be used when a custom UMAP or CountVectorizer is used.
* Set `calculate_probabilities` to False as a default. Calculating probabilities with HDBSCAN significantly increases computation time and memory usage. Better to remove calculating probabilities or only allow it by manually turning this on.
* Use the newest version of `sentence-transformers` as it speeds ups encoding significantly

## **Version 0.4.2**
*Release date:  10 Januari, 2021*

**Fixes**:

* Selecting `embedding_model` did not work when `language` was also used. This led to the user needing
to set `language` to None before being able to use `embedding_model`. Fixed by using `embedding_model` when
`language` is used (as a default parameter).

## **Version 0.4.1**
*Release date:  07 Januari, 2021*

**Fixes**:

* Simple fix by lowering the languages variable to match the lowered input language.

## **Version 0.4.0**
*Release date:  21 December, 2020*

**Highlights**:

* Visualize Topics similar to [LDAvis](https://github.com/cpsievert/LDAvis)
* Added option to reduce topics after training
* Added option to update topic representation after training
* Added option to search topics using a search term
* Significantly improved the stability of generating clusters
* Finetune the topic words by selecting the most coherent words with the highest c-TF-IDF values
* More extensive tutorials in the documentation

**Notable Changes**:

* Option to select language instead of sentence-transformers models to minimize the complexity of using BERTopic
* Improved logging (remove duplicates)
* Check if BERTopic is fitted
* Added TF-IDF as an embedder instead of transformer models (see tutorial)
* Numpy for Python 3.6 will be dropped and was therefore removed from the workflow.
* Preprocess text before passing it through c-TF-IDF
* Merged `get_topics_freq()` with `get_topic_freq()`

**Fixes**:

* Fix error handling topic probabilities

## **Version 0.3.2**
*Release date:  16 November, 2020*

**Highlights**:

* Fixed a bug with the topic reduction method that seems to reduce the number of topics but not to the nr_topics as defined in the class. Since this was, to a certain extend, breaking the topic reduction method a new release was necessary.

## **Version 0.3.1**
*Release date:  4 November, 2020*

**Highlights**:

* Adding the option to use custom embeddings or embeddings that you generated beforehand with whatever package you'd like to use. This allows users to further customize BERTopic to their liking.

## **Version 0.3.0**
*Release date:  29 October, 2020*

**Highlights**:

- transform() and fit_transform() now also return the topic probability distributions
- Added visualize_distribution() which visualizes the topic probability distribution for a single document

## **Version 0.2.2**
*Release date:  17 October, 2020*

**Highlights**:

- Fixed n_gram_range not being used
- Added option for using stopwords

## **Version 0.2.1**
*Release date:  11 October, 2020*

**Highlights**:

* Improved the calculation of the class-based TF-IDF procedure by limiting the calculation to sparse matrices. This prevents out-of-memory problems when faced with large datasets.

## **Version 0.2.0**
*Release date:  11 October, 2020*

**Highlights**:

- Changed c-TF-IDF procedure such that it implements a version of scikit-learns procedure. This should also speed up the calculation of the sparse matrix and prevent memory errors.
- Added automated unit tests

## **Version 0.1.2**
*Release date:  1 October, 2020*

**Highlights**:

* When transforming new documents, self.mapped_topics seemed to be missing. Added to the init.

## **Version 0.1.1**
*Release date:  24 September, 2020*

**Highlights**:

* Fixed requirements --> Issue with pytorch
* Update documentation

## **Version 0.1.0**
*Release date:  24 September, 2020*

**Highlights**:

- First release of `BERTopic`
- Added parameters for UMAP and HDBSCAN
- Option to choose sentence-transformer model
- Method for transforming unseen documents
- Save and load trained models (UMAP and HDBSCAN)
- Extract topics and their sizes

**Notable Changes**:

- Optimized c-TF-IDF
- Improved documentation
- Improved topic reduction

repository: MaartenGr/BERTopic

gha_language: Python

repo_name: MaartenGr/BERTopic


text:
# FritzToSpace

Date: 01-02-2013
Authors: [Jonathan Pirnay](http://johnnycrab.com), [Jörn Röder](http://joernroeder.de)
Keywords: net art, hacking, axe appollo, nodejs
Description: short description here!

---
---

![](portrait.jpg)

In 2013, Unilever's male cosmetics subsidiary »AXE« launched a campaign called the »AXE Apollo Space Academy« in order to advertise its new »AXE Apollo« product range – a deodorant, shower gel, shampoo/conditioner and styling gel. In a large scale commercial campaign with the slogan ___»Leave a man, come back a hero«___, AXE tried to depict the character of the astronaut as the most masculine of all males, by women preferred to all other men.
Within the campaign, you had been given the opportunity to register on the website [axeapollo.com](http://axeapollo.com) and encourage your friends to vote for you by clicking a »vote«-button, entering a valid email address and a reCAPTCHA (to prove you're human) . After approx. two months, the champions would be drawn at random from the participants with the most votes, winning a place at the »AASA Space Camp« in Orlando, Florida. Quoting AXE, there the winners would be _»put through rigorous astronaut training. Be warned – this is not for the faint-hearted. […] We'll assess your abilities and the top recruits will be chosen to go into space – as in, actual space«_.

![](_MG_0508.jpg)

Under the motto ___»Leave a nerd, come back a hero«___, we coded a system with the intention to send our friend and fellow student [Fritz Laszlo Weber](http://fritz-weber.de) to first position – and maybe space.
Technically speaking, a Google Chrome extension takes the place of the user, simulating clicking and typing, surfing anonymously through a Tor proxy. A node.js server provides each Chrome extension with email addresses from a list of thousands of valid addresses from facebook.com which have been collected in an earlier process.
Furthermore, the server accesses each user's Tor controller, thus taking care of requesting new identities, providing the voter with fresh IPs from all around the world.
In short, all the voter has to do is type in the reCAPTCHA, hit enter, wait a second for a new identity and mail address, type in the captcha, hit enter, wait for address and identity, etc. etc.

![](server_terminal_screen.png)

In the evening of 15 February 2013, some fellow students gathered to register Fritz on axeapollo.com and »vote« him from around position 15000 to position 1 on the German/Austrian/Swiss leaderboard with a small margin – but no further; all actions would be limited to this one event.

![](fritz_stats_15498.jpg)<iframe src="//player.vimeo.com/video/61797215?title=0&byline=0&portrait=0&color=ffffff" width="760" height="507" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe>

With that in mind, FritzToSpace should be seen as an experiment.
It is an attempt to confront the absurdity of such commercial campaigns with even more insanity.
It is an attempt to slowly approach the exploitation of computer system flaws as a form of creating something new. Not always necessarily art, but beyond the usual finding and fixing, or finding and destructively capitalising system flaws.
FritzToSpace is an experiment to treat hacking as a form of communication and social expression, exploring its possibilities and how this consciousness – as well as the consciousness of possible security flaws – shapes creative decisions.

Fritz Weber reached first place around 1.30 a.m., 16 February 2013, with 4.000 votes.
The day after, he was overtaken, but stayed on second place until the end of the competition.
He participated in the drawing, but didn't win a ticket to the AASA Space Camp in Florida.
Unfortunately, Fritz Weber didn't fly to space, instead he was awarded with a consolation prize: a packet full of AXE Apollo products.

![](_MG_0662.jpg)

_FritzToSpace has been exhibited at the »Rundgang 2013« in Kassel, Germany._

---

![](_MG_0536.jpg)
![](_MG_0504.jpg)
![](_MG_0517.jpg)
![](_MG_0545.jpg)
![](_MG_0557.jpg)
![](_MG_0556.jpg)
![](_MG_0597.jpg)
![](terminal_code_screen.png)
![](_MG_0588.jpg)
![](2013-02-15_19.29.40.jpg)
![](_MG_0652.jpg)
![](_MG_0626.jpg)
![](shot_529.jpg)

---

![](_MG_0752_760.jpg)
![](_MG_0762_760.jpg)
![](_MG_0772_760.jpg)

gha_language: PHP


text:
# Scale

Scale is a suite of interfaces that can be used to replace hard-coded constants.

## Installation

To use Scale, simple run:

```bash
composer require flavioheleno/scale
```

## Usage

This library usage is straightforward.

```php
// before
usleep(30000);

// after
usleep(30 * Milliseconds::IN_MICROSECONDS)
```

```php
// before
$item->expiresAfter(86400);

// after
$item->expiresAfter(Days::IN_SECONDS);
```

## License

This library is licensed under the [MIT License](LICENSE).

repository: flavioheleno/scale

repo_name: flavioheleno/scale