Docker Multi-GPU DDP training hang on `destroy_process_group()`  with `wandb` option 3

Hello, when I try to training using multi gpu based on docker file images. I got the below error. I use Ubuntu 18.04, python 3.8.
<<<<<<<<<<<<<<<<<ERROR>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

```
root@5a70a5f2d489:/usr/src/app# python -m torch.distributed.run --nproc_per_node 2 train.py --batch 64 --data data.yaml --weights yolov5s.pt --device 0,1
WARNING:__main__:*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
*****************************************
Traceback (most recent call last):
  File "train.py", line 620, in <module>
    main(opt)
  File "train.py", line 497, in main
    check_file(opt.data), check_yaml(opt.cfg), check_yaml(opt.hyp), str(opt.weights), str(opt.project)  # checks
  File "/usr/src/app/utils/general.py", line 326, in check_file
    assert len(files), f'File not found: {file}'  # assert file was found
AssertionError: File not found: data.yaml
wandb: (1) Create a W&B account
wandb: (2) Use an existing W&B account
wandb: (3) Don't visualize my results
wandb: Enter your choice: (30 second timeout) ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 1 (pid: 405) of binary: /opt/conda/bin/python
/opt/conda/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py:367: UserWarning:

**********************************************************************
               CHILD PROCESS FAILED WITH NO ERROR_FILE
**********************************************************************
CHILD PROCESS FAILED WITH NO ERROR_FILE
Child process 405 (local_rank 1) FAILED (exitcode 1)
Error msg: Process failed with exitcode 1
Without writing an error file to <N/A>.
While this DOES NOT affect the correctness of your application,
no trace information about the error will be available for inspection.
Consider decorating your top level entrypoint function with
torch.distributed.elastic.multiprocessing.errors.record. Example:

  from torch.distributed.elastic.multiprocessing.errors import record

  @record
  def trainer_main(args):
     # do train
**********************************************************************
  warnings.warn(_no_error_file_warning_msg(rank, failure))
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/run.py", line 702, in <module>
    main()
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 361, in wrapper
    return f(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/run.py", line 698, in main
    run(args)
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/run.py", line 689, in run
    elastic_launch(
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 116, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 244, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
***************************************
            train.py FAILED
=======================================
Root Cause:
[0]:
  time: 2021-10-13_04:30:25
  rank: 1 (local_rank: 1)
  exitcode: 1 (pid: 405)
  error_file: <N/A>
  msg: "Process failed with exitcode 1"
=======================================
Other Failures:
  <NO_OTHER_FAILURES>
***************************************

root@5a70a5f2d489:/usr/src/app#

```


Remove unnecessary configs such as: tdd, tdd_plus, clarify, respec

If we have time: benchmark them and store insights before deletion


UnicodeDecodeError - help will be appriciate!

_Exception in thread Thread-1 (save_and_display_stream):
Traceback (most recent call last):
  File "C:\Users\ziv\AppData\Local\Programs\Python\Python311\Lib\threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "C:\Users\ziv\AppData\Local\Programs\Python\Python311\Lib\threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\ziv\AppData\Local\Programs\Python\Python311\Lib\site-packages\interpreter\code_interpreter.py", line
293, in save_and_display_stream
    for line in iter(stream.readline, ''):
  File "C:\Users\ziv\AppData\Local\Programs\Python\Python311\Lib\encodings\cp1255.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8e in position 3284: character maps to <undefined>_

I am a Windows User, running with Conda, on Python version 3.11.2
 should I change the encoding?

{'base_commit': 'd57ed889c27d5e95e39ea7db59fe518b5f18f942', 'files': [{'path': 'interpreter/code_interpreters/subprocess_code_interpreter.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [0]}, "('SubprocessCodeInterpreter', 'start_process', 39)": {'add': [42, 50]}}}]}


ansible fact_cache permissions changed after ansible-core update

### Summary

After update to ansible-core 2.13.2 or higher (It is still an issue with 2.16.3), the default permission of ansible fact cache files changed.

ansible-core 2.13.1 is OK and uses 0644 on the fact files. 2.13.2 and higher uses 0600.

I could not figure out how to change the behavior back.
We need read permission for the group per default.
This is a breaking change for us.

I did not find a hint in the release notes, so I assume this is a bug

https://github.com/ansible/ansible/compare/v2.13.1...v2.13.2

We have a multi user system, and now ansible user cannot read the cache if a ansible run has been executed by another user.

### Issue Type

Bug Report

### Component Name

cache

### Ansible Version
(EDIT: section updated for bot to detect latest version, tested with 2.13.1,2.13.2 and 2.16.3)
```console
$ ansible --version

(ansible-venv-old) xxx:~> ansible --version
ansible [core 2.16.3]
  config file = None
  configured module search path = ['/home/xxx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/xxx/ansible-venv-old/lib64/python3.11/site-packages/ansible
  ansible collection location = /home/xxx/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/xxx/ansible-venv-old/bin/ansible
  python version = 3.11.5 (main, Sep 06 2023, 11:21:05) [GCC] (/home/xxx/ansible-venv-old/bin/python3.11)
  jinja version = 3.1.3
  libyaml = True
```

### Configuration

```console
# if using a version older than ansible-core 2.12 you should omit the '-t all'
$ ansible-config dump --only-changed -t all

CACHE_PLUGIN(env: ANSIBLE_CACHE_PLUGIN) = jsonfile
CACHE_PLUGIN_CONNECTION(env: ANSIBLE_CACHE_PLUGIN_CONNECTION) = /home/xxx/facts_cache

CACHE:
=====

jsonfile:
________
_uri(env: ANSIBLE_CACHE_PLUGIN_CONNECTION) = /home/xxx/facts_cache
```

### OS / Environment

It is reproducible in with python3.11 venv
Linux

### Steps to Reproduce

After update to 2.13.2, the facts files have 600.
```
(ansible-venv-old) xxx:~> pip install ansible-core==2.13.2
...

xxx:~> rm -r "$HOME/facts_cache"
xxx:~> export ANSIBLE_CACHE_PLUGIN=jsonfile
xxx:~> export ANSIBLE_CACHE_PLUGIN_CONNECTION="$HOME/facts_cache"
xxx:~> ansible -m setup localhost > /dev/null
xxx:~> ls -lisa facts_cache/
total 64
535518  0 drwxr-xr-x 2 xxx yyy 23 Feb  8 13:00 .
262283  0 drwx------ 6 xxx yyy 247 Feb  8 13:00 ..
535519 64 -rw------- 1 xxx yyy 65091 Feb  8 13:00 localhost
```

### Expected Results

With 2.13.1, the permission on the fact file are 644:
```
(ansible-venv-old) xxx:~> ansible --version | head -1
ansible [core 2.13.1]

xxx:~> rm -r "$HOME/facts_cache"
xxx:~> export ANSIBLE_CACHE_PLUGIN=jsonfile
xxx:~> export ANSIBLE_CACHE_PLUGIN_CONNECTION="$HOME/facts_cache"
xxx:~> ansible -m setup localhost > /dev/null
xxx:~> ls -lisa facts_cache/
total 64
535518  0 drwxr-xr-x 2 xxx yyy 23 Feb  8 12:54 .
262283  0 drwx------ 6 xxx yyy 247 Feb  8 12:54 ..
535519 64 -rw-r--r-- 1 xxx yyy 63445 Feb  8 12:54 localhost
```

### Actual Results

After update to 2.13.2 or higher (even latest 2.16.3), the facts files have 600.
See steps to reproduce

### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct

{'base_commit': 'd9e798b48f62fdc2b604a84c36eb83c985f87754', 'files': [{'path': 'lib/ansible/plugins/cache/__init__.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [30]}, "('BaseFileCacheModule', 'set', 154)": {'add': [166]}}}]}


[Bug]: Extension Updates Overwrite with a git reset --hard

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

I can't rely on users config files not being overwritten. If I use `install.py` to rename them, `install.py` does not run until next cold boot. This causes the extension to not run when first installed.  I can probably come with another workaround, like hardcoding the modifications in the apps script.
I shouldn't be having to try workarounds when someone else can't bother to just chmod their files.

https://github.com/AUTOMATIC1111/stable-diffusion-webui/pull/4646#issuecomment-1364629164

### Steps to reproduce the problem

It's in the code:

![image](https://user-images.githubusercontent.com/9631031/209453808-85c862f3-3802-4961-a082-09a6fcde823a.png)

### What should have happened?

Config files to run the extensions should not be overwritten.

### Commit where the problem happens

current to aee611adb874fbabcdeea154a35908ae1f9a4bbf

### What platforms do you use to access UI ?

Windows

### What browsers do you use to access the UI ?

Mozilla Firefox, Google Chrome

### Command Line Arguments

_No response_

### Additional information, context and logs

https://github.com/AUTOMATIC1111/stable-diffusion-webui/pull/4646
https://github.com/Gerschel/sd_web_ui_preset_utils/issues/23

{'base_commit': '98947d173e3f1667eba29c904f681047dea9de90', 'files': [{'path': 'modules/extensions.py', 'status': 'modified', 'Loc': {"('Extension', None, 17)": {'mod': [68]}, "('Extension', 'pull', 68)": {'mod': [70]}}}, {'path': 'modules/ui_extensions.py', 'status': 'modified', 'Loc': {"(None, 'apply_and_restart', 23)": {'mod': [39, 41]}}}]}


cloud
aws
module
affects_2.4
support:core
bug

Add support for Timeout (--timeout-in-minutes) parameter in Cloudformation module

##### ISSUE TYPE

- Bug Report

##### COMPONENT NAME

Cloudformation module

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes below -->
```
ansible 2.4.2.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/ubuntu/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.12 (default, Nov 20 2017, 18:23:56) [GCC 5.4.0 20160609]

```

##### CONFIGURATION

No changes

##### OS / ENVIRONMENT

Ubuntu 16.04

##### SUMMARY

I believe this is a bug, that Ansible Cloudformation module does not support important Timeout parameter (--timeout-in-minutes key in aws-cli 'create-stack' call).

(Documentation:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-add-tags.html )

##### STEPS TO REPRODUCE

Check module documentation

##### EXPECTED RESULTS

Timeout parameter is supported

##### ACTUAL RESULTS

Timeout parameter is not supported

{'base_commit': 'f8d20f970f16806aee1ef555f9f2db115cec7f34', 'files': [{'path': 'lib/ansible/modules/cloud/amazon/cloudformation.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [32, 209]}, "(None, 'create_stack', 300)": {'add': [306], 'mod': [304, 305]}, "(None, 'main', 535)": {'add': [544]}}}]}


Bug with using TreeClassifier with OOB score and sparse matrices

When using the ExtraTreesClassifier (and likely other classes that are derived from BaseTreeClassifier), there is a problem when using sparsematrices: `ValueError: X should be in csr_matrix format, got <class 'scipy.sparse.csc.csc_matrix'>`.

I tracked the issue down to the following lines:

On line 195 of forest.py the sparse matrix is changed to a csc matrix:
`X = check_array(X, dtype=DTYPE, accept_sparse="csc")`

However on line 369 of forest.py, the following is call is made with `check_input=false`:
`p_estimator = estimator.predict_proba(X[mask_indices, :], check_input=False)`

This leads to a ValueError in predict `ValueError: X should be in csr_matrix format, got <class 'scipy.sparse.csc.csc_matrix'>`.

Changing check_input to True seems to fix the issue. It's probably best to also include a test case for this problem, I just made a quick PR with only the False -> True fix.

{'base_commit': 'abb31d0a7ca769a1e6406553a58a7fb0bd3b259a', 'files': [{'path': 'doc/whats_new.rst', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [114]}}}, {'path': 'sklearn/ensemble/forest.py', 'status': 'modified', 'Loc': {"('ForestClassifier', '_set_oob_score', 374)": {'add': [375]}, "('ForestRegressor', '_set_oob_score', 659)": {'add': [660]}}}, {'path': 'sklearn/ensemble/tests/test_forest.py', 'status': 'modified', 'Loc': {"(None, 'test_oob_score', 261)": {'add': [264]}, '(None, None, None)': {'add': [270]}}}]}


dependency_overrides does not play well with scopes

**Describe the bug**
When working with `Security()` dependencies, the scopes disappear when `app.dependency_overrides` is executed. The callable dealing with the scopes gets an empty list instead of the scopes.

**To Reproduce**

```python
from fastapi import FastAPI, Header, Security, Depends
from fastapi.security import SecurityScopes

from starlette.testclient import TestClient

app = FastAPI()

def get_user(required_scopes: SecurityScopes):
    print(required_scopes.scopes)

    return "John Doe"

def data():
    return [1,2,3]

def other_data():
    return [3,4,5]

@app.get("/test")
def test(user: str = Security(get_user, scopes=["foo", "bar"]), data = Depends(data)):
    return data

client = TestClient(app)
response = client.get("/test")

app.dependency_overrides[data] = other_data
response = client.get("/test")

# prints: ["foo", "bar"] and [] instead of ["foo", "bar"] and ["foo", "bar"]
```

**Expected behavior**
In the above example I expect `get_user()` to print the same scopes twice. Instead, before the `dependency_overrides` it prints the correct scpoes, but an empty list afterwards.

**Environment:**
 - OS: Linux
 - FastAPI Version 0.43.0
- Python 3.7.4

{'base_commit': '543ef7753aff639ad3aed7c153e42f719e361d38', 'files': [{'path': 'fastapi/dependencies/utils.py', 'status': 'modified', 'Loc': {"(None, 'solve_dependencies', 432)": {'add': [480]}}}]}


custom anchors get flushed when loading pretrain weights

Before submitting a bug report, please be aware that your issue **must be reproducible** with all of the following, otherwise it is non-actionable, and we can not help you:
 - **Current repo**: run `git fetch && git status -uno` to check and `git pull` to update repo
 - **Common dataset**: coco.yaml or coco128.yaml
 - **Common environment**: Colab, Google Cloud, or Docker image. See https://github.com/ultralytics/yolov5#reproduce-our-environment

If this is a custom dataset/training question you **must include** your `train*.jpg`, `test*.jpg` and `results.png` figures, or we can not help you. You can generate these with `utils.plot_results()`.

## 🐛 Bug
in train.py  , the anchors set by user in yaml file are flushed by pretrain weights.
```

    if weights.endswith('.pt'):  # pytorch format
        ckpt = torch.load(weights, map_location=device)  # load checkpoint

        # load model
        try:
            ckpt['model'] = {k: v for k, v in ckpt['model'].float().state_dict().items()
                             if model.state_dict()[k].shape == v.shape}  # to FP32, filter
            #print(ckpt['model'].keys())
            **#ckpt['model'].pop('model.27.anchors')
            #ckpt['model'].pop('model.27.anchor_grid')**

            model.load_state_dict(ckpt['model'], strict=False)
        except KeyError as e:
            s = "%s is not compatible with %s. This may be due to model differences or %s may be out of date. " \
                "Please delete or update %s and try again, or use --weights '' to train from scratch." \
                % (opt.weights, opt.cfg, opt.weights, opt.weights)
            raise KeyError(s) from e

```
## To Reproduce (REQUIRED)

Input:
in ./model/yolov5x.yaml
change anchors' shape to any other than default.

Output:
the anchors set in yaml file didn't activated .

## Expected behavior
A clear and concise description of what you expected to happen.

## Environment
If applicable, add screenshots to help explain your problem.

 - OS: [Ubuntu]
 - GPU [2080 Ti]

## Additional context
if the anchors set by user in yaml file, is more than 9 anchors, the bug didn't get triggered because it did not match the pretrain weight's anchors' shape.


"No API key provided" - altough it is provided in the .env file

## Expected Behavior

If the OpenAI API key is provided in the .env file, it should be recognized and used.

## Current Behavior

Runtime error message: openai.error.AuthenticationError: No API key provided.

### Steps to Reproduce

1. Set the key in the .env file
2. Run the app with gpt-engineer projects/my-new-project

### Solution

When I added the line `openai.api_key = os.getenv("OPENAI_API_KEY")` to the end of the function `load_env_if_needed()` in the file `main.py`, as well as `import openai` at the beginning of this file _(thanks, engerlina, for reminder)_, the issue was resolved.

{'base_commit': '65d7a9b9902ad85f27b17d759bd13b59c2afc474', 'files': [{'path': 'gpt_engineer/main.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [5]}, "(None, 'load_env_if_needed', 19)": {'add': [21]}}}]}


Fix and document asyncio reactor problems on Windows

As described in https://twistedmatrix.com/trac/ticket/9766 you cannot just enable AsyncioSelectorReactor on Windows with recent Python, you either need fixed Twisted (which is not released yet, the merged fix is https://github.com/twisted/twisted/pull/1338) or, supposedly, add some manual fix as documented [here](https://github.com/twisted/twisted/blob/09b96850c2ebcb635f448ed3f9bbf5f157be3693/src/twisted/internet/asyncioreactor.py#L35-L44). So if it's possible to add this code to Scrapy we should probably do that, at least until the next Twisted release, and even after it we should document that new enough Twisted is needed in this use case.

{'base_commit': '57dc58123b98e2026025cc87bdee474bf0656dcb', 'files': [{'path': '.github/workflows/tests-windows.yml', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [25]}}}, {'path': 'docs/topics/asyncio.rst', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [38]}}}, {'path': 'scrapy/utils/reactor.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [1]}, "(None, 'install_reactor', 53)": {'add': [59]}}}, {'path': 'tests/CrawlerProcess/asyncio_enabled_reactor.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [1, 3]}}}, {'path': 'tests/test_commands.py', 'status': 'modified', 'Loc': {"('RunSpiderCommandTest', None, 557)": {'mod': [677, 678, 679, 702, 703, 704]}, "('RunSpiderCommandTest', 'test_custom_asyncio_loop_enabled_false', 705)": {'mod': [710]}}}, {'path': 'tests/test_crawler.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'mod': [7]}, "('CrawlerRunnerHasSpider', None, 231)": {'mod': [287, 288, 289]}, "('CrawlerProcessSubprocess', None, 323)": {'mod': [331, 332, 333, 339, 340, 341, 380, 381, 382, 407, 408, 409]}}}, {'path': 'tests/test_downloader_handlers.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [3]}, "('HttpTestCase', None, 209)": {'add': [289, 298]}, "('FTPTestCase', None, 1055)": {'add': [1057]}}}, {'path': 'tests/test_utils_asyncio.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'mod': [1, 2, 3]}, "('AsyncioTest', None, 11)": {'mod': [17, 18, 19]}}}]}


Replacing multiple columns (or just one) with iloc does not work

#### Code Sample, a copy-pastable example if possible

```python
import pandas

columns = pandas.DataFrame({'a2': [11, 12, 13], 'b2': [14, 15, 16]})
inputs = pandas.DataFrame({'a1': [1, 2, 3], 'b1': [4, 5, 6], 'c1': [7, 8, 9]})

inputs.iloc[:, [1]] = columns.iloc[:, [0]]

print(inputs)
```

#### Problem description

I have a code which is replacing a set of columns with another set of columns, based on column indices. To make things done without a special case, I assumes I could just use `iloc` to both select and set columns in a DataFrame. But it seems that this not work and fails in strange ways.

#### Expected Output

```
   a1  b1  c1
0   1  11   7
1   2  12   8
2   3  13   9
```

But in reality, you get:

```
    a1  b1   c1
0  1.0 NaN  7.0
1  2.0 NaN  8.0
2  3.0 NaN  9.0
```

See how values converted to float and how column is `NaN`s?

But, if I do the following I get expected results:

```
inputs.iloc[:, [1]] = [[11], [12], [13]]
```

This also works:

```
inputs.iloc[:, [1]] = columns.iloc[:, [0]].values
```

So if it works with lists and ndarrays, one would assume it would also work with DataFrames themselves. But it does not.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.3.final.0
python-bits: 64
OS: Linux
OS-release: 4.13.0-46-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.23.3
pytest: None
pip: 18.0
setuptools: 40.0.0
Cython: None
numpy: 1.15.0
scipy: None
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.7.3
pytz: 2018.5
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

</details>

{'base_commit': '9b4dfa195e3f23d81389745c26bff8e0087e74b0', 'files': [{'path': 'doc/source/whatsnew/v1.2.0.rst', 'status': 'modified', 'Loc': {'(None, None, 591)': {'add': [591]}}}, {'path': 'pandas/core/indexing.py', 'status': 'modified', 'Loc': {"('_LocationIndexer', '__setitem__', 675)": {'mod': [684]}, "('_iLocIndexer', None, 1322)": {'mod': [1520, 1631, 1717, 1790]}, "('_iLocIndexer', '_setitem_with_indexer', 1520)": {'mod': [1596, 1627, 1629]}, "('_iLocIndexer', '_setitem_with_indexer_split_path', 1631)": {'mod': [1645, 1660]}, "('_iLocIndexer', '_setitem_with_indexer_frame_value', 1717)": {'mod': [1727]}, "('_iLocIndexer', '_setitem_single_block', 1790)": {'mod': [1819, 1825]}, "('_iLocIndexer', '_setitem_with_indexer_missing', 1836)": {'mod': [1857]}}}, {'path': 'pandas/tests/frame/indexing/test_setitem.py', 'status': 'modified', 'Loc': {"('TestDataFrameSetItem', None, 24)": {'mod': [292, 293, 294, 295, 296, 297, 298, 299]}}}, {'path': 'pandas/tests/indexing/test_iloc.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [803]}, "('TestILocSeries', 'test_iloc_getitem_nonunique', 966)": {'add': [968]}}}, {'path': 'pandas/tests/indexing/test_indexing.py', 'status': 'modified', 'Loc': {"('TestMisc', 'test_rhs_alignment', 668)": {'mod': [671, 690, 696, 697, 700, 703, 707]}, "('TestMisc', 'run_tests', 671)": {'mod': [678, 682, 686]}}}]}


DO NOT FORCE VALIDATE '{'Required Python packages'}' by default

**Bug description**
`metagpt\actions\action_node.py", line 432, in _aask_v1
    instruct_content = output_class(**parsed_data)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "..........\Lib\site-packages\pydantic\main.py", line 171, in __init__
    self.__pydantic_validator__.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 1 validation error for PM_NODE_AN
  Value error, Missing fields: {'Required Python packages'} `

**Bug solved method**
DO NOT VALIDATE THIS FIELD. user may ask the agents to do non py related stuff,why would we force this validate and introduce a hard error? Seems silly.

**Environment information**
irrelevant

- LLM type and model name:
- MetaGPT version or branch:0.8.1

**Screenshots or logs**
`action_node.py", line 432, in _aask_v1
    instruct_content = output_class(**parsed_data)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".........\Lib\site-packages\pydantic\main.py", line 171, in __init__
    self.__pydantic_validator__.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 1 validation error for PM_NODE_AN
  Value error, Missing fields: {'Required Python packages'} [type=value_error, input_value={'Required Rust packages'...ption for backup data.'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.6/v/value_error

The above exception was the direct cause of the following exception:

Traceback (most recent call last):`

{'base_commit': '5446c7e490e7203c61b2ff31181551b2c0f4a86b', 'files': [{'path': 'metagpt/actions/design_api_an.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [47], 'mod': [8, 50, 69]}}}, {'path': 'metagpt/actions/project_management_an.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'mod': [8, 14]}}}]}


Pytorch synthesizer

Splitting this off from #370, which will remain for tensorflow2 conversion. I would prefer this route if we can get it to work. Asking for help from the community on this one.

One example of a pytorch-based tacotron is: https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/SpeechSynthesis/Tacotron2

Another option is to manually convert the code and pretrained models which would be extremely time-consuming, but also an awesome learning experience.


### Describe the bug

When I run `interpreter --os`

And then attempt a command like:
`Play a boiler room set on youtube`

I get a `NameError`:

```
▌ OS Control enabled

To find items on the screen, Open Interpreter has been instructed to send screenshots to api.openinterpreter.com (we do not store them). Add --offline to attempt this locally.

Make sure that screen recording permissions are enabled for your Terminal or Python environment.

Warning: In this mode, Open Interpreter will not require approval before performing actions. Be ready to close your terminal.

> Play a boiler room set on youtube

  The first step is to open YouTube. We can do this by using a keyboard hotkey to open Spotlight and then typing "Safari" to open the browser. Once Safari is open, we can navigate
  to YouTube by writing the URL in the address bar. Let's start by opening Safari.

Expecting value: line 1 column 1 (char 0)
Continuing.

  computer.keyboard.hotkey("space", "command") # Opens spotlight
  computer.keyboard.write("Safari")
  computer.keyboard.press("enter") # To open Safari

  ---------------------------------------------------------------------------
  NameError                                 Traceback (most recent call last)
  Input In [4], in <cell line: 2>()
        1 print('##active_line1##')
  ----> 2 computer.keyboard.hotkey('space', 'command')
        3 print('##active_line2##')
        4 computer.keyboard.write('Safari')

  NameError: name 'computer' is not defined

  It seems there was a mistake in my attempt to run the code. "computer" is not recognized as defined within the current execution environment. Let's correct this by using the
  appropriate syntax to execute the code on your machine. We will try to open Safari using the correct approach. Let's run the correct code.

Expecting value: line 1 column 1 (char 0)
Continuing.

  # Opens spotlight and types Safari to open the browser
  computer.keyboard.hotkey("space", "command")
  computer.keyboard.write("Safari")
  computer.keyboard.press("enter")

  ---------------------------------------------------------------------------
  NameError                                 Traceback (most recent call last)
  Input In [7], in <cell line: 4>()
        2 pass
        3 print('##active_line2##')
  ----> 4 computer.keyboard.hotkey('space', 'command')
        5 print('##active_line3##')
        6 computer.keyboard.write('Safari')

  NameError: name 'computer' is not defined
```

And it just gets stuck in this loop where computer is not defined.

### Reproduce

1. `interpreter --os`
2. `Play a boiler room set on youtube`

### Expected behavior

For it to be able to open Safari or my default web browser without a Name Error of computer.

### Screenshots

_No response_

### Open Interpreter version

0.2.0

### Python version

3.9.6

### Operating System name and version

macOS 14.0

### Additional context

I have 2 python versions installed. 3.9.6 and 3.10.8. I installed interpreter on both.

{'base_commit': '3c922603c0a7d1ad4113245a3d2bcd23bf4b1619', 'files': [{'path': 'interpreter/core/computer/terminal/terminal.py', 'status': 'modified', 'Loc': {"('Terminal', 'run', 36)": {'mod': [40]}}}, {'path': 'interpreter/terminal_interface/start_terminal_interface.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'mod': [4]}, "(None, 'start_terminal_interface', 19)": {'mod': [303, 544, 545, 546, 548, 593, 603, 608, 633]}}}]}


SileroTTS creates new audio file for each token

### Describe the bug

I've just performed a fresh install to confirm this.

Unless i turn on no stream, SileroTTS will attempt to create an audio file for each word / token.

Silero should not attempt to create audio until the response is complete.

Silero extension output directory is being filled up with audio clips that only add one word to the previous file. Is this known to be broken like this?

Turning off stream works, but it means that the text stream doesn't work. Is there a way to turn off streaming for Silero only?

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Reproduction

1. Enable Silero Extension
2. Disable Auto Play
3. Start Chat

### Screenshot

_No response_

### Logs

```shell
N/A
```

### System Info

```shell
Windows 11 / Firefox or Edge
```


DOC: pandas.Series(data=None, index=None, dtype=None, name=None, copy=False, fastpath=False)

No proper information on "copy" is present under [Documentation](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.html)


CUDA version incorrect in Dockerfile.gpu

The Dockerfile.gpu doesn't work for me. The built doesn't use GPU at all.

I found that tensorflow cannot find shared library file libXXXX.so.11.0 (If I remember correctly, it's libcudart.so.11.0). I realize that the tensorflow version installed needs CUDA 11.0. But the original Dockerfile.gpu installs the CUDA 10.1.

If someone had similar issue, please modify the Dockerfile with:

FROM nvidia/cuda:11.0.3-cudnn8-devel-ubuntu16.04


[Bug]: Editing Error "No replacement was performed" is not informative enough

### Is there an existing issue for the same bug?

- [X] I have checked the existing issues.

### Describe the bug and reproduction steps

The agent got this error:
```
ERROR:
No replacement was performed. Multiple occurrences of old_str `            output_path = Path.joinpath(self._output_dir, "recipe_state.pt")
            torch.save(state_dict, output_path)
            logger.info(
                "Recipe checkpoint of size "
                f"{os.path.getsize(output_path) / 1000**3:.2f} GB "
                f"saved to {output_path}"
            )` in lines []. Please ensure it is unique.
```

`in lines []. Please ensure it is unique.` does look right? Should we give out the specific line number?

Full trajectory: https://www.all-hands.dev/share?share_id=7c05665906ffb699d93426129b1ee8c50c3cc5c7dcb5e164de9c54f6468e7876

cc @ryanhoangt

### OpenHands Installation

Docker command in README

### OpenHands Version

_No response_

### Operating System

None

### Logs, Errors, Screenshots, and Additional Context

_No response_


STYLE place standard library imports at top of file

Imports should typically be placed at the top of files. Sometimes, imports are placed inside functions to:
- avoid circular imports
- avoid `ImportError` if it's an optional dependency

Standard library imports should really always be at the top of files.

Noticed in https://github.com/pandas-dev/pandas/pull/49645 that this is often not the case

I've made a script to automate detecting when this is the case. So the task is:
```
git checkout -b standard-library-imports main
git pull git@github.com:MarcoGorelli/pandas.git standard-library-imports
git reset --hard FETCH_HEAD
pre-commit run stdlib-imports --all-files
```
Then, fixup any errors that are reported. Finally, stage your changes, commit them, push them to your fork, and open a pull request

Feel free to reach out if you into any issues along the way

If any wants to take this, it would be a nice and welcome clean up!

---

EDIT: after going through a PR, I'm not sure it's worth introducing a check for this - but we can still take some of the cleanups it found

{'base_commit': 'fa78ea801392f4f0d37ea7ddbbfe44e9c8c102bd', 'files': [{'path': 'pandas/tests/apply/test_series_apply.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [4]}, "(None, 'test_apply', 35)": {'mod': [40]}, "(None, 'test_map_decimal', 527)": {'mod': [528]}}}, {'path': 'pandas/tests/arrays/test_datetimelike.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [2]}, "(None, 'array_likes', 1337)": {'mod': [1349, 1350]}}}, {'path': 'pandas/tests/frame/indexing/test_indexing.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [5]}, "('TestDataFrameIndexing', 'test_setitem_ambig', 468)": {'mod': [470]}}}, {'path': 'pandas/tests/frame/methods/test_to_records.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [1]}, "('TestDataFrameToRecords', 'test_to_records_with_Mapping_type', 60)": {'mod': [61, 62]}}}, {'path': 'pandas/tests/frame/test_constructors.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [0, 3, 4, 10]}, "('TestDataFrameConstructors', 'test_constructor_ordereddict', 468)": {'mod': [469]}, "('TestDataFrameConstructors', 'test_constructor_defaultdict', 719)": {'mod': [721]}, "('TestDataFrameConstructors', 'test_constructor_stdlib_array', 1343)": {'mod': [1346]}, "('TestDataFrameConstructors', 'test_constructor_list_of_namedtuples', 1545)": {'mod': [1547]}, "('TestDataFrameConstructors', 'test_constructor_list_of_dataclasses', 1560)": {'mod': [1562]}, "('TestDataFrameConstructors', 'test_constructor_list_of_dataclasses_with_varying_types', 1571)": {'mod': [1573]}, "('TestDataFrameConstructors', 'test_constructor_list_of_dataclasses_error_thrown', 1587)": {'mod': [1589]}}}, {'path': 'pandas/tests/groupby/test_filters.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [0]}, "(None, 'test_filter_against_workaround', 173)": {'mod': [195]}}}, {'path': 'pandas/tests/groupby/test_grouping.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'mod': [1]}, "('TestGrouping', 'test_grouper_multilevel_freq', 169)": {'mod': [173, 174, 175, 176]}}}, {'path': 'pandas/tests/groupby/test_timegrouper.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'mod': [1, 3]}, "('TestGroupBy', 'test_first_last_max_min_on_time_data', 762)": {'mod': [766, 777]}}}, {'path': 'pandas/tests/indexes/test_common.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [5]}, "('TestCommon', 'test_copy_and_deepcopy', 134)": {'mod': [135, 136, 137, 138]}}}, {'path': 'pandas/tests/indexing/multiindex/test_slice.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [0]}, "('TestMultiIndexSlicers', 'test_multiindex_slicers_datetimelike', 247)": {'mod': [251, 253, 254, 255, 256]}}}, {'path': 'pandas/tests/io/excel/test_readers.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [7]}, "('TestReaders', 'test_read_from_file_url', 890)": {'mod': [900]}}}, {'path': 'pandas/tests/io/formats/test_printing.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [0]}, "(None, 'test_repr_binary_type', 21)": {'mod': [22]}}}, {'path': 'pandas/tests/io/formats/test_to_csv.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [5]}, "('TestToCSV', 'test_to_csv_doublequote', 84)": {'mod': [97]}}}, {'path': 'pandas/tests/io/json/test_pandas.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [7]}, "('TestPandasContainer', 'test_to_s3', 1732)": {'mod': [1733]}}}, {'path': 'pandas/tests/io/parser/test_c_parser_only.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [7]}, "(None, 'test_precise_conversion', 171)": {'mod': [172]}}}, {'path': 'pandas/tests/io/parser/test_encoding.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'mod': [5]}, "(None, 'test_utf16_bom_skiprows', 47)": {'mod': [62]}}}, {'path': 'pandas/tests/io/parser/test_python_parser_only.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [12]}, "(None, 'test_sniff_delimiter_encoding', 100)": {'mod': [111]}}}, {'path': 'pandas/tests/io/pytables/test_store.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [3], 'mod': [1]}, "(None, 'test_repr', 110)": {'mod': [129, 130]}, "(None, 'test_table_mixed_dtypes', 431)": {'mod': [444, 445]}, "(None, 'test_calendar_roundtrip_issue', 454)": {'mod': [461, 467, 468]}, "(None, 'test_same_name_scoping', 524)": {'mod': [537]}, "(None, 'test_store_index_name_numpy_str', 558)": {'mod': [561, 565]}, "(None, 'do_copy', 878)": {'mod': [880]}}}, {'path': 'pandas/tests/io/test_orc.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [2]}, "(None, 'test_orc_reader_decimal', 100)": {'mod': [101]}}}, {'path': 'pandas/tests/io/xml/test_xml.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [10]}, "(None, 'test_empty_string_etree', 493)": {'mod': [494]}, "(None, 'test_wrong_file_path_etree', 513)": {'mod': [514]}}}, {'path': 'pandas/tests/plotting/frame/test_frame.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [5, 9]}, "('TestDataFramePlots', 'test_memory_leak', 1783)": {'mod': [1785, 1786]}}}, {'path': 'pandas/tests/reshape/concat/test_concat.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [4]}, "('TestConcatenate', 'test_dtype_coerceion', 337)": {'mod': [346, 348, 349, 350]}}}, {'path': 'pandas/tests/reshape/concat/test_index.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [0]}, "('TestMultiIndexConcat', 'test_concat_multiindex_dfs_with_deepcopy', 241)": {'mod': [243]}}}, {'path': 'pandas/tests/reshape/test_get_dummies.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [1]}, "('TestGetDummies', 'test_get_dummies_unicode', 165)": {'mod': [167]}}}, {'path': 'pandas/tests/series/test_arithmetic.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [1, 4]}, "('TestSeriesArithmetic', 'test_add_na_handling', 224)": {'mod': [225, 226]}}}]}


Remove simplejson

In modern Python it's unlikely to be significantly better than the built-in `json`. The module used by `JSONMixin` is overridable, so users can plug it in again if they want.

See pallets/itsdangerous#146 and pallets/werkzeug#1766.


Outdated DockerFile dependencies

The DockerFile inside the manim-master still contains the python version 2.7.12. Considering that manim had no longer support the python 2. This could lead to a syntax error. Please fix this issue ASAP.


[TikTok] New API fix adds non-playable video codec in available formats

### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting that yt-dlp is broken on a **supported** site
- [X] I've verified that I have **updated yt-dlp to nightly or master** ([update instructions](https://github.com/yt-dlp/yt-dlp#update-channels))
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [X] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

Global

### Provide a description that is worded well enough to be understood

Hi! New API fix adds for some videos a new codec bytevc2 which cannot be played by multimedia players (I used VLC). By default yt_dlp chooses normal codec, but I use `-S res:1080,vcodec:avc1,ext:mp4:m4a` format selection, so yt_dlp downloads video with this new codec.

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [ ] If using API, add `'verbose': True` to `YoutubeDL` params instead
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
user@host:~$ python3 -m yt_dlp -F https://vm.tiktok.com/ZMMPDNEJL/
[vm.tiktok] Extracting URL: https://vm.tiktok.com/ZMMPDNEJL/
[vm.tiktok] ZMMPDNEJL: Downloading webpage
[TikTok] Extracting URL: https://www.tiktok.com/@soyko_max/video/7351538939712359713?_t=8l5sekUiLWo&_r=1
[TikTok] 7351538939712359713: Downloading video feed
[info] Available formats for 7351538939712359713:
ID                    EXT RESOLUTION │   FILESIZE  TBR PROTO │ VCODEC  ACODEC MORE INFO
───────────────────────────────────────────────────────────────────────────────────────────────────────────────
download_addr-0       mp4 720x1280   │    1.31MiB      https │ h264    aac    Download video, watermarked (API)
download_addr-1       mp4 720x1280   │    1.31MiB      https │ h264    aac    Download video, watermarked
download_addr-2       mp4 720x1280   │    1.31MiB      https │ h264    aac    Download video, watermarked
h264_540p_986746-0    mp4 1048x576   │    1.24MiB 986k https │ h264    aac    Direct video (API)
h264_540p_986746-1    mp4 1048x576   │    1.24MiB 986k https │ h264    aac    Direct video
h264_540p_986746-2    mp4 1048x576   │    1.24MiB 986k https │ h264    aac    Direct video
bytevc1_540p_263555-0 mp4 1048x576   │  339.96KiB 263k https │ h265    aac    Playback video (API)
bytevc1_540p_263555-1 mp4 1048x576   │  339.96KiB 263k https │ h265    aac    Playback video
bytevc1_540p_263555-2 mp4 1048x576   │  339.96KiB 263k https │ h265    aac    Playback video
bytevc1_540p_344910-0 mp4 1048x576   │  444.91KiB 344k https │ h265    aac    Playback video (API)
bytevc1_540p_344910-1 mp4 1048x576   │  444.91KiB 344k https │ h265    aac    Playback video
bytevc1_540p_344910-2 mp4 1048x576   │  444.91KiB 344k https │ h265    aac    Playback video
bytevc1_540p_507345-0 mp4 1048x576   │  654.43KiB 507k https │ h265    aac    Direct video (API)
bytevc1_540p_507345-1 mp4 1048x576   │  654.43KiB 507k https │ h265    aac    Direct video
bytevc1_540p_507345-2 mp4 1048x576   │  654.43KiB 507k https │ h265    aac    Direct video
bytevc2_720p_616180-0 mp4 1280x704   │  794.82KiB 616k https │ bytevc2 aac    Playback video (API)
bytevc2_720p_616180-1 mp4 1280x704   │  794.82KiB 616k https │ bytevc2 aac    Playback video
bytevc2_720p_616180-2 mp4 1280x704   │  794.82KiB 616k https │ bytevc2 aac    Playback video
bytevc1_720p_595186-0 mp4 1280x704   │  767.74KiB 595k https │ h265    aac    Playback video (API)
bytevc1_720p_595186-1 mp4 1280x704   │  767.74KiB 595k https │ h265    aac    Playback video
bytevc1_720p_595186-2 mp4 1280x704   │  767.74KiB 595k https │ h265    aac    Playback video

user@host:~$ python3 -m yt_dlp -vU -S res:1080,vcodec:avc1,ext:mp4:m4a https://vm.tiktok.com/ZMMPDNEJL/
[debug] Command-line config: ['-vU', '-S', 'res:1080,vcodec:avc1,ext:mp4:m4a', 'https://vm.tiktok.com/ZMMPDNEJL/']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2024.03.10 from yt-dlp/yt-dlp [615a84447]
[debug] Lazy loading extractors is disabled
[debug] Python 3.10.12 (CPython x86_64 64bit) - Linux-5.15.0-101-generic-x86_64-with-glibc2.35 (OpenSSL 3.0.2 15 Mar 2022, glibc 2.35)
[debug] exe versions: ffmpeg 4.4.2 (setts), ffprobe 4.4.2
[debug] Optional libraries: Cryptodome-3.20.0, brotli-1.1.0, certifi-2024.02.02, mutagen-1.47.0, requests-2.31.0, secretstorage-3.3.1, sqlite3-3.37.2, urllib3-2.2.1, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1807 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: stable@2024.03.10 from yt-dlp/yt-dlp
yt-dlp is up to date (stable@2024.03.10 from yt-dlp/yt-dlp)
[vm.tiktok] Extracting URL: https://vm.tiktok.com/ZMMPDNEJL/
[vm.tiktok] ZMMPDNEJL: Downloading webpage
[TikTok] Extracting URL: https://www.tiktok.com/@soyko_max/video/7351538939712359713?_t=8l5sekUiLWo&_r=1
[debug] [TikTok] iid=7351149742343391009
[TikTok] 7351538939712359713: Downloading video feed
[debug] Sort order given by user: res:1080, vcodec:avc1, ext:mp4:m4a
[debug] Sort order given by extractor: quality, codec, size, br
[debug] Formats sorted by: hasvid, ie_pref, res:1080(1080.0), vcodec:avc1(7), vext:mp4(6), aext:m4a(8), quality, acodec, size, br, lang, fps, hdr:12(7), channels, asr, proto, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] 7351538939712359713: Downloading 1 format(s): bytevc2_720p_616180-2
[debug] Invoking http downloader on "https://v16m.byteicdn.com/366e989edb75b43b4e545b0f6f94180e/6607a32c/video/tos/useast2a/tos-useast2a-ve-0068-euttp/oIEDMCMe2rmeIExnh6mGk70AEkerG1aIRgfFj6/?a=0&bti=OHYpOTY0Zik3OjlmOm01MzE6ZDQ0MDo%3D&ch=0&cr=13&dr=0&lr=all&cd=0%7C0%7C0%7C&cv=1&br=1202&bt=601&cs=5&ds=3&ft=teSL~8QLodzR12NvvEh3hIxR34DaRq_45SY&mime_type=video_mp4&qs=14&rc=aTwzOGdkOGRnMzw1NDQ1OUBpam9pbng5cjx4cjMzZjczM0AvL18yYmBgXjMxXy4tLy5eYSNsa2FnMmRrZy9gLS1kMWNzcw%3D%3D&vvpl=1&l=20240329232904ECF8D33243F1CE142F0B&btag=e00088000&cc=10"
[download] Destination: Оригинал) [7351538939712359713].mp4
[download] 100% of  794.82KiB in 00:00:00 at 7.49MiB/s
```

{'base_commit': '3699eeb67cad333272b14a42dd3843d93fda1a2e', 'files': [{'path': 'yt_dlp/extractor/tiktok.py', 'status': 'modified', 'Loc': {"('TikTokBaseIE', 'extract_addr', 275)": {'add': [276, 288], 'mod': [290]}}}]}


Disable g4f logging completely

**Bug description**
In my script I have my customized logging, but whenever I use it it prints 2 times (one from my logger, one from g4f logger).
How can I turn off the logger inside the library? Already tried a bunch of stuff with no results.

P.S. Are you using the root logger maybe? If that is the case, please use it with the module name

ex.
1. Create a new logger in new class
2. Set logging level to DEBUG
3. Log something
4. Enjoy duplicated output

**Screenshots**
![image](https://github.com/user-attachments/assets/c24a1b65-b513-4449-8842-870552289de0)

**Environment**
- python version: 3.11
- location ( are you in a cloudfare flagged country ) ? nope


BUG: loc should not fallback for integer indexing for multi-index

{'base_commit': '324208eaa66a528f1e88f938c71c2d8efb8304f3', 'files': [{'path': 'doc/source/v0.14.1.txt', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [64]}}}, {'path': 'pandas/core/index.py', 'status': 'modified', 'Loc': {"('Index', '_convert_list_indexer_for_mixed', 607)": {'mod': [612]}}}, {'path': 'pandas/tests/test_indexing.py', 'status': 'modified', 'Loc': {"('TestIndexing', None, 86)": {'add': [808]}}}]}


DataFrame.ix losing row ordering when index has duplicates

``` python
import pandas as pd

ind = ['A', 'A', 'B', 'C']i
df = pd.DataFrame({'test':range(len(ind))}, index=ind)

rows = ['C', 'B']
res = df.ix[rows]
assert rows == list(res.index) # fails
```

The problem is that the resulting DataFrame keeps the ordering of the `df.index` and not the `rows` key. You'll notice that the `rows` key doesn't reference a duplicate value.

{'base_commit': '6d2c57fa010c12f21f700034b5651519670b9b9d', 'files': [{'path': 'RELEASE.rst', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [93, 150]}}}, {'path': 'doc/source/indexing.rst', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [1370]}}}, {'path': 'pandas/core/index.py', 'status': 'modified', 'Loc': {"('Index', None, 50)": {'add': [861]}}}, {'path': 'pandas/core/indexing.py', 'status': 'modified', 'Loc': {"('_NDFrameIndexer', '_getitem_iterable', 412)": {'mod': [461, 462]}, "('_NDFrameIndexer', '_convert_to_indexer', 464)": {'mod': [572, 573, 574, 575, 576, 577, 578, 579, 581, 582, 584]}}}, {'path': 'pandas/index.pyx', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [269, 270, 271]}}}, {'path': 'pandas/lib.pyx', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [418]}}}, {'path': 'pandas/tests/test_frame.py', 'status': 'modified', 'Loc': {"('TestDataFrame', '_check_df', 4667)": {'mod': [4671, 4672]}}}, {'path': 'pandas/tests/test_indexing.py', 'status': 'modified', 'Loc': {"('TestIndexing', None, 85)": {'add': [786]}}}]}


good first issue
frontend
severity:low
small effort

Rename and/or properly document the two different `changeAgentState` functions

There are two `changeAgentState` functions that should probably be renamed and properly documented to avoid confusion for the future.

https://github.com/OpenDevin/OpenDevin/blob/01ce1e35b5b40e57d96b15a7fc9bee4eb8f6966d/frontend/src/state/agentSlice.tsx#L10-L12

https://github.com/OpenDevin/OpenDevin/blob/01ce1e35b5b40e57d96b15a7fc9bee4eb8f6966d/frontend/src/services/agentStateService.ts#L7-L18

{'base_commit': 'ce8a11a62f8a126ed54dd0ede51cf2c196ed310d', 'files': [{'path': 'frontend/src/services/observations.ts', 'status': 'modified', 'Loc': {'(None, None, None)': {'mod': [1]}, "(None, 'handleObservationMessage', 10)": {'mod': [28]}}}, {'path': 'frontend/src/state/agentSlice.tsx', 'status': 'modified', 'Loc': {'(None, None, None)': {'mod': [10, 16]}}}]}


Align rotation of input faces for GAN conversions

Currently, the extractor finds a rotation matrix for each face using umeyama so it can generate a faceset with all the faces mostly upright. Unfortunately this rotation matrix isn't stored in the alignments file, only the bbox (of the un-rotated face) and facial alignments. For the GAN model, when it comes time to convert, the faces aren't rotated upright before being fed through the model so I doubt anyone has been able to get good results for faces that aren't completely upright.

I propose we store the rotation matrix in the alignments file during extract, then at conversion, re-apply it to the cropped face to make it upright before feeding through the model. The swapped output face then needs to be rotated in the inverse direction to match it with the frame again. Hopefully this is possible.


item_dropped signal should pass response arg as item_scraped does

I highly use request and response.meta in item_scraped signal handler.
Why item_dropped doesn't pass response argument as well as item_scraper does?

{'base_commit': '2bf09b8a2026b79b11d178d391327035dde9f948', 'files': [{'path': 'docs/topics/signals.rst', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [98], 'mod': [86]}}}, {'path': 'scrapy/core/scraper.py', 'status': 'modified', 'Loc': {"('Scraper', '_itemproc_finished', 198)": {'mod': [208]}}}]}


Hello,
I would like to suggest implementing my paper: Differential Diffusion: Giving Each Pixel Its Strength.
The paper allows a user to edit a picture by a change map that describes how much each region should change.
The editing process is typically guided by textual instructions, although it can also be applied without guidance.
We support both continuous and discrete editing.
Our framework is training and fine tuning free! And has negligible penalty of the inference time.
Our implementation is diffusers-based.
We already tested it on 4 different diffusion models (Kadinsky, DeepFloyd IF, SD, SD XL).
We are confident that the framework can also be ported to other diffusion models, such as SD Turbo, Stable Cascade, and amused.
I notice that you usually stick to white==change convention, which is opposite to the convention we used in the paper.
The paper can be thought of as a generalization to some of the existing techniques.
A black map is just regular txt2img ("0"),
A map of one color (which isn't black) can be thought as img2img,
A map of two colors which one color is white can be thought as inpaint.
And the rest? It's completely new!
In the paper, we suggest some further applications such as soft inpainting and strength visualization.

Site:
https://differential-diffusion.github.io/
Paper:
https://differential-diffusion.github.io/paper.pdf
Repo:
https://github.com/exx8/differential-diffusion

{'base_commit': 'f81dbe26e2e363c28ad043db67b59c11bb33f446', 'files': [{'path': 'comfy/samplers.py', 'status': 'modified', 'Loc': {"('KSamplerX0Inpaint', 'forward', 277)": {'add': [278]}}}, {'path': 'nodes.py', 'status': 'modified', 'Loc': {"(None, 'init_custom_nodes', 1936)": {'add': [1963]}}}]}


ddof for np.std in df.agg changes depending on how given & lambda expression does not work correctly in a list of functions

#### Code Sample, a copy-pastable example if possible

```python
In [31]: import numpy as np

In [32]: import pandas as pd

In [33]: df = pd.DataFrame(np.arange(6).reshape(3, 2), columns=['A', 'B'])

In [34]: df
Out[34]:
   A  B
0  0  1
1  2  3
2  4  5

In [35]: df.agg(np.std)  # Behavior of ddof=0
Out[35]:
A    1.632993
B    1.632993
dtype: float64

In [36]: df.agg([np.std])  # Behavior of ddof=1
Out[36]:
       A    B
std  2.0  2.0

In [37]: # So how to get the ddof=0 behavior when giving a list of functions?

In [39]: df.agg([lambda x: np.std(x)])  # This gives a numerically unexpected result.
Out[39]:
         A        B
  <lambda> <lambda>
0      0.0      0.0
1      0.0      0.0
2      0.0      0.0

In [40]: df.agg([np.mean, lambda x: np.std(x)])  # This gives an error.
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-40-52f4ec4195b5> in <module>()
----> 1 df.agg([np.mean, lambda x: np.std(x)])

/Users/ikeda/.pyenv/versions/anaconda3-4.4.0/lib/python3.6/site-packages/pandas/core/frame.py in aggregate(self, func, axis, *args, **kwargs)
   4740         if axis == 0:
   4741             try:
-> 4742                 result, how = self._aggregate(func, axis=0, *args, **kwargs)
   4743             except TypeError:
   4744                 pass

/Users/ikeda/.pyenv/versions/anaconda3-4.4.0/lib/python3.6/site-packages/pandas/core/base.py in _aggregate(self, arg, *args, **kwargs)
    537             return self._aggregate_multiple_funcs(arg,
    538                                                   _level=_level,
--> 539                                                   _axis=_axis), None
    540         else:
    541             result = None

/Users/ikeda/.pyenv/versions/anaconda3-4.4.0/lib/python3.6/site-packages/pandas/core/base.py in _aggregate_multiple_funcs(self, arg, _level, _axis)
    594         # if we are empty
    595         if not len(results):
--> 596             raise ValueError("no results")
    597
    598         try:

ValueError: no results

```
#### Problem description

When using, e.g., `df.agg`, the `ddof` (degrees of freedom) value for the function `np.std` changes depending on how the function is given (single function or a list of functions), which may be so confusing for many people. I believe the behavior should be unified in some way.

Furthermore, I could not find the way to obtain to the `np.std` result with `ddof=0` by supplying it as one of the members of a list of functions. The `lambda` expression does not work well in a list of functions (this gives numerically unexpected results or even gives errors). This prohibits us to use many useful methods like `df.agg`, `df.apply`, and `df.describe` when we hope the `ddof=0` behavior.

From https://github.com/pandas-dev/pandas/issues/13344, I guess Developers prefer the `ddof=1` behavior in pandas. So the expected behavior should be as below.

#### Expected Output
```
In [35]: df.agg(np.std)  # Behavior of ddof=1
Out[35]:
A    2.0
B    2.0
dtype: float64

In [38]: df.agg([lambda x: np.std(x)])  # To obtain the ddof=0 results
Out[38]:
                     A             B
<lambda>      1.632993      1.632993

In [41]: df.agg([np.mean, lambda x: np.std(x)])
                     A             B
mean          2.0           3.0
<lambda>      1.632993      1.632993
```

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None

pandas: 0.21.0
pytest: 3.0.7
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.25.2
numpy: 1.13.3
scipy: 0.19.0
pyarrow: None
xarray: None
IPython: 5.3.0
sphinx: 1.5.6
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: 1.2.1
tables: 3.3.0
numexpr: 2.6.2
feather: None
matplotlib: 2.0.2
openpyxl: 2.4.7
xlrd: 1.0.0
xlwt: 1.2.0
xlsxwriter: 0.9.6
lxml: 3.7.3
bs4: 4.6.0
html5lib: 0.999
sqlalchemy: 1.1.9
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

</details>

{'base_commit': 'bcc5160b3a5b0fc9c531da194c6bb83619045434', 'files': [{'path': 'pandas/tests/apply/test_frame_apply.py', 'status': 'modified', 'Loc': {"(None, 'test_agg_list_like_func_with_args', 1648)": {'add': [1667]}}}]}


spin docs --no-plot runs the examples

Seen at the EuroScipy sprint

Commands run by spin:
```
$ export SPHINXOPTS=-W -D plot_gallery=0 -j auto
$ cd doc
$ make html
```

Looks like our Makefile does not use SPHINXOPTS the same way as expected:
Probably we have a slightly different way of building the doc

```
❯ make html-noplot -n
sphinx-build -D plot_gallery=0 -b html -d _build/doctrees  -T  . -jauto \
    _build/html/stable
echo
echo "Build finished. The HTML pages are in _build/html/stable."
```


Add support for ONNX-TensorRT conversion for GPT-J6B (and possible bug in rotary embedding)

### Who can help
@patil-suraj

## Information

Model I am using: GPT-J

The problem arises when using:
* [x] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

## Description
I opened this issue for two reasons:
1. This is not strictly a bug report, rather a change that enables converting this model to ONNX and then parsing it using the current TensorRT ONNX parser.
2. Possible implementation bug in GPT-J.

## Details
1. When exporting GPT-J to ONNX using the latest version (v4.16.2), one of the ops that is exported is [SplitToSequence](https://github.com/onnx/onnx/blob/main/docs/Operators.md#SplitToSequence) (along with more Sequence* ops) that is currently not supported in the [TensorRT ONNX parser](https://github.com/onnx/onnx-tensorrt/blob/master/docs/operators.md).
This is entirely due to just 1 line of code that uses `torch.repeat_interleave`. ([relevant line](https://github.com/huggingface/transformers/blob/52d2e6f6e904ef9b75c78716ce77b98196ed837a/src/transformers/models/gptj/modeling_gptj.py#L67))
```
sin, cos = map(lambda t: t[None, offset : x.shape[1] + offset, None, :].repeat_interleave(2, 3), sincos)
```
By replacing `lambda t` with this:
```
lambda t: t.view(-1, 1).repeat(1, 2).view(seq_len, -1)[None, offset : x.shape[1] + offset, None, :]
```
we get the exact same output tensors but now exporting to ONNX doesn't include any Sequence* ops, and TensorRT can parse it successfully.
The suggested function is even faster, although probably not critical in this huge model (benched only on CPU):
```
original: 106 µs ± 20.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
suggested: 32.4 µs ± 6.55 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

2. I was following the implementation in EleutherAI for rotary positional embeddings and I'm trying to understand if this is a bug or I'm simply missing something (would love an explanation if you can spare the time) but there (EleutherAI) they implement this function (rotary positional embedding) using `torch.cat` instead of `torch.repeat_interleave`, as can be seen [here](https://github.com/EleutherAI/gpt-neox/blob/b30afd1d0a1d06220be9b5f2c9c9c1523defba96/megatron/model/positional_embeddings.py#L41).

If I'm not missing something, the EleutherAI version transforms a tensor from
```
[[1,2,3],
 [4,5,6]]
```
to
```
[[1,2,3,1,2,3],
 [4,5,6,4,5,6]]
```
and HF version (using repeat_interleave):
```
[[1,2,3],
 [4,5,6]]
```
to
```
[[1,1,2,2,3,3],
 [4,4,5,5,6,6]]
```
Can anyone confirm the current implementation is indeed correct? Because otherwise `cat` and `repeat_interleave` are very different, and the rest of the implementation doesn't take it into account.

{'base_commit': '147c8166852db64de12b851b8307f44c9e8fe0dd', 'files': [{'path': 'src/transformers/models/gptj/modeling_gptj.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [64]}, "(None, 'apply_rotary_pos_emb', 65)": {'mod': [66]}}}]}


added image sort tool to faceswap

I added image sort tool to faceswap, which very useful to extract one face from various faces

Example original aligned folder:
![fsviewer_2018-03-08_20-08-06](https://user-images.githubusercontent.com/8076202/37161724-ecbfac92-230c-11e8-8346-5d71c471edc7.png)

Sort it by similarity:
`python.exe faceswap\sorttool.py -i %WORKSPACE%\data_src\aligned -by similarity`
result:
![fsviewer_2018-03-08_20-10-27](https://user-images.githubusercontent.com/8076202/37161776-0b94041a-230d-11e8-9841-b77a562f6120.png)

easy delete faces which you dont need:
![fsviewer_2018-03-08_20-12-52](https://user-images.githubusercontent.com/8076202/37161908-5fae29e0-230d-11e8-88c4-9f89632b05f3.png)

Sort by blur:
`python.exe faceswap\sorttool.py -i %WORKSPACE%\data_src\aligned -by blur`

most sharp 00000.png:
![2018-03-08_20-15-42](https://user-images.githubusercontent.com/8076202/37162142-f37376e4-230d-11e8-8eb7-52892f4fd46c.png)

most blurred 00140.png:
![fsviewer_2018-03-08_20-15-51](https://user-images.githubusercontent.com/8076202/37162187-0bc53f48-230e-11e8-8c23-646c72898704.png)

{'base_commit': '5007d8e996cbe6c23dcf2b5792775d8fde104128', 'files': [{'path': 'plugins/PluginLoader.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [0]}, "('PluginLoader', '_import', 20)": {'add': [23]}}}, {'path': 'scripts/convert.py', 'status': 'modified', 'Loc': {"('ConvertImage', 'add_optional_arguments', 24)": {'mod': [43, 44]}}}, {'path': 'scripts/train.py', 'status': 'modified', 'Loc': {"('TrainingProcessor', 'parse_arguments', 25)": {'mod': [75, 76]}}}]}


Bing stopped working

**Bug description**
Yesterday, Bing still worked, but today brings up only:
```
Using Bing provider
0, message='Attempt to decode JSON with unexpected mimetype: text/html; charset=utf-8', url=URL('https://www.bing.com/turing/conversation/create?bundleVersion=1.1381.8')
127.0.0.1 - - [14/Dec/2023 20:22:32] "POST /backend-api/v2/conversation HTTP/1.1" 200 -
```

**Screenshots**
![image](https://github.com/xtekky/gpt4free/assets/11407417/3f562cfb-2596-4f65-ba75-4efec77d0f3e)

**Environment**
- python version: 3.12
- location ( are you in a cloudfare flagged country ) : Ukraine


Create an Extra for Better SSL Support

So right now the SSL connections when you use pyOpenSSL, ndg-httspclient, and pyasn1 are more secure than if you just use the stdlib options. However it's hard to actually remember those three things. It would be cool if requests would add an extra to it's setup.py so that people can install requests with betterssl, something like:

``` python
setup(
    extras_require={
        "betterssl": ["pyOpenSSL", "ndg-httpsclient", "pyasn1"],
    },
)
```

Would make it so people can install requests like `pip install requests[betterssl]` and get all of those dependencies without having to manually track those down. It also means people could depend on `requests[betterssl]` instead of just `requests` in their own setup.py's.

Extra name can of course be bikeshed here :)


[Bug]: Generating using LoRA fails with Runtime Error with `Lora/Networks: use old method` enabled

### Is there an existing issue for this?

- [x] I have searched the existing issues and checked the recent builds/commits

### What happened?

I'm on commit 68f336bd994bed5442ad95bad6b6ad5564a5409a, master HEAD at time of posting.
None of my LORAs seem to be working anymore. Normal prompting works fine, but as soon as I try generating after adding a LORA to my prompt I receive the following:

 `RuntimeError: mat1 and mat2 shapes cannot be multiplied (77x3072 and 768x128)`

I'm not well versed in python nor the inner workings of stable diffusion, so I can't debug this myself effectively.
I don't think providing the LORA file or prompt is necessary, as I can reproduce this with any combination of checkpoints and LORAs which would previously work fine.

### Steps to reproduce the problem

1. Select any SD checkpoint
2. txt2image tab
3. Any combination of prompt and negative prompt, doesnt seem to matter
4. Add a LORA to the prompt, no need to even add the activation token.
5. Any generation settings (for my tests I'm using Euler a, 20 steps, 512x512, CFG 7, no scripts, no hires. fix, no face restore).
6. Generate

### What should have happened?

I would expect the LORA to perform as it did in earlier versions with the same configuration, at the very least, generate an image. I haven't done a bisect, but I tried a commit from a week ago or so and it worked fine there. Every time I pull I delete venv and repositories folders beforehand.

### Version or Commit where the problem happens

68f336bd994bed5442ad95bad6b6ad5564a5409a

### What Python version are you running on ?

Python 3.10.x

### What platforms do you use to access the UI ?

Windows

### What device are you running WebUI on?

Nvidia GPUs (RTX 20 above)

### Cross attention optimization

xformers

### What browsers do you use to access the UI ?

Google Chrome

### Command Line Arguments

```Shell
--xformers --reinstall-xformers --precision full --no-half --skip-torch-cuda-test --opt-split-attention
```

### List of extensions

ddetailer, sd-webui-supermerger, stable-diffusion-webui-dataset-tag-editor, stable-diffusion-webui-wd14-tagger

### Console logs

```Shell
venv "C:\sd\sdwebui\webui\venv\Scripts\Python.exe"
Python 3.10.9 (tags/v3.10.9:1dd9be6, Dec  6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)]
Version: v1.5.1
Commit hash: 68f336bd994bed5442ad95bad6b6ad5564a5409a
Installing xformers
Collecting xformers==0.0.20
  Using cached xformers-0.0.20-cp310-cp310-win_amd64.whl (97.6 MB)
Installing collected packages: xformers
Successfully installed xformers-0.0.20

[notice] A new release of pip available: 22.3.1 -> 23.2
[notice] To update, run: C:\sd\sdwebui\webui\venv\Scripts\python.exe -m pip install --upgrade pip

Launching Web UI with arguments: --xformers --reinstall-xformers --precision full --no-half --skip-torch-cuda-test --opt-split-attention
Check config files...
Done
Loading weights [cb15a7187a] from C:\sd\sdwebui\webui\models\Stable-diffusion\Deliberate-inpainting.safetensors
Creating model from config: C:\sd\sdwebui\webui\configs\v1-inpainting-inference.yaml
LatentInpaintDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.54 M params.
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 84.3s (launcher: 11.6s, import torch: 31.4s, import gradio: 8.7s, setup paths: 10.6s, other imports: 8.2s, opts onchange: 0.4s, setup codeformer: 0.4s, list SD models: 0.3s, load scripts: 11.1s, create ui: 1.0s, gradio launch: 0.5s).
Applying attention optimization: xformers... done.
Model loaded in 10.0s (load weights from disk: 0.8s, create model: 1.2s, apply weights to model: 6.4s, move model to device: 1.5s).
Loading weights [f36b3ca4d1] from C:\sd\sdwebui\webui\models\Stable-diffusion\edgeOfRealism_edgeOfRealismBakedVAE.safetensors
Creating model from config: C:\sd\sdwebui\webui\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Applying attention optimization: xformers... done.
Model loaded in 8.3s (create model: 0.6s, apply weights to model: 5.9s, move model to device: 1.7s).
*** Error completing request
*** Arguments: ('task(yot3zok0bchp1w0)', 'pov of a beautiful asian woman, formal dress, perfect eyes, petite body, in the forest, colorful, yellow leaves, autumn, hair bun, black hair, facing the viewer, bokeh, soft lighting, perfect face, eye contact, brown eyes, <lora:iu_eor_new-000017:1>', 'badhandv4 easynegative ng_deepnegative_v1_75t', [], 20, 0, False, False, 1, 1, 7, -1.0, -1.0, 0, 0, 0, False, 512, 512, False, 0.7, 2, 'Latent', 0, 0, 0, 0, '', '', [], <gradio.routes.Request object at 0x000001E14BCBFBB0>, 0, False, False, 'positive', 'comma', 0, False, False, '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, '', 'None', 30, 4, 0, 0, False, 'None', '<br>', 'None', 30, 4, 0, 0, 4, 0.4, True, 32) {}
    Traceback (most recent call last):
      File "C:\sd\sdwebui\webui\modules\call_queue.py", line 58, in f
        res = list(func(*args, **kwargs))
      File "C:\sd\sdwebui\webui\modules\call_queue.py", line 37, in f
        res = func(*args, **kwargs)
      File "C:\sd\sdwebui\webui\modules\txt2img.py", line 62, in txt2img
        processed = processing.process_images(p)
      File "C:\sd\sdwebui\webui\modules\processing.py", line 677, in process_images
        res = process_images_inner(p)
      File "C:\sd\sdwebui\webui\modules\processing.py", line 783, in process_images_inner
        p.setup_conds()
      File "C:\sd\sdwebui\webui\modules\processing.py", line 1191, in setup_conds
        super().setup_conds()
      File "C:\sd\sdwebui\webui\modules\processing.py", line 364, in setup_conds
        self.uc = self.get_conds_with_caching(prompt_parser.get_learned_conditioning, negative_prompts, self.steps * self.step_multiplier, [self.cached_uc], self.extra_network_data)
      File "C:\sd\sdwebui\webui\modules\processing.py", line 353, in get_conds_with_caching
        cache[1] = function(shared.sd_model, required_prompts, steps)
      File "C:\sd\sdwebui\webui\modules\prompt_parser.py", line 163, in get_learned_conditioning
        conds = model.get_learned_conditioning(texts)
      File "C:\sd\sdwebui\webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 669, in get_learned_conditioning
        c = self.cond_stage_model(c)
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\sdwebui\webui\modules\sd_hijack_clip.py", line 234, in forward
        z = self.process_tokens(tokens, multipliers)
      File "C:\sd\sdwebui\webui\modules\sd_hijack_clip.py", line 271, in process_tokens
        z = self.encode_with_transformers(tokens)
      File "C:\sd\sdwebui\webui\modules\sd_hijack_clip.py", line 324, in encode_with_transformers
        outputs = self.wrapped.transformer(input_ids=tokens, output_hidden_states=-opts.CLIP_stop_at_last_layers)
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 811, in forward
        return self.text_model(
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 721, in forward
        encoder_outputs = self.encoder(
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 650, in forward
        layer_outputs = encoder_layer(
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 389, in forward
        hidden_states = self.mlp(hidden_states)
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 344, in forward
        hidden_states = self.fc1(hidden_states)
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\sdwebui\webui\extensions-builtin\Lora\networks.py", line 357, in network_Linear_forward
        return network_forward(self, input, torch.nn.Linear_forward_before_network)
      File "C:\sd\sdwebui\webui\extensions-builtin\Lora\networks.py", line 345, in network_forward
        y = module.forward(y, input)
      File "C:\sd\sdwebui\webui\extensions-builtin\Lora\network_lora.py", line 84, in forward
        return y + self.up_model(self.down_model(x)) * self.multiplier() * self.calc_scale()
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\sdwebui\webui\extensions-builtin\Lora\networks.py", line 357, in network_Linear_forward
        return network_forward(self, input, torch.nn.Linear_forward_before_network)
      File "C:\sd\sdwebui\webui\extensions-builtin\Lora\networks.py", line 337, in network_forward
        y = original_forward(module, input)
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\torch\nn\modules\linear.py", line 114, in forward
        return F.linear(input, self.weight, self.bias)
    RuntimeError: mat1 and mat2 shapes cannot be multiplied (77x3072 and 768x128)
```

### Additional information

None of the extensions listed are used in the context of this issue

{'base_commit': '458eda13211ac3498485f1e5154d90808fbcfb60', 'files': [{'path': 'extensions-builtin/Lora/networks.py', 'status': 'modified', 'Loc': {"(None, 'network_forward', 338)": {'mod': [360]}}}]}


MinibatchKMeans bad center reallocation causes duplicate centers

For instance have a look at:

  http://scikit-learn.org/dev/auto_examples/cluster/plot_dict_face_patches.html

some of the centroids are duplicated, presumably because of a bug in the bad cluster reallocation heuristic.


Overriding the MailSender class

I'd like to use the built-in email notification service for when a scraper exceeds a certain memory limit (`MEMUSAGE_NOTIFY_MAIL` setting), but it looks like it's not possible to specify the MailSender class to use to send the email. I don't want to use SMTP, I'd like to use a third-party mail sender (e.g. sendgrid).
Is there a way around this?
Thanks


Impossible to open 'CC:/manim/manim_3_feb/media/videos/example_scenes/480p15/partial_movie_files/SquareToCircle/00000.mp4'

![Screenshot_1](https://user-images.githubusercontent.com/65260808/81756436-479cd380-94ab-11ea-89e1-af386563bfcd.png)
Help me solve this

{'base_commit': 'dbdd7996960ba46ed044a773290b02f17478c760', 'files': [{'path': 'manimlib/scene/scene_file_writer.py', 'status': 'modified', 'Loc': {"('SceneFileWriter', 'combine_movie_files', 253)": {'mod': [289]}}}]}


Allow detecting Dev Drive on Windows

Windows just announced a new [Dev Drive](https://learn.microsoft.com/en-us/windows/dev-drive/) feature, optimised for high I/O scenarios such as build and test. It also works as a very clear signal that the user is a developer and is doing developer-like tasks.

We should add a function to allow querying whether a specific path is on a Dev Drive. The API is relatively low level, and cannot currently be used from Python, but would allow Python apps to detect when the user is operating on a Dev Drive (e.g. installing or compiling something on one), or choose or offer a more performant temporary or cache location than the user directory.

(For a variety of mostly compatibility reasons, there's no way for Windows to redirect `%TEMP%` onto a Dev Drive, but apps that are aware of it can do it for themselves.)

<!-- gh-linked-prs -->
### Linked PRs
* gh-104805
* gh-105054
<!-- /gh-linked-prs -->

{'base_commit': 'ae00b810d1d3ad7f1f7e226b02ece37c986330e7', 'files': [{'path': 'Doc/library/os.path.rst', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [306]}}}, {'path': 'Lib/ntpath.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [869]}}}, {'path': 'Lib/test/test_ntpath.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [994]}}}, {'path': 'Modules/clinic/posixmodule.c.h', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [1717, 11381], 'mod': [11925]}}}, {'path': 'Modules/posixmodule.c', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [4532, 15799]}}}]}


Document Reppy Python version support

The optional dependency on reppy for one of the built-in robots.txt parsers is [preventing us from running the extra-dependencies CI job with Python 3.9+](https://github.com/seomoz/reppy/issues/122). https://github.com/seomoz/reppy has not have a commit for ~1.5 years.

So I think we should deprecate the component.

If we don’t, we should document this limitation, and schedule a deprecation for 1 year before Python 3.8 reaches end of life, ~~i.e. in 9 months~~, because once we drop Python 3.8 support we will be forced to remove this component anyway, so giving a deprecation warning 1 year before is probably in the best interest of any user of the component.

{'base_commit': '2814e0e1972fa38151b6800c881d49f50edf9c6b', 'files': [{'path': 'docs/topics/downloader-middleware.rst', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [1072]}}}]}


Requests doesn't catch requests.packages.urllib3.exceptions.ProxyError and translate it into a requests module specific exception which derives from RequestException as it does for other errors originating from urllib3. This means if trying to catch any exception derived from RequestException so as to treat it specially, the urllib3 ProxyError will be missed.

{'base_commit': '9968a10fcfad7268b552808c4f8946eecafc956a', 'files': [{'path': 'requests/adapters.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [24], 'mod': [26]}, "('HTTPAdapter', 'send', 283)": {'add': [355]}}}, {'path': 'requests/exceptions.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [29]}}}]}


Outpainting script does not save multiple images when using batch sliders

When using the batch-count slider and the batch-size slider, the outpainting script does not save multiple images, but just the first one.

Looking at the console window we can see the actual processing is happening for all the N images (batch-count * batch-size), but at the end of the process only the first one is saved to disk.


Add numpy doc strings

Add numpy style doc strings to all functions apart from the main.py file.

<details>
<summary>Checklist</summary>

- [X] `gpt_engineer/ai.py`
> • For each function in this file, add or replace the existing docstring with a numpy-style docstring. The docstring should include a brief description of the function, a list of parameters with their types and descriptions, and a description of the return value.

- [X] `gpt_engineer/chat_to_files.py`
> • For each function in this file, add or replace the existing docstring with a numpy-style docstring. The docstring should include a brief description of the function, a list of parameters with their types and descriptions, and a description of the return value.

- [X] `gpt_engineer/collect.py`
> • For each function in this file, add or replace the existing docstring with a numpy-style docstring. The docstring should include a brief description of the function, a list of parameters with their types and descriptions, and a description of the return value.

- [X] `gpt_engineer/db.py`
> • For each function in this file, add or replace the existing docstring with a numpy-style docstring. The docstring should include a brief description of the function, a list of parameters with their types and descriptions, and a description of the return value.

- [X] `gpt_engineer/learning.py`
> • For each function in this file, add or replace the existing docstring with a numpy-style docstring. The docstring should include a brief description of the function, a list of parameters with their types and descriptions, and a description of the return value.

</details>


Allow user to open the preview website in a new window


Need suggestion on contributing TFDPR

# 🌟 New model addition

## Model description
Hi, I would love to try contributing TFDPR . This is the first time to me, so I need some suggestions.
I have followed @sshleifer 's [great PR on TFBart model](https://github.com/huggingface/transformers/commit/829842159efeb1f920cbbb1daf5ad67e0114d0b9) on 4 files :` __init__.py , convert_pytorch_checkpoint_to_tf2.py , utils/dummy_tf_objects.py` and (newly created) `modeling_tf_dpr.py `

Now the TF model works properly and can load Pytorch's weights successfully the same output as Pytorch's counterparts **except** small random noise (1e-5) which I suspect of some dtypes different , but I could not find the cause.

I guess I need to add document on  docs/source/model_doc/dpr.rst , and that's all ?
**My question is do I need to change / fix any other files ? and/or do I need to do some other thing before making PR ?**

<!-- Important information -->
To resolve TF vs. Pytorch naming issues, there's one change regarding `TFBertModel` vs. `TFBertMainLayer` as [discussed here](https://discuss.huggingface.co/t/solved-issue-on-translating-dpr-to-tfdpr-on-loading-pytorch-weights-to-tf-model/1764) .
Thanks to @sshleifer for his help to solve the issue.

## Open source status

* [X] the model implementation is available: (give details)
You can see all the modified codes with test run at :
https://colab.research.google.com/drive/1lU4fx7zkr-Y3CXa3wmHIY8yJhKdiN3DI?usp=sharing
(to easily navigate the changes, please “find on page” for e.g. `TFDPRContextEncoder` )

* [X] the model weights are available: (give details)
At the moment, I use existing Pytorch weights, but will upload TF weights too.

* [X] who are the authors: (mention them, if possible by @gh-username)
@ratthachat

{'base_commit': 'eb3bd73ce35bfef56eeb722d697f2d39a06a8f8d', 'files': [{'path': 'docs/source/model_doc/dpr.rst', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [101]}}}, {'path': 'src/transformers/__init__.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [408, 715]}}}, {'path': 'src/transformers/convert_pytorch_checkpoint_to_tf2.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [27, 45, 61, 100, 149]}}}, {'path': 'src/transformers/modeling_tf_auto.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [45, 89, 194]}}}, {'path': 'src/transformers/utils/dummy_pt_objects.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [737]}}}, {'path': 'src/transformers/utils/dummy_tf_objects.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [497]}}}, {'path': 'tests/test_modeling_dpr.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [26]}, "('DPRModelTest', 'test_model_from_pretrained', 214)": {'add': [229]}}}, {'path': 'utils/check_repo.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [35, 59, 89]}}}]}


[mask2former] torch.export error for Mask2Former

### System Info

- `transformers` version: 4.46.0.dev0
- Platform: Linux-6.8.0-47-generic-x86_64-with-glibc2.35
- Python version: 3.11.9
- Huggingface_hub version: 0.25.2
- Safetensors version: 0.4.5
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.4.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA GeForce RTX 4090

### Who can help?

@amyeroberts, @qubvel, @ylacombe

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
import torch
from transformers import Mask2FormerForUniversalSegmentation

model = Mask2FormerForUniversalSegmentation.from_pretrained(
    "facebook/mask2former-swin-base-coco-panoptic", torchscript=True
)

scripted_model = torch.export.export(model, args=(torch.randn(1, 3, 800, 1280),))
```
which causes
```
UserError: Could not extract specialized integer from data-dependent expression u0 (unhinted: u0).  (Size-like symbols: none)

Potential framework code culprit (scroll up for full backtrace):
  File "/home/philkuz/.pyenv/versions/3.11.9/envs/gml311/lib/python3.11/site-packages/torch/_dynamo/utils.py", line 2132, in run_node
    return node.target(*args, **kwargs)

For more information, run with TORCH_LOGS="dynamic"
For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0"
If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1
For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing

User Stack (most recent call last):
  (snipped, see stack below for prefix)
  File "/home/philkuz/dev/transformers/src/transformers/models/mask2former/modeling_mask2former.py", line 2499, in forward
    outputs = self.model(
  File "/home/philkuz/.pyenv/versions/3.11.9/envs/gml311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philkuz/dev/transformers/src/transformers/models/mask2former/modeling_mask2former.py", line 2270, in forward
    pixel_level_module_output = self.pixel_level_module(
  File "/home/philkuz/.pyenv/versions/3.11.9/envs/gml311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philkuz/dev/transformers/src/transformers/models/mask2former/modeling_mask2former.py", line 1395, in forward
    decoder_output = self.decoder(backbone_features, output_hidden_states=output_hidden_states)
  File "/home/philkuz/.pyenv/versions/3.11.9/envs/gml311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philkuz/dev/transformers/src/transformers/models/mask2former/modeling_mask2former.py", line 1319, in forward
    encoder_outputs = self.encoder(
  File "/home/philkuz/.pyenv/versions/3.11.9/envs/gml311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philkuz/dev/transformers/src/transformers/models/mask2former/modeling_mask2former.py", line 1165, in forward
    reference_points = self.get_reference_points(spatial_shapes, valid_ratios, device=inputs_embeds.device)
  File "/home/philkuz/dev/transformers/src/transformers/models/mask2former/modeling_mask2former.py", line 1106, in get_reference_points
    torch.linspace(0.5, height - 0.5, height, dtype=valid_ratios.dtype, device=device),

For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1
For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example

from user code:
   File "/home/philkuz/dev/transformers/src/transformers/models/mask2former/modeling_mask2former.py", line 2499, in forward
    outputs = self.model(
  File "/home/philkuz/.pyenv/versions/3.11.9/envs/gml311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philkuz/dev/transformers/src/transformers/models/mask2former/modeling_mask2former.py", line 2270, in forward
    pixel_level_module_output = self.pixel_level_module(
  File "/home/philkuz/.pyenv/versions/3.11.9/envs/gml311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philkuz/dev/transformers/src/transformers/models/mask2former/modeling_mask2former.py", line 1395, in forward
    decoder_output = self.decoder(backbone_features, output_hidden_states=output_hidden_states)
  File "/home/philkuz/.pyenv/versions/3.11.9/envs/gml311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philkuz/dev/transformers/src/transformers/models/mask2former/modeling_mask2former.py", line 1319, in forward
    encoder_outputs = self.encoder(
  File "/home/philkuz/.pyenv/versions/3.11.9/envs/gml311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philkuz/dev/transformers/src/transformers/models/mask2former/modeling_mask2former.py", line 1165, in forward
    reference_points = self.get_reference_points(spatial_shapes, valid_ratios, device=inputs_embeds.device)
  File "/home/philkuz/dev/transformers/src/transformers/models/mask2former/modeling_mask2former.py", line 1106, in get_reference_points
    torch.linspace(0.5, height - 0.5, height, dtype=valid_ratios.dtype, device=device),
 ```

### Expected behavior

torch.export works for this model.

{'base_commit': '9bee9ff5db6e68fb31065898d7e924d07c1eb9c1', 'files': [{'path': 'src/transformers/models/mask2former/modeling_mask2former.py', 'status': 'modified', 'Loc': {"('Mask2FormerPixelDecoder', 'forward', 1280)": {'add': [1333], 'mod': [1305, 1307, 1323, 1337, 1339, 1341, 1345]}, "('Mask2FormerPixelDecoderEncoderMultiscaleDeformableAttention', 'forward', 921)": {'mod': [929, 939, 960, 973]}, "('Mask2FormerPixelDecoderEncoderLayer', 'forward', 998)": {'mod': [1004, 1018, 1019, 1036]}, "('Mask2FormerPixelDecoderEncoderOnly', None, 1069)": {'mod': [1089]}, "('Mask2FormerPixelDecoderEncoderOnly', 'get_reference_points', 1089)": {'mod': [1094, 1095, 1104]}, "('Mask2FormerPixelDecoderEncoderOnly', 'forward', 1120)": {'mod': [1125, 1143, 1144, 1165, 1179]}, "('Mask2FormerMaskedAttentionDecoder', 'forward', 1792)": {'mod': [1879]}}}, {'path': 'tests/models/mask2former/test_modeling_mask2former.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [22]}, "('Mask2FormerModelIntegrationTest', 'test_with_segmentation_maps_and_loss', 466)": {'add': [483]}}}]}


scrapy selector fails when large lines are present response

Originally encoutered when scraping [Amazon restaurant](https://www.amazon.com/restaurants/zzzuszimbos0015gammaloc1name-new-york/d/B01HH7CS44?ref_=amzrst_pnr_cp_b_B01HH7CS44_438).
This page contains multiple script tag with lines greater then 64,000 character in one line.
The selector (xpath and css) does not search beyond these lines.

Due to this the following xpath `'//h1[contains(@class, "hw-dp-restaurant-name")]/text()'` to extract name of the restaurant returns empty even though there is a matching tag is present.

PFA the response text at [original_response.html.txt.gz](https://github.com/scrapy/scrapy/files/1631425/original_response.html.txt.gz)

{'base_commit': '19d0942c74731d797a3590b1d8d46ece5a6d751f', 'files': [{'path': 'docs/contributing.rst', 'status': 'modified', 'Loc': {'(None, None, None)': {'mod': [76]}}}, {'path': 'scrapy/tests/test_utils_url.py', 'status': 'modified', 'Loc': {"('UrlUtilsTest', None, 8)": {'add': [50]}, '(None, None, None)': {'mod': [3, 4]}, "('MySpider', 'test_url_is_from_spider_with_allowed_domains_class_attributes', 52)": {'mod': [54]}}}, {'path': 'scrapy/utils/url.py', 'status': 'modified', 'Loc': {"(None, 'url_is_from_spider', 25)": {'mod': [27, 28]}, "(None, 'canonicalize_url', 33)": {'mod': [33]}}}]}


TST/CLN: remove TestData from frame-tests; replace with fixtures

Following review in #22236:
> ok, pls open a new issue that refs this, to remove use of `TestData` in favor of fixtures

Started the process in that PR by creating a `conftest.py` that translates all the current attributes of `TestData` to fixtures, with the following "translation guide":

* `frame` -> `float_frame`
* `frame2` -> `float_frame2`
* `intframe` -> `int_frame`
* `tsframe` -> `datetime_frame`
* `mixed_frame` -> `float_string_frame`
* `mixed_float` -> `mixed_float_frame`
* `mixed_float2` -> `mixed_float_frame2`
* `mixed_int` -> `mixed_int_frame`
* `all_mixed` -> `mixed_type_frame`
* `tzframe` -> `timezone_frame`
* `empty` -> `empty_frame`
* `ts1` -> `datetime_series`
* `ts2` -> `datetime_series_short`
* `simple` -> `simple_frame`

Need to incrementally replace their usages in `pandas/tests/frame/` (example below).

- [x] Create `conftest.py` and translate `TestData`-attributes into fixtures (#22236)
- [x] `test_alter_axes.py` (#22236)
- [x] `test_analytics.py` (#22733)
- [x] `test_api.py` (#22738)
- [x] `test_apply.py` (#22735)
- [x] `test_arithmetic.py` (#22736)
- [x] `test_asof.py` (#25628)
- [x] `test_axis_select_reindex.py` (#25627)
- [x] `test_block_internals.py` (#22926)
- [x] `test_combine_concat.py` (#25634)
- [ ] `test_constructors.py` (#25635)
- [ ] `test_convert_to.py`
- [ ] `test_dtypes.py` (#25636)
- [x] `test_duplicates.py`
- [x] `test_indexing.py` (#25633)
- [x] `test_join.py` (#25639)
- [x] `test_missing.py` (#25640)
- [x] `test_mutate_columns.py` (#25642)
- [ ] `test_nonunique_indexes.py`
- [x] `test_operators.py` (#25641)
- [ ] `test_period.py`
- [ ] `test_quantile.py`
- [ ] `test_query_eval.py`
- [ ] `test_rank.py`
- [ ] `test_replace.py`
- [ ] `test_repr_info.py`
- [ ] `test_reshape.py`
- [ ] `test_sort_values_level_as_str.py`
- [ ] `test_sorting.py`
- [ ] `test_subclass.py`
- [ ] `test_timeseries.py`
- [ ] `test_timezones.py`
- [ ] `test_to_csv.py`
- [ ] `test_validate.py`

Things for follow-ups:
- Remove other class-based test-methods
- Turn tests from class- to function-based

An example from #22236 - before:
```
def test_set_columns(self):
    cols = Index(np.arange(len(self.mixed_frame.columns)))
    self.mixed_frame.columns = cols
    with tm.assert_raises_regex(ValueError, 'Length mismatch'):
        self.mixed_frame.columns = cols[::2]
```
After:
```
def test_set_columns(self, float_string_frame):
    cols = Index(np.arange(len(float_string_frame.columns)))
    float_string_frame.columns = cols
    with tm.assert_raises_regex(ValueError, 'Length mismatch'):
        float_string_frame.columns = cols[::2]
```

Basically, it comes down to replacing all the occurrences of `self.<name>` with `translation_guide[<name>]` (and specifying`<name>` as a parameter to the function).

PS. Note that some fixtures added by #22236 have now been removed by #24885. Please check #24885 which code was removed, in case you should need it for the fixturisation. Alternatively, you can ping me, @jbrockmendel or @jreback.


API Rate Limit Reached with new key

I just create a new key and it's failing to run:
```
Continue (y/n): y
Error:  API Rate Limit Reached. Waiting 10 seconds...
Error:  API Rate Limit Reached. Waiting 10 seconds...
Error:  API Rate Limit Reached. Waiting 10 seconds...
Error:  API Rate Limit Reached. Waiting 10 seconds...
Error:  API Rate Limit Reached. Waiting 10 seconds...
Error:  API Rate Limit Reached. Waiting 10 seconds...
Error:  API Rate Limit Reached. Waiting 10 seconds...
Error:  API Rate Limit Reached. Waiting 10 seconds...
```

![image](https://user-images.githubusercontent.com/1215497/229545231-7b463bc9-4630-45d5-a8cc-41df10e4e4be.png)


Support for ChatGLM

**Description**

[ChatGLM-6B](https://github.com/THUDM/ChatGLM-6B)

A Chinese chat AI based on GLM was released by THU.

{'base_commit': '6a03ad082492268d60fa23ba5f3dcebd1630593e', 'files': [{'path': 'README.md', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [221]}}}, {'path': 'download-model.py', 'status': 'modified', 'Loc': {"(None, 'get_download_links_from_huggingface', 82)": {'mod': [111]}}}, {'path': 'models/config.yaml', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [47]}}}, {'path': 'modules/chat.py', 'status': 'modified', 'Loc': {"(None, 'generate_chat_prompt', 21)": {'mod': [52, 63]}}}, {'path': 'modules/models.py', 'status': 'modified', 'Loc': {"(None, 'load_model', 41)": {'add': [46, 122], 'mod': [50, 82, 159, 168, 188]}, '(None, None, None)': {'mod': [13, 14]}}}, {'path': 'modules/shared.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [115, 164]}}}]}


StratifiedKFold should do its best to preserve the dataset dependency structure

As highlighted in this [notebook](http://nbviewer.ipython.org/urls/raw.github.com/ogrisel/notebooks/master/Non%2520IID%2520cross-validation.ipynb) the current implementation of `StratifiedKFold` (which is used by default by `cross_val_score` and `GridSearchCV` for classification problems) breaks the dependency structure of the dataset by computing the folds based on the sorted labels.

Instead one should probably do an implementation that performs individual dependency preserving KFold on for each possible label value and aggregate the folds to get the `StratifiedKFold` final folds.

This might incur a refactoring to get rid of the `_BaseKFold` base class. It might also make it easier to implement a `shuffle=True` option for `StratifiedKFold`.