Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ValueError: chroma_server_nofile cannot be set to a value greater than the current hard limit of 8192 #1404

Closed
pangpang97 opened this issue Nov 16, 2023 · 4 comments · Fixed by #1409
Closed

ValueError: chroma_server_nofile cannot be set to a value greater than the current hard limit of 8192 #1404

pangpang97 opened this issue Nov 16, 2023 · 4 comments · Fixed by #1409
Assignees
@beggers

Comments

@pangpang97
Copy link

Hello ，i am new to use chroma. And I want to run chroma in server mode. So when I run the code : chroma run --path /home/xxxx/kb/chroma_kb/doc_db/, I got a error: "ValueError: chroma_server_nofile cannot be set to a value greater than the current hard limit of 8192".

So what should I do to fix that error? Thanks.
my chroma version is : chromadb : 0.4.17 and chroma-hnswlib: 0.7.3. My os system is centos.
@HammadB
Copy link
Collaborator

HammadB commented Nov 16, 2023

You can either raise the hard limit or decrease the requested limit. We should probably log instead of failing. I'll issue a bug. In the meantime you can do

export CHROMA_SERVER_NOFILE = 8192

@beggers beggers self-assigned this Nov 16, 2023
@beggers
Copy link
Contributor

beggers commented Nov 17, 2023

I learned some interesting things about my machine repro-ing this[1]. I was able to repro by setting CHROMA_SERVER_NOFILE to 9223372036854775808[2].

[1]

$ ulimit -nH
unlimited

$ sysctl -A |grep maxfile
kern.maxfiles: 6553
kern.maxfilesperproc: 256

$ python
>>> import resource
>>> resource.getrlimit(resource.RLIMIT_NOFILE)
(256, 9223372036854775807)

[2] 9223372036854775808 is just the number from resource.getrlimit(resource.RLIMIT_NOFILE) + 1.

@beggers beggers mentioned this issue Nov 17, 2023
Merged
2 tasks
beggers added a commit that referenced this issue Nov 17, 2023
@beggers @HammadB
[CLN] Warn when chroma_server_nofile is higher than sys hard limit (#…
c64cdc9 
…1409)

## Description of changes

*Summarize the changes made by this PR.*
 - Improvements & Bug fixes
	 - Fixes #1404

## Test plan
*How are these changes tested?*

- [x] Tests pass locally with `pytest` for python, `yarn test` for js
- [x] Local repro and fix

This is one contiguous terminal session, broken into parts so shell and
python syntax highlighting both look nice:

```sh
(.venv) 0 beggers % git checkout main
M	chromadb/cli/cli.py
Already on 'main'
Your branch is up to date with 'origin/main'.
(.venv) 0 beggers % export CHROMA_SERVER_NOFILE=9223372036854775808                                                      ~/chroma/chroma
(.venv) 0 beggers % python
```
```python
Python 3.11.6 (main, Oct  2 2023, 20:46:14) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import chromadb
>>> c = chromadb.Client()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/beggers/chroma/chroma/chromadb/__init__.py", line 197, in Client
    return ClientCreator(tenant=tenant, database=database, settings=settings)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/beggers/chroma/chroma/chromadb/api/client.py", line 138, in __init__
    super().__init__(settings=settings)
  File "/Users/beggers/chroma/chroma/chromadb/api/client.py", line 41, in __init__
    SharedSystemClient._create_system_if_not_exists(self._identifier, settings)
  File "/Users/beggers/chroma/chroma/chromadb/api/client.py", line 48, in _create_system_if_not_exists
    new_system = System(settings)
                 ^^^^^^^^^^^^^^^^
  File "/Users/beggers/chroma/chroma/chromadb/config.py", line 322, in __init__
    raise ValueError(
ValueError: chroma_server_nofile cannot be set to a value greater than the current hard limit of 9223372036854775807
>>> ^D
```
```sh
(.venv) 0 beggers % git checkout beggers/nofile-limit 
M	chromadb/cli/cli.py
Switched to branch 'beggers/nofile-limit'
Your branch is up to date with 'origin/beggers/nofile-limit'.
(.venv) 0 beggers % python 
```
```python
Python 3.11.6 (main, Oct  2 2023, 20:46:14) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import chromadb
>>> c = chromadb.Client()
WARNING:root:chroma_server_nofile cannot be set to a value greater than the current hard limit of 9223372036854775807
>>> 
```

## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*

---------

Co-authored-by: Hammad Bashir <[email protected]>
@jeanfredd
Copy link

jeanfredd commented May 15, 2024 •
edited
Loading

I'm not having an error but a warning when i launch the server :
chroma_server_nofile cannot be set to a value greater than the current hard limit of 4096. Keeping soft limit at 1024.
I'm quite new in chromadb server (i used the langchain application until today), the database behaviour seems not to be affected by this warning compared to the previous usage with langchain, do you have an explanation of this warning message ? Could it mean that a subpart of the database is forgotten ?

@tazarov
Copy link
Contributor

tazarov commented May 15, 2024

hey @jeanfredd, as you point out this is just a warning message indicating that the requested file descriptor limit is not possible and to what value it would be set.

By default the docker entrypoint will set this to 65535:

chroma/bin/docker_entrypoint.sh

Line 5 in 0c4f7ca

export CHROMA_SERVER_NOFILE=65535

However, the configuration logic in Chroma will check that limit against the actual system limits in your case:

  • The soft limit is 1024
  • The hard limit is 4096

chroma/chromadb/config.py

Line 353 in 0c4f7ca

f"chroma_server_nofile cannot be set to a value greater than the current hard limit of {curr_hard}. Keeping soft limit at {curr_soft}"

For testing and experimentation, this is fine, but for production use, you may want to adjust the limits (kernel params in /etc/security/limits.conf) to ensure Chroma doesn't run out of file descriptors.

tazarov added a commit that referenced this issue May 15, 2024
@tazarov
bug: CHROMA_SERVER_NOFILE cannot be overriden with ENV var in container
7db70a9 
Refs: #1404
@tazarov tazarov mentioned this issue May 15, 2024
Merged
tazarov added a commit that referenced this issue Jul 16, 2024
@tazarov
bug: CHROMA_SERVER_NOFILE cannot be overriden with ENV var in container
23034d0 
Refs: #1404
tazarov added a commit that referenced this issue Jul 16, 2024
@tazarov
[BUG] CHROMA_SERVER_NOFILE cannot be overriden with env var in contai…
b148bf0 
…ner (#2209)

## Description of changes

Ref: #1404

*Summarize the changes made by this PR.*
 - Improvements & Bug fixes
- CHROMA_SERVER_NOFILE is not correctly overriden by user env var
provided in docker compose or .env file; defaults to 65536 otherwise.

## Test plan
*How are these changes tested?*

Manually tested

## Documentation Changes
N/A
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@beggers beggers

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[CLN] Warn when chroma_server_nofile is higher than sys hard limit chroma-core/chroma
5 participants
@tazarov @HammadB @pangpang97 @beggers @jeanfredd