A single file implementation of key-value database for Python 3.
I use this to store millions of image metadata in many machine learning projects.
SFDB is especially useful when I want to store millions of Stable Diffusion prompts. Its effectiveness is also validated in my many other projects.
Just copy the single file "sfdb.py" to any project.
No installation. No dependencies.
The only one python file has less than 150 lines of codes and only uses Python 3 standard libraries.
import sfdb
# Will create a new file or open an existing file.
db = sfdb.Database(filename='test.db')
# Add or update items
db['hello'] = 1
db['123'] = 'hi'
db['a'] = [1, 2, 3]
db['bad'] = 'garbage'
# Delete items
del db['bad']
# Read items
print(db['123']) # Will print "hi".
print(len(db)) # Will print "3".Your data is stored with sqlite format, so that you can view all your data with any sqlite viewers like SQLiteStudio or Jetbrain Datagrip.
All items are human-readable json text.
You can access all data even without using SFDB.
View (and even edit) your data anytime outside your project.
This is what you see in your file explorer:
If your database is small, you can just read everything to memory in various format with one line of code.
import sfdb
db = sfdb.Database(filename='test.db')
cache = db.tolist()
print(cache)
# [('hello', 1), ('123', 'hi'), ('a', [1, 2, 3])]
cache = db.todict()
print(cache)
# {'hello': 1, '123': 'hi', 'a': [1, 2, 3]}
cache = db.keys()
print(cache)
# ['123', 'a', 'hello']You can process any large data without loading everything to your memory.
import sfdb
# Oh god this database has 10 TB data.
db = sfdb.Database(filename='very_large_database_with_10TB.db')
# Update it without loading database to memory.
db['anything'] = 123456
# Search item without loading database to memory.
if 'another_thing' in db:
print('Cool!')
# Get item
print(db['another_thing'])
# Try to get item with default value as None if item not found.
print(db.get('another_thing', default=None))
for key, value in db:
# Read data items one-by-one.
# This only requires very small memory.
print(key)
print(value)Everything is thread-safe.
Do anything you want to do.
Your data are safe.
All data are valid if you only write valid data.
Quit you application with Ctrl+C does not damage the integrity of database structure.
A reference is here:
An SQLite database is highly resistant to corruption. If an application crash,
or an operating-system crash, or even a power failure occurs in the middle of a
transaction, the partially written transaction should be automatically rolled
back the next time the database file is accessed. The recovery process is fully
automatic and does not require any action on the part of the user or the application.
The hard disk writing is optimized to speed up the processing and protect your SSD/HDD drive.
All data updating are updated immediately from the perspective of your python program (i.e., your code logic), but the actual writing to the hard disk only happens when
(1) your program quit/end, OR
(2) every 16384 (16 * 1024) updates, OR
(3) every 60 seconds.
You can force the current updates to be written to the hard disk with
db.commit() # But you do not need to do it.but you do NOT need to do so, since it is slow, and we have already optimized it. Or you can use
db.close() # Close a database when you quit (but you do not need to do it).or
del db # Dispose the instance when you quit (but you do not need to do it).If you want to look professional, you can use "with" context manager like
import sfdb
with sfdb.Database(filename='test.db') as db:
print(db['hello'])Or if you like "tqdm" you can use
import sfdb
from tqdm import tqdm
db = sfdb.Database(filename='test.db')
for key, value in tqdm(db):
# You will have a nice progress bar provided by tqdm.
print(key)
print(value)CC-By 4.0 - Do whatever you want to do.