dulwich.object_store.PackBasedObjectStore

class documentation

class PackBasedObjectStore(PackCapableObjectStore, PackedObjectContainer):

Known subclasses: dulwich.object_store.BucketBasedObjectStore, dulwich.object_store.DiskObjectStore

Constructor: PackBasedObjectStore(pack_compression_level, pack_index_version, pack_delta_window_size, pack_window_memory, ...)

View In Hierarchy

Object store that uses pack files for storage.

This class provides a base implementation for object stores that use Git pack files as their primary storage mechanism. It handles caching of open pack files and provides configuration for pack file operations.

Method	`__contains__`	Check if a particular object is present by SHA1.
Method	`__del__`	Warn if the object store is being deleted with unclosed packs.
Method	`__init__`	Initialize a PackBasedObjectStore.
Method	`__iter__`	Iterate over the SHAs that are present in this store.
Method	`add_objects`	Add a set of objects to this object store.
Method	`add_pack`	Add a new pack to this object store.
Method	`add_pack_data`	Add pack data to this object store.
Method	`close`	Close the object store and release resources.
Method	`contains_loose`	Check if a particular object is present by SHA1 and is loose.
Method	`contains_packed`	Check if a particular object is present by SHA1 and is packed.
Method	`count_pack_files`	Count the number of pack files.
Method	`delete_loose_object`	Delete a loose object.
Method	`generate_pack_bitmaps`	Generate bitmap indexes for all packs that don't have them.
Method	`generate_pack_data`	Generate pack data objects for a set of wants/haves.
Method	`get_raw`	Obtain the raw fulltext for an object.
Method	`get_reachability_provider`	Get the best reachability provider for the object store.
Method	`get_unpacked_object`	Obtain the unpacked object.
Method	`iter_unpacked_subset`	Iterate over a subset of objects, yielding UnpackedObject instances.
Method	`iterobjects_subset`	Iterate over a subset of objects in the store.
Method	`pack_loose_objects`	Pack loose objects.
Method	`repack`	Repack the packs in this repository.
Instance Variable	`delta_base_cache_limit`	Undocumented
Instance Variable	`pack_big_file_threshold`	Undocumented
Instance Variable	`pack_compression_level`	Undocumented
Instance Variable	`pack_delta_cache_size`	Undocumented
Instance Variable	`pack_delta_window_size`	Undocumented
Instance Variable	`pack_depth`	Undocumented
Instance Variable	`pack_index_version`	Undocumented
Instance Variable	`pack_threads`	Undocumented
Instance Variable	`pack_window_memory`	Undocumented
Instance Variable	`packed_git_limit`	Undocumented
Property	`alternates`	Return list of alternate object stores.
Property	`packs`	List with pack objects.
Method	`_add_cached_pack`	Add a newly appeared pack to the cache by path.
Method	`_clear_cached_packs`	Undocumented
Method	`_enforce_packed_git_limit`	Evict least-recently-used packs if the memory limit is exceeded.
Method	`_get_loose_object`	Undocumented
Method	`_iter_alternate_objects`	Iterate over the SHAs of all the objects in alternate stores.
Method	`_iter_cached_packs`	Undocumented
Method	`_iter_loose_objects`	Iterate over the SHAs of all loose objects.
Method	`_mark_pack_used`	Mark a pack as recently used for LRU tracking.
Method	`_remove_pack`	Undocumented
Method	`_total_pack_mmap_size`	Return the total mmapped memory across all cached packs.
Method	`_update_pack_cache`	Undocumented
Instance Variable	`_pack_access_order`	Undocumented
Instance Variable	`_pack_cache`	Undocumented

Inherited from BaseObjectStore (via PackCapableObjectStore):

Method	`__getitem__`	Obtain an object by SHA1.
Method	`add_object`	Add a single object to this object store.
Method	`determine_wants_all`	Determine which objects are wanted based on refs.
Method	`find_common_revisions`	Find which revisions this store has in common using graphwalker.
Method	`find_missing_objects`	Find the missing objects required for a set of revisions.
Method	`get_commit_graph`	Get the commit graph for this object store.
Method	`get_object_mtime`	Get the modification time of an object.
Method	`iter_prefix`	Iterate over all SHA1s that start with a given prefix.
Method	`iter_tree_contents`	Iterate the contents of a tree and all subtrees.
Method	`peel_sha`	Peel all tags from a SHA.
Method	`prune`	Prune/clean up this object store.
Method	`tree_changes`	Find the differences between the contents of two trees.
Method	`write_commit_graph`	Write a commit graph file for this object store.
Instance Variable	`object_format`	Undocumented
Method	`_get_depth`	Return the current available depth for the given head.

def __contains__(self, sha: ObjectID | RawObjectID) -> bool: ¶

overrides dulwich.object_store.BaseObjectStore.__contains__

Check if a particular object is present by SHA1.

This method makes no distinction between loose and packed objects.

def __del__(self): ¶

Warn if the object store is being deleted with unclosed packs.

overrides dulwich.object_store.BaseObjectStore.__init__

overridden in dulwich.cloud.gcs.GcsObjectStore, dulwich.object_store.DiskObjectStore

Initialize a PackBasedObjectStore.

Parameters
pack_compression_level:`int`	Compression level for pack files (-1 to 9)
pack_index_version:`int \| None`	Pack index version to use
pack_delta_window_size:`int \| None`	Window size for delta compression
pack_window_memory:`int \| None`	Maximum memory to use for delta window
pack_delta_cache_size:`int \| None`	Cache size for delta operations
pack_depth:`int \| None`	Maximum depth for pack deltas
pack_threads:`int \| None`	Number of threads to use for packing
pack_big_file_threshold:`int \| None`	Threshold for treating files as "big"
packed_git_limit:`int \| None`	Maximum total bytes for mmapped pack files. When exceeded, least-recently-used packs are closed to free memory.
delta_base_cache_limit:`int \| None`	Maximum bytes for caching delta base objects. Controls memory used to cache resolved base objects during delta unpacking, corresponding to Git's core.deltaBaseCacheLimit.
object_format:`ObjectFormat \| None`	Hash algorithm to use

def __iter__(self) -> Iterator[ObjectID]: ¶

overrides dulwich.object_store.BaseObjectStore.__iter__

Iterate over the SHAs that are present in this store.

def add_objects(self, objects: Sequence[tuple[ShaFile, str | None]], progress: Callable[[str], None] | None = None) -> Pack | None: ¶

overrides dulwich.object_store.BaseObjectStore.add_objects

Add a set of objects to this object store.

Returns: Pack object of the objects written.

Parameters
objects:`Sequence[tuple[ShaFile, str \| None]]`	Iterable over (object, path) tuples, should support __len__.
progress:`Callable[[str], None] \| None`	Optional progress reporting function.

Returns
`Pack \| None`	Undocumented

def add_pack(self) -> tuple[BinaryIO, Callable[[], None], Callable[[], None]]: ¶

overrides dulwich.object_store.PackCapableObjectStore.add_pack

overridden in dulwich.object_store.BucketBasedObjectStore, dulwich.object_store.DiskObjectStore

Add a new pack to this object store.

def add_pack_data(self, count: int, unpacked_objects: Iterator[UnpackedObject], progress: Callable[..., None] | None = None) -> Pack | None: ¶

overrides dulwich.object_store.PackCapableObjectStore.add_pack_data

Add pack data to this object store.

Parameters
count:`int`	Number of items to add
unpacked_objects:`Iterator[UnpackedObject]`	Iterator of UnpackedObject instances
progress:`Callable[..., None] \| None`	Optional progress callback

Returns
`Pack \| None`	Undocumented

def close(self): ¶

overrides dulwich.object_store.BaseObjectStore.close

overridden in dulwich.object_store.DiskObjectStore

Close the object store and release resources.

This method closes all cached pack files and frees associated resources. Can be called multiple times safely.

def contains_loose(self, sha: ObjectID | RawObjectID) -> bool: ¶

overrides dulwich.object_store.BaseObjectStore.contains_loose

Check if a particular object is present by SHA1 and is loose.

This does not check alternates.

def contains_packed(self, sha: ObjectID | RawObjectID) -> bool: ¶

overrides dulwich.object_store.BaseObjectStore.contains_packed

overridden in dulwich.object_store.DiskObjectStore

Check if a particular object is present by SHA1 and is packed.

This does not check alternates.

def count_pack_files(self) -> int: ¶

Count the number of pack files.

Returns
`int`	Number of pack files (excluding those with .keep files)

def delete_loose_object(self, sha: ObjectID): ¶

overridden in dulwich.object_store.BucketBasedObjectStore, dulwich.object_store.DiskObjectStore

Delete a loose object.

This method only handles loose objects. For packed objects, use repack(exclude=...) to exclude them during repacking.

def generate_pack_bitmaps(self, refs: dict[Ref, ObjectID], *, commit_interval: int | None = None, progress: Callable[[str], None] | None = None) -> int: ¶

Generate bitmap indexes for all packs that don't have them.

This generates .bitmap files for packfiles, enabling fast reachability queries. Equivalent to the bitmap generation part of 'git repack -b'.

Parameters
refs:`dict[Ref, ObjectID]`	Dictionary of ref names to commit SHAs
commit_interval:`int \| None`	Include every Nth commit in bitmap index (None for default)
progress:`Callable[[str], None] \| None`	Optional progress reporting callback

Returns
`int`	Number of bitmaps generated

def generate_pack_data(self, have: Iterable[ObjectID], want: Iterable[ObjectID], *, shallow: Set[ObjectID] | None = None, progress: Callable[..., None] | None = None, ofs_delta: bool = True) -> tuple[int, Iterator[UnpackedObject]]: ¶

overrides dulwich.object_store.BaseObjectStore.generate_pack_data

Generate pack data objects for a set of wants/haves.

Parameters
have:`Iterable[ObjectID]`	List of SHA1s of objects that should not be sent
want:`Iterable[ObjectID]`	List of SHA1s of objects that should be sent
shallow:`Set[ObjectID] \| None`	Set of shallow commit SHA1s to skip
progress:`Callable[..., None] \| None`	Optional progress reporting method
ofs_delta:`bool`	Whether OFS deltas can be included

Returns
`tuple[int, Iterator[UnpackedObject]]`	Undocumented

def get_raw(self, name: RawObjectID | ObjectID) -> tuple[int, bytes]: ¶

overrides dulwich.object_store.BaseObjectStore.get_raw

overridden in dulwich.object_store.DiskObjectStore

Obtain the raw fulltext for an object.

Returns: tuple with numeric type and object contents.

Parameters
name:`RawObjectID \| ObjectID`	sha for the object.

Returns
`tuple[int, bytes]`	Undocumented

def get_reachability_provider(self, prefer_bitmaps: bool = True) -> ObjectReachabilityProvider: ¶

overrides dulwich.object_store.BaseObjectStore.get_reachability_provider

Get the best reachability provider for the object store.

Parameters
prefer_bitmaps:`bool`	Whether to use bitmaps if available

Returns
`ObjectReachabilityProvider`	ObjectReachabilityProvider implementation (either bitmap-accelerated or graph traversal)

def get_unpacked_object(self, sha1: bytes, *, include_comp: bool = False) -> UnpackedObject: ¶

overrides dulwich.object_store.PackCapableObjectStore.get_unpacked_object

Obtain the unpacked object.

Parameters
sha1:`bytes`	sha for the object.
include_comp:`bool`	Whether to include compression metadata.

Returns
`UnpackedObject`	Undocumented

def iter_unpacked_subset(self, shas: Iterable[ObjectID | RawObjectID], include_comp: bool = False, allow_missing: bool = False, convert_ofs_delta: bool = True) -> Iterator[UnpackedObject]: ¶

overrides dulwich.object_store.BaseObjectStore.iter_unpacked_subset

Iterate over a subset of objects, yielding UnpackedObject instances.

Parameters
shas:`Iterable[ObjectID \| RawObjectID]`	Set of object SHAs to retrieve
include_comp:`bool`	Whether to include compressed data
allow_missing:`bool`	If True, skip missing objects; if False, raise KeyError
convert_ofs_delta:`bool`	Whether to convert OFS_DELTA objects

Returns
`Iterator[UnpackedObject]`	Iterator of UnpackedObject instances

Raises
`KeyError`	If an object is missing and allow_missing is False

def iterobjects_subset(self, shas: Iterable[ObjectID], *, allow_missing: bool = False) -> Iterator[ShaFile]: ¶

overrides dulwich.object_store.PackCapableObjectStore.iterobjects_subset

Iterate over a subset of objects in the store.

This method searches for objects in pack files, alternates, and loose storage.

Parameters
shas:`Iterable[ObjectID]`	Iterable of object SHAs to retrieve
allow_missing:`bool`	If True, skip missing objects; if False, raise KeyError

Returns
`Iterator[ShaFile]`	Iterator of ShaFile objects

Raises
`KeyError`	If an object is missing and allow_missing is False

def pack_loose_objects(self, progress: Callable[[str], None] | None = None) -> int: ¶

overridden in dulwich.object_store.BucketBasedObjectStore

Pack loose objects.

Returns: Number of objects packed

Parameters
progress:`Callable[[str], None] \| None`	Optional progress reporting callback

Returns
`int`	Undocumented

def repack(self, exclude: Set[bytes] | None = None, progress: Callable[[str], None] | None = None) -> int: ¶

Repack the packs in this repository.

Note that this implementation is fairly naive and currently keeps all objects in memory while it repacks.

Parameters
exclude:`Set[bytes] \| None`	Optional set of object SHAs to exclude from repacking
progress:`Callable[[str], None] \| None`	Optional progress reporting callback

Returns
`int`	Undocumented

delta_base_cache_limit = ¶

Undocumented

pack_big_file_threshold = ¶

Undocumented

pack_compression_level = ¶

overridden in dulwich.object_store.DiskObjectStore

Undocumented

pack_delta_cache_size = ¶

Undocumented

pack_delta_window_size = ¶

Undocumented

pack_depth = ¶

Undocumented

pack_index_version = ¶

overridden in dulwich.object_store.DiskObjectStore

Undocumented

pack_threads = ¶

Undocumented

pack_window_memory = ¶

Undocumented

packed_git_limit = ¶

Undocumented

@property

alternates: list[BaseObjectStore] = ¶

overridden in dulwich.object_store.DiskObjectStore

Return list of alternate object stores.

@property

packs: list[Pack] = ¶

overrides dulwich.object_store.BaseObjectStore.packs

List with pack objects.

def _add_cached_pack(self, base_name: str, pack: Pack): ¶

Add a newly appeared pack to the cache by path.

def _clear_cached_packs(self): ¶

Undocumented

def _enforce_packed_git_limit(self): ¶

Evict least-recently-used packs if the memory limit is exceeded.

def _get_loose_object(self, sha: ObjectID | RawObjectID) -> ShaFile | None: ¶

overridden in dulwich.object_store.BucketBasedObjectStore, dulwich.object_store.DiskObjectStore

Undocumented

def _iter_alternate_objects(self) -> Iterator[ObjectID]: ¶

Iterate over the SHAs of all the objects in alternate stores.

def _iter_cached_packs(self) -> Iterator[Pack]: ¶

Undocumented

def _iter_loose_objects(self) -> Iterator[ObjectID]: ¶

overridden in dulwich.object_store.BucketBasedObjectStore, dulwich.object_store.DiskObjectStore

Iterate over the SHAs of all loose objects.

def _mark_pack_used(self, pack_hash: str): ¶

Mark a pack as recently used for LRU tracking.

def _remove_pack(self, pack: Pack): ¶

overridden in dulwich.object_store.DiskObjectStore

Undocumented

def _total_pack_mmap_size(self) -> int: ¶

Return the total mmapped memory across all cached packs.

def _update_pack_cache(self) -> list[Pack]: ¶

overridden in dulwich.object_store.BucketBasedObjectStore, dulwich.object_store.DiskObjectStore

Undocumented

_pack_access_order: list[str] = ¶

Undocumented

_pack_cache: dict[str, Pack] = ¶

Undocumented