jacinle.io package

Submodules

jacinle.io.common module

jacinle.io.common.get_ext(fname: str, match_first: bool = False)

Get the extension of a file name.

Parameters:

• fname – the file name.
• match_first – if True, match the left-most “.”.

jacinle.io.common.fsize_format(num)

Human-readable file size.

Examples

>>> fsize_format(0)
'0 bytes'
>>> fsize_format(1024)
'1 kB'

Source: https://stackoverflow.com/questions/1094841/reusable-library-to-get-human-readable-version-of-file-size

jacinle.io.common.auto_close(file)

A context manager that automatically closes the file.

jacinle.io.fs module

jacinle.io.fs.as_file_descriptor(fd_or_fname: Union[str, io.IOBase], mode: str = 'r') → io.IOBase

Convert a file descriptor or a file name to a file descriptor.

Parameters:

• fd_or_fname – a file descriptor or a file name.
• mode – the mode to open the file if fd_or_fname is a file name.

Returns:

a file descriptor.

jacinle.io.fs.fs_verbose(mode=True)

A context manager to enable/disable verbose mode in file system operations.

jacinle.io.fs.set_fs_verbose(mode: bool = True)

Enable/disable verbose mode in file system operations.

jacinle.io.fs.open(filename: str, mode: str, **kwargs)

Open a file.

jacinle.io.fs.open_txt(filename, mode, **kwargs)

Open a text file.

jacinle.io.fs.open_h5(filename: str, mode: str, **kwargs)

Open a HDF5 file.

jacinle.io.fs.open_gz(filename, mode)

Open a gzip file.

jacinle.io.fs.load(filename: str, **kwargs)

Load a file with automatic file type detection.

jacinle.io.fs.load_txt(filename, **kwargs)

Load a text file.

jacinle.io.fs.load_h5(filename: str, **kwargs)

Load a HDF5 file.

jacinle.io.fs.load_pkl(fd_or_filename: Union[str, io.IOBase], **kwargs)

Load a pickle file.

jacinle.io.fs.load_pklgz(filename: str, **kwargs)

Load a gziped pickle file.

jacinle.io.fs.load_npy(fd_or_filename: Union[str, io.IOBase], **kwargs)

Load a npy numpy file.

jacinle.io.fs.load_npz(fd_or_filename: Union[str, io.IOBase], **kwargs)

Load a npz numpy file.

jacinle.io.fs.load_mat(filename: str, **kwargs)

Load a matlab file.

jacinle.io.fs.load_pth(filename, **kwargs)

Load a PyTorch file.

jacinle.io.fs.dump(filename, obj, **kwargs)

Dump a file with automatic file type detection.

jacinle.io.fs.dump_pkl(fd_or_filename: Union[str, io.IOBase], obj, **kwargs)

Dump a pickle file.

jacinle.io.fs.dump_pklgz(filename: str, obj, **kwargs)

Dump a gziped pickle file.

jacinle.io.fs.dump_npy(filename: str, obj, **kwargs)

Dump a npy numpy file.

jacinle.io.fs.dump_npz(filename: str, obj, **kwargs)

Dump a npz numpy file.

jacinle.io.fs.dump_mat(filename, obj, **kwargs)

Dump a matlab file.

jacinle.io.fs.dump_pth(filename, obj, **kwargs)

Dump a PyTorch file.

jacinle.io.fs.safe_dump(filename: str, data, use_lock=True, use_temp=True, lock_timeout=10) → bool

Dump data to a file in a safe way. Basically, it will dump the data to a temporary file and then move it to the target file. This is to avoid the case that the target file is corrupted when the program is interrupted during the dumping process. It also supports file locking to avoid the case that multiple processes are dumping data to the same file at the same time.

Parameters:

• filename – the target file name.
• data – the data to be dumped.
• use_lock – whether to use file locking.
• use_temp – whether to use a temporary file.
• lock_timeout – the timeout for file locking.

Returns:

If uses temp file, return True if the data is dumped to the temp file successfully, otherwise False. If not use temp file, return the result of the dump operation.

jacinle.io.fs.link(path_origin: str, *paths, use_relative_path=True)

Create a symbolic link to a file or directory.

Parameters:

• path_origin – the original file or directory.
• paths – the symbolic links to be created.
• use_relative_path – whether to use relative path.

jacinle.io.fs.mkdir(path)

Create a directory if it does not exist without raising errors when the directory already exists.

jacinle.io.fs.lsdir(dirname: str, pattern: Optional[str] = None, return_type: Union[str, jacinle.io.fs.LSDirectoryReturnType] = 'full', sort: bool = True) → List[str]

List all files in a directory.

Parameters:

• dirname – the directory name.
• pattern – the file name pattern in glob format.
• return_type – the return type. Can be one of the following: ‘base’: return the base name of the file. ‘name’: return the file name. ‘rel’: return the relative path of the file. ‘full’: return the full path of the file. ‘real’: return the real path of the file.
• sort – whether to sort the file names.

Returns:

a list of file names.

jacinle.io.fs.remove(file_or_dirname: str)

Remove a file or directory.

jacinle.io.fs.locate_newest_file(dirname: str, pattern: str) → Optional[str]

Locate the newest file in a directory. If there is no file matching the pattern, return None.

Parameters:

• dirname – the directory name.
• pattern – the file name pattern in glob format.

Returns:

the full path of the newest file.

jacinle.io.fs.tempfile(mode: str = 'w+b', suffix: str = '', prefix: str = 'tmp')

A context manager that creates a temporary file and deletes it after use.

Example

with tempfile() as f:
    f.write(b'hello world')
    f.seek(0)
    print(f.read())

Parameters:

• mode – the mode to open the file.
• suffix – the suffix of the file name.
• prefix – the prefix of the file name.

jacinle.io.network module

jacinle.io.network.download(url: str, dirname: str, cli: bool = True, filename: str = None, md5: str = None)

Download URL to a directory. Will figure out the filename automatically from URL. Will figure out the filename automatically from URL, if not given.

Parameters:

• url – URL to download.
• dirname – directory to download to.
• cli – whether to use CLI progress bar.
• filename – filename to save to.
• md5 – md5 hash to check.

Source: https://github.com/ppwwyyxx/tensorpack/blob/master/tensorpack/utils/fs.py

jacinle.io.network.check_integrity(filename: str, md5: str) → bool

Check data integrity using md5 hashing.

Parameters:

• filename – path to the file.
• md5 – md5 hash to check.

jacinle.io.pretty module

Functions to dump Python objects into human-readable formats.

jacinle.io.pretty.iter_txt(fd: io.IOBase, strip: bool = True) → Iterable[str]

Iterate over lines in a text file. This function will ignore empty lines.

Parameters:

• fd – a file descriptor.
• strip – whether to strip the line.

jacinle.io.pretty.pretty_dump(file, obj, **kwargs)

Dump a file with pretty-printing.

jacinle.io.pretty.pretty_load(file, **kwargs)

Load a file with pretty-printing.

jacinle.io.pretty.dumps_json(value: Any, compressed: bool = True) → str

Dump a JSON object into a string. In addition to the standard JSON format, this function also supports

• __jsonify__: an instance method for objects that returns a JSON-serializable object.
• For classes, it will store __dict__ as the JSON object.

Note that both features can not be preserved when loading the JSON object back.

Parameters:

• value – the object to dump.
• compressed – whether to use compressed format. If set to False, the dumped string will be pretty-printed.

Returns:

a string.

jacinle.io.pretty.dump_json(value: Any, compressed: bool = True) → str

Dump a JSON object into a file. In addition to the standard JSON format, this function also supports

• __jsonify__: an instance method for objects that returns a JSON-serializable object.
• For classes, it will store __dict__ as the JSON object.

Note that both features can not be preserved when loading the JSON object back.

Parameters:

• value – the object to dump.
• compressed – whether to use compressed format. If set to False, the dumped string will be pretty-printed.

Returns:

a file.

jacinle.io.pretty.loads_json(value: str) → Any

Load a JSON object from a string.

jacinle.io.pretty.load_json(value: str) → Any

Load a JSON object from a string.

jacinle.io.pretty.pretty_dumps_json(value: Any, compressed: bool = False) → str

Dump a JSON object into a string, with pretty-printing.

jacinle.io.pretty.pretty_dump_json(value: Any, compressed: bool = False) → str

Dump a JSON object into a file, with pretty-printing.

jacinle.io.pretty.dumps_jsonc(value: Iterable[Dict[KT, VT]]) → str

Dump a list of dictionary into a JSON string, separated by new lines, with compressed format.

jacinle.io.pretty.dump_jsonc(value: Iterable[Dict[KT, VT]]) → str

Dump a list of dictionary into a JSON string, separated by new lines, with compressed format.

jacinle.io.pretty.loads_jsonc(value: str) → List[Dict[KT, VT]]

Load a list of JSON dictionaries from a string. This function supports multiple JSON objects in a single string, separated by newlines. Note that this function only support a list of plain dictionaries. Do not use this function to load JSON objects with nested lists or dictionaries.

Parameters:value – a string. Returns:a list of dictionaries.

jacinle.io.pretty.load_jsonc(value: str) → List[Dict[KT, VT]]

Load a list of JSON dictionaries from a string. This function supports multiple JSON objects in a single string, separated by newlines. Note that this function only support a list of plain dictionaries. Do not use this function to load JSON objects with nested lists or dictionaries.

Parameters:value – a string. Returns:a list of dictionaries.

jacinle.io.pretty.dumps_xml(value: Dict[KT, VT], **kwargs) → str

Dump an XML object into a string.

jacinle.io.pretty.dump_xml(value: Dict[KT, VT], **kwargs) → str

Dump an XML object into a file.

jacinle.io.pretty.loads_xml(value: str, name_key: str = '__name__', attribute_key: Optional[str] = '__attribute__') → Dict[KT, VT]

Load an XML object from a string. It will return a dictionary as the root node. For each node, it will have a key named “__name__” as the tag name, and a key “__attributes__” as a dictionary of attributes. Each child node will be a nested dictionary under the root node. If there are multiple child nodes with the same tag name, they will be stored in a list.

Parameters:

• value – a string.
• name_key – the key name for the tag name.
• attribute_key – the key name for the attributes. If set to None, the attributes will be stored in the root node.

Returns:

a dictionary.

jacinle.io.pretty.load_xml(value: str, name_key: str = '__name__', attribute_key: Optional[str] = '__attribute__') → Dict[KT, VT]

Load an XML object from a string. It will return a dictionary as the root node. For each node, it will have a key named “__name__” as the tag name, and a key “__attributes__” as a dictionary of attributes. Each child node will be a nested dictionary under the root node. If there are multiple child nodes with the same tag name, they will be stored in a list.

Parameters:

• value – a string.
• name_key – the key name for the tag name.
• attribute_key – the key name for the attributes. If set to None, the attributes will be stored in the root node.

Returns:

a dictionary.

jacinle.io.pretty.dumps_yaml(value: Any) → str

Dump a YAML object into a string.

jacinle.io.pretty.dump_yaml(value: Any) → str

Dump a YAML object into a file.

jacinle.io.pretty.loads_yaml(value: str) → Any

Load a YAML object from a string.

jacinle.io.pretty.load_yaml(value: str) → Any

Load a YAML object from a string.

jacinle.io.pretty.dumps_txt(value: Iterable[str]) → str

Dump a list of strings into a string, separated by newlines.

Parameters:value – a list of strings. Returns:a string.

jacinle.io.pretty.dump_txt(value: Iterable[str]) → str

Dump a list of strings into a file, separated by newlines.

Parameters:value – a list of strings. Returns:a file.

jacinle.io.pretty.dumps_struct(value)

Dump a structured object into a string, using jacinle.utils.printing.stformat().

jacinle.io.pretty.dump_struct(value)

Dump a structured object into a file, using jacinle.utils.printing.stformat().

jacinle.io.pretty.dumps_kv(value)

Dump a structured object into a string, using jacinle.utils.printing.kvformat().

jacinle.io.pretty.dump_kv(value)

Dump a structured object into a file, using jacinle.utils.printing.kvformat().

jacinle.io.pretty.dumps_env(value)

Dump a structured object into a string, similar to os.environ().

jacinle.io.pretty.dump_env(value)

Dump a structured object into a file, similar to os.environ().

jacinle.io.tempfile module

A context manager that creates a temporary file and deletes it after use.

Example

with tempfile() as f:
    f.write(b'hello world')
    f.seek(0)
    print(f.read())

param mode:the mode to open the file. param suffix:the suffix of the file name. param prefix:the prefix of the file name.