Keep artifacts local in a cloud instance¶

If you want to default to keeping artifacts local in a cloud instance, enable keep_artifacts_local.

import lamindb as ln

ln.settings.transform.stem_uid = "l9lFf83aPwRc"
ln.settings.transform.version = "1"
ln.track()

💡 connected lamindb: testuser1/keep-artifacts-local

💡 notebook imports: lamindb==0.73.2

💡 saved: Transform(uid='l9lFf83aPwRc5zKv', version='1', name='Keep artifacts local in a cloud instance', key='keep-artifacts-local', type='notebook', created_by_id=1, updated_at='2024-06-13 12:19:28 UTC')

💡 saved: Run(uid='lKZsIMPnKnCtOJEgwsk1', transform_id=1, created_by_id=1)

Run(uid='lKZsIMPnKnCtOJEgwsk1', started_at='2024-06-13 12:19:28 UTC', is_consecutive=True, transform_id=1, created_by_id=1)

You can register a managed local storage location as follows:

ln.settings.storage_local = "./my_storage_local"

💡 defaulting to local storage: /home/runner/work/lamindb/lamindb/docs/faq/my_storage_local

Now, you have two storage locations: one in the S3 bucket, and the other locally.

ln.Storage.df()

Show code cell output Hide code cell output

	uid	root	description	type	region	instance_uid	run_id	created_by_id	updated_at
id
2	E5sJZHPi2i1b	/home/runner/work/lamindb/lamindb/docs/faq/my_...	None	local	None	6uGWmLpZlNoJ	None	1	2024-06-13 12:19:29.798459+00:00
1	TQpDCAVxAWqt	s3://lamindb-ci/keep-artifacts-local	None	s3	us-west-1	6uGWmLpZlNoJ	None	1	2024-06-13 11:56:26.669501+00:00

Update storage description¶

You can add a description to the storage by using the description parameter:

storage_record = ln.Storage.filter(root=ln.settings.storage_local).one()
storage_record.description = "Files stored locally in site X on server Y for reason ABC"
storage_record.save()
ln.Storage.df()

	uid	root	description	type	region	instance_uid	run_id	created_by_id	updated_at
id
2	E5sJZHPi2i1b	/home/runner/work/lamindb/lamindb/docs/faq/my_...	Files stored locally in site X on server Y for...	local	None	6uGWmLpZlNoJ	None	1	2024-06-13 12:19:29.831075+00:00
1	TQpDCAVxAWqt	s3://lamindb-ci/keep-artifacts-local	None	s3	us-west-1	6uGWmLpZlNoJ	None	1	2024-06-13 11:56:26.669501+00:00

Use local storage¶

If you save an artifact, by default, it’s stored in local storage.

original_filepath = ln.core.datasets.file_fcs()
artifact = ln.Artifact(original_filepath, description="My fcs file").save()
local_path = artifact.path
local_path

You’ll see the .fcs file named by the uid in your .lamindb/ directory under ./my_storage_local/:

ln.settings.storage_local.view_tree()

If you’d like to upload an artifact, you pass upload=True to the save() method.

artifact.save(upload=True)

You now see the artifact in the S3 bucket:

ln.setup.settings.storage.root.view_tree()

And it’s no longer present in local storage:

ln.setup.settings.instance.storage_local.root.view_tree()

Direct upload¶

You can also directly upload a file by passing upload=True:

filepath = ln.core.datasets.file_mini_csv()
artifact2 = ln.Artifact(filepath, description="My csv file").save(upload=True)
artifact2.path

Now we have two files on S3:

ln.Artifact.df(include="storage__root")

Show code cell output Hide code cell output

	storage__root	uid	version	description	key	suffix	accessor	size	hash	hash_type	n_objects	n_observations	visibility	key_is_virtual	storage_id	transform_id	run_id	created_by_id	updated_at
id
2	s3://lamindb-ci/keep-artifacts-local	qEPpjGK0XgNOxNA3v9oa	None	My csv file	None	.csv	None	11	z1LdF2qN4cN0M2sXrcW8aw	md5	None	None	1	True	1	1	1	1	2024-06-13 12:19:31.700470+00:00
1	s3://lamindb-ci/keep-artifacts-local	mIrAFVcLJn4q1dKxATZy	None	My fcs file	None	.fcs	None	6785467	KCEXRahJ-Ui9Y6nksQ8z1A	md5	None	None	1	True	1	1	1	1	2024-06-13 12:19:30.228114+00:00

Pre-existing artifacts¶

Assume we already have a file in our registered local storage location:

1 sub-directory & 2 files with suffixes '', '.bam'
/home/runner/work/lamindb/lamindb/docs/faq/my_storage_local
├── .lamindb/
│   └── _is_initialized
└── output.bam

If we create an artifact from it, it remains where it is during saving:

my_existing_file = ln.Artifact("./my_storage_local/output.bam", description="my existing file").save()
ln.UPath("my_storage_local/").view_tree()

The storage path of the artifact is constructed using key because key_is_virtual=False:

my_existing_file

However, if we decide to upload the artifact, we’ll use the uid for constructing the storage path and switch key_is_virtual=True:

my_existing_file.save(upload=True)

Here is the remote path of the artifact:

my_existing_file.path

And here are the contents of the storage locations:

# the path on S3
ln.setup.settings.storage.root.view_tree()
# the local path
ln.setup.settings.instance.storage_local.root.view_tree()

Delete the test instance¶

Delete the artifacts:

artifact.delete(permanent=True)
artifact2.delete(permanent=True)
my_existing_file.delete(permanent=True)

Delete the instance:

ln.setup.delete("keep-artifacts-local", force=True)