Working with data assets
List all your data assets
If you want to retrieve all the data assets you saved for one experiment, here is how :
from picsellia.client import Client
api_token = '4d388e237d10b8a19a93517ffbe7ea32ee7f4787'
project_token = '9c68b4ae-691d-4c3a-9972-8fe49ffb2799'
experiment = Client.Experiment(
api_token=api_token,
project_token=project_token
)
experiment.checkout('my_new_experiment')
experiment.list_data()
[
{
'id': 72,
'date_created': '2021-02-09T12:32:18.293746Z',
'last_update': '2021-02-09T12:32:18.293556Z',
'name': 'parameters',
'data': {'steps': 200000,
'nb_gpu': 1,
'batch_size': 8,
'learning_rate': 0.005,
'annotation_type': 'classification'
},
'type': 'table'
}
]
Delete all your data assets
If you think that your experiments needs a bit of cleansing, you can delete all the data assets you saved at once like this :
experiment.delete_all_data()
Retrieve an asset
If you want to retrieve a particular data asset, let's say the parameters of your last training, here is how you can do it :
You will only return what is stored in the data field of the data asset, not all the information about the asset.
experiment.get_data('parameters')
{'steps': 200000,
'nb_gpu': 1,
'batch_size': 8,
'learning_rate': 0.005,
'annotation_type': 'classification'}
Updating an asset
Now let's say that you want to change the value list of parameters for this training, here is how :
parameters = {
'steps': 5e6,
'nb_gpu': 8,
'batch_size': 64,
'learning_rate': 0.0055,
'annotation_type': 'detection'
}
experiment.update_data('parameters', data=parameters)
The method return the updated object
{
'id': 72,
'date_created': '2021-02-09T12:32:18.293746Z',
'last_update': '2021-02-09T12:32:18.293556Z',
'name': 'parameters',
'data': {'steps': 5000000.0,
'nb_gpu': 8,
'batch_size': 64,
'learning_rate': 0.0055,
'annotation_type': 'detection'},
'type': 'table'
}
Delete an asset
If you want to completely remove a data asset from all your visualizations, here is how :
experiment.delete_data('parameters')
Working with file assets
List all your file assets
If you want to retrieve all the file assets you saved for one experiment, here is how :
from picsellia.client import Client
api_token = '4d388e237d10b8a19a93517ffbe7ea32ee7f4787'
project_token = '9c68b4ae-691d-4c3a-9972-8fe49ffb2799'
experiment = Client.Experiment(
api_token=api_token,
project_token=project_token
)
experiment.checkout('my_new_experiment')
experiment.list_files()
The method return the list of files
[{'id': 22,
'date_created': '2021-02-09T12:32:18.022694Z',
'last_update': '2021-02-09T12:32:18.022465Z',
'large': False,
'name': 'config',
'object_name': '9a141ede-03dc-4e6c-a695-38661d9a97c3/pipeline.config'},
{'id': 23,
'date_created': '2021-02-09T12:32:18.068900Z',
'last_update': '2021-02-09T12:32:18.068723Z',
'large': True,
'name': 'model-latest',
'object_name': '9a141ede-03dc-4e6c-a695-38661d9a97c3/0/saved_model.zip'},
{'id': 24,
'date_created': '2021-02-09T12:32:18.112582Z',
'last_update': '2021-02-09T12:32:18.112410Z',
'large': True,
'name': 'checkpoint-data-latest',
'object_name': '9a141ede-03dc-4e6c-a695-38661d9a97c3/ckpt-0.data-00000-of-00001'},
{'id': 25,
'date_created': '2021-02-09T12:32:18.156945Z',
'last_update': '2021-02-09T12:32:18.156767Z',
'large': False,
'name': 'checkpoint-index-latest',
'object_name': '9a141ede-03dc-4e6c-a695-38661d9a97c3/ckpt-0.index'}]
Delete all your file assets
f you think that your experiments needs a bit of cleansing, you can delete all the file assets you saved at once like this :
experiment.delete_all_files()
Retrieve and download a file asset
If you only need to retrieve informations about a file you can use the following method
experiment.get_file('config')
{'id': 22,
'date_created': '2021-02-09T12:32:18.022694Z',
'last_update': '2021-02-09T12:32:18.022465Z',
'large': False,
'name': 'config',
'object_name': '9a141ede-03dc-4e6c-a695-38661d9a97c3/pipeline.config'}
If you need to download the file you can use the following method
experiment.download('config')
If the file size exceeds 5Mb or if you get errors while using download, you might need to set the large parameter to True
experiment.download('config', large=True)
Alternatively, you can download the file to a specified folder, just set the path parameter to the path of the folder (must be an existing folder)
experiment.download('config', path='training')
Update a file asset
The update function is there if you need to perform update on file information such as its name.
experiment.update_file('config', name='new-config')
If you call the store method again, it will automatically erase the old file and replace if with the new one
Delete a file asset
If you want to remove a file asset from one of your experiment, here is how to do it.
from picsellia.client import Client
api_token = '4d388e237d10b8a19a93517ffbe7ea32ee7f4787'
project_token = '9c68b4ae-691d-4c3a-9972-8fe49ffb2799'
experiment = Client.Experiment(
api_token=api_token,
project_token=project_token
name='my_new_experiment'
)
experiment.delete_file('model-latest')
Be aware that this will also permanently delete the file from our storage.
