Python SDK

Python SDK Reference

Here we will describe every method of the picsellia package

Installation

The first thing you need to do is to download our python package from PyPI, you can install it with pip

pip install picsellia

Dependencies

Here is the several packages that will be installed along the picsellia package, they are used to perform mathematical operations, array or image manipulations and HTTP requests

  • numpy>=1.18.5

  • Pillow>=7.2.0

  • requests>=2.24.0

  • scipy>=1.4.1

Client

To connect your code to Picsell.ia, you must initialize our client.

from picsellia.client import Client

api_token = '4d388e237d10b8a19a93517ffbe7ea32ee7f4787'

clt = Client(api_token)

You will be greeted by this nice message (Pierre-Nicolas is my name, you should see your username here, unless you are named Pierre-Nicolas too 👀 ) :

Hi Pierre-Nicolas, welcome back.

Datalake

You need to instantiate a Datalake object in order to interact with your pictures or datasets.

__init__

If you need to interact with your datalake, you must initialize the Datalake class. It is a subclass of the Client.

from picsellia.client import Client

datalake = Client.Datalake(
    api_token=None,
    organization=None
    )

Arguments:

  • api_token (string) Your personal API token (Find it in your profile page 🔥)

  • organization (string, optional) the name of the organization you want to work with (None default to your organization)

Returns:

None

upload

This upload method allows you to upload your Dataset, (images and annotations or annotations only).

datalake.upload(
      name: str=None, 
      version: str=None,
      imgdir: str=None, -> "path/to/imgdir"
      ann_path: str=None, -> "path/to/annotations"
      ann_format: str=None, -> "COCO", "PASCAL-VOC", "PICSELLIA"
      tags: List[str]=[], 
      nb_jobs: int=5,
      rectangle: bool=False):

Arguments

  • name (string, required) Name of the dataset to be created or fetched if you want to upload only annotations

  • version (string, optionnal) Version of the dataset to be fetched, if None we will fetch the latest created with this name.

  • imgdir (string, optionnal) Path to the directory of the pictures to upload, leave blank if you just want to upload the annotations.

  • ann_path (string, required) Path to the annotations files, it can be a directory or a file.

  • ann_format (string, required) you can upload COCO format, PASCAL-VOC or PICSELLIA

  • tags (list, optionnal) List of tags to add to the uploaded images, if None tags will be the upload date.

  • nb_jobs (int, optionnal) Number of process to use to upload, put nb_jobs=-1 if you want to use all available processes.

  • rectangle (bool, optionnal) set to True if you want to force to upload bbox annotations when both polygons and bbox are available in your annotation files.

Picture

The picture object allow you to interact with your assets only.

__init__

If you need to interact with your experiments, you must initialize the Picture class. It is a subclass of the Datalake.

from picsellia.client import Client

picture = Client.Datalake.Picture(
    api_token=None,
    organization=None
    )

Arguments:

  • api_token (string) Your personal API token (Find it in your profile page 🔥)

  • organization (string, optional) the name of the organization you want to work with (None default to your organization)

upload

To upload assets to your lake

picture.upload(filepath=None, tags=[], source='sdk')

Arguments:

  • filtepath (string or list) Either one filepath pointing to the asset to upload or a list of path.

  • tags (list, optional) the list of tags to attach to the upload assets

  • source (string, optional) Specify the source of the upload, default is "sdk"

list

picture.list()

Returns:

A list containing the pictures objects for your datalake.

{'pictures': [{'picture_id': '25d76bee-a6d3-43d7-8620-6ff18f7a5557',
   'internal_key': '15288614-bedb-4cab-97c1-23684cf9c761.jpg',
   'external_url': 'GE_121.jpg',
   'creation_date': '2021-02-07',
   'height': 310,
   'width': 322,
   'tag': []},
  {'picture_id': 'd1cf0d96-5c05-4fb4-aa4a-5e90f3c748da',
   'internal_key': 'ed6e12e3-0db3-461f-bcd9-54d48509680b.jpg',
   'external_url': 'GE_55.jpg',
   'creation_date': '2021-02-07',
   'height': 2908,
   'width': 4800,
   'tag': []},
  {'picture_id': '716e45a8-09f6-4ec5-9dd1-29c313ae2cdf',
   'internal_key': 'a9597e66-584d-4568-b1cc-31b951154edd.jpg',
   'external_url': 'GE_309.jpg',
   'creation_date': '2021-02-07',
   'height': 208,
   'width': 254,
   'tag': []},
  {'picture_id': '88d2b82d-2a38-4c30-912b-79474a617072',
   'internal_key': 'c83493f7-ce61-4a0b-8166-54698d071792.jpg',
   'external_url': 'Test85.jpg',
   'creation_date': '2021-02-07',
   'height': 663,
   'width': 710,
   'tag': []},
  {'picture_id': 'd9d4684f-a2d1-4431-93e7-dce352aff471',
   'internal_key': 'b7a8bb6b-d3f2-46d5-9599-dfb9b4d2f1cd.jpg',
   'external_url': 'GE_466.jpg',
   'creation_date': '2021-02-07',
   'height': 462,
   'width': 520,
   'tag': []},
  }

fetch

Fetch images with corresponding tags

pictures.fetch(
    quantity=1,
    tags=["drone", "coco"],
    )

Parameters:

  • quantity (float, optional) the percentage of assets to fetch ( 1 meaning 100%)

  • tags (list, required) a list of tags used to search in your Datalake

Returns:

The list of all the fetched assets

 [{'picture_id': '8b536f4c-c95b-4f5f-afbe-a9f31242a235',
  'internal_key': '51ee5ee9-5176-4e98-b173-0687ed6c7b2f.jpg',
  'external_url': '9999966_00000_d_0000055.jpg',
  'creation_date': '2021-02-07',
  'height': 1050,
  'width': 1400,
  'tag': ['drone', 'coco', 'vizdrone']},
 {'picture_id': '426ce7bd-7535-4fe5-80cd-c41e07f84c99',
  'internal_key': '7f4f1b60-d1bb-4458-b3bb-5f3d01a8f7eb.jpg',
  'external_url': '9999955_00000_d_0000312.jpg',
  'creation_date': '2021-02-07',
  'height': 788,
  'width': 1400,
  'tag': ['drone', 'coco', 'vizdrone']},
 {'picture_id': '320e69fc-964a-478e-b689-05351213578e',
  'internal_key': '5aa4036e-8050-4fef-9c3c-af9ba46db511.jpg',
  'external_url': '9999955_00000_d_0000043.jpg',
  'creation_date': '2021-02-07',
  'height': 788,
  'width': 1400,
  'tag': ['drone', 'coco', 'vizdrone']},
 {'picture_id': 'bed1ddab-7cf1-460b-99a8-c4125612caa3',
  'internal_key': 'f73fedbf-d87e-4483-859a-77a3f8e38702.jpg',
  'external_url': '9999982_00000_d_0000167.jpg',
  'creation_date': '2021-02-07',
  'height': 1050,
  'width': 1400,
  'tag': ['drone', 'coco', 'vizdrone']},
 {'picture_id': '0bc695d1-03bb-48ea-bd89-5bef5bf02c23',
  'internal_key': '98ac39c5-a157-4a5e-bafc-a8399b90f230.jpg',
  'external_url': '9999974_00000_d_0000049.jpg',
  'creation_date': '2021-02-07',
  'height': 1078,
  'width': 1916,
  'tag': ['drone', 'coco', 'vizdrone']},
  ]

status

Once, you have fetched pictures, you can call status method to visualize the number of assets fetched

pictures.status()

Return:

Number of Assets selected : 1472

delete

Delete the list of pictures 

pictures.delete(
    pictures=None
    )

Arguments:

  • pictures (list, optional) The list of pictures to delete from your lake, if none will delete the latest fetched pictures.

Returns:

None

add_tags

Add tags to selected pictures

pictures.add_tags(
    pictures=[], 
    tags=["tag_to_add"]
)

Arguments:

  • pictures (list, optional) The list of pictures to selected from your lake, if none will add tags to the last fetched pictures.

  • tags (list, required) The list of tags to add to the selected pictures

Returns:

None

remove_tags

Remove tags from selected pictures

pictures.remove_tags(
    pictures=[], 
    tags=["tag_to_add"]
)

Arguments:

  • pictures (list, optional) The list of pictures to selected from your lake, if none will remove tags from the last fetched pictures.

  • tags (list, required) The list of tags to delete from the selected pictures

Returns:

None

Dataset

The dataset object allow you to interact with your dataset ( annotations, labels, question and answers ).

__init__

If you need to interact with your experiments, you must initialize the Dataset class. It is a subclass of the Datalake.

from picsellia.client import Client

dataset = Client.Datalake.Dataset(
    api_token=None,
    organization=None
    )

Arguments:

  • api_token (string) Your personal API token (Find it in your profile page 🔥)

  • organization (string, optional) the name of the organization you want to work with (None default to your organization)

list

dataset.list()

Returns:

A list containing the dataset objects for your account.

-------------
Dataset Name: GoogleEarthShip
Dataset Version: first
Nb Assets: 793
-------------
-------------
Dataset Name: VizDrone2017
Dataset Version: first
Nb Assets: 6470
-------------
-------------
Dataset Name: FaceMaskDetection
Dataset Version: first
Nb Assets: 6024
-------------
-------------
Dataset Name: TrashDataset
Dataset Version: first
Nb Assets: 1435
-------------

fetch

Fetch dataset with its name and version

dataset.fetch(
    name="myDataset",
    version="latest",
    )

Parameters:

  • name (string, optional) the name of the dataset to fetch

  • version (string, optional) the version of the dataset to fetch, if None, the client will fetch latest

Returns:

A Dataset object relative to the fetched dataset

dataset = dataset.fetch(name="myDataset",version="latest")
print(dataset)
{
  "dataset_id": "9061846f-597a-47d6-9711-7f75671841a2",
  "dataset_name": "myDataset",
  "version": "latest",
  "size": 128,
  "description": "None",
  "private": true
}

create

Create a a new dataset and attach pictures to it, to do so you first need to fetch pictures

dataset.create(
    name="myDataset",
    description='',
    private=True,
    pictures=[]
    )

Parameters:

  • name (string, optional) the name ot the dataset to create

  • description (string, optional) the description of the dataset to create

  • private (bool, optional) If True, your dataset will be accessible to anyone

  • pictures (list, required) The list of pictures to attach to your dataset

Return:

the id of the created dataset

new_version

Create a a new dataset and attach pictures to it, to do so you first need to fetch pictures

dataset.new_version(
    name="myDataset",
    version='newVersion',
    from_version='latest',
    pictures=[]
    )

Parameters:

  • name (string, optional) the name of the dataset

  • version (string, optional) the version name of the dataset to create

  • from_version (string, optional)The origin version for your new version, if None we'll create a new version from the latest version

  • pictures (list, required) The list of pictures to attach to your version

Return:

None

create_labels

Sets up the labels (tools for drawing bounding-boxes, polygons...) for your dataset.

dataset.create_labels(
    name=None,
    ann_type=None
)

Arguments:

  • name (str, required) name of the label want to set up (e.g. car, bird, plane...)

  • ann_type (str, required) type of shape that will be used for annotations :

    • 'rectangle': bounding-boxes for object-detection

    • 'polygon': polygons for segmentation

Returns:

None

list_pictures

Get the list of all the images in the fetched dataset

dataset.list_pictures(
    dataset_id=None
    )

Arguments:

  • dataset_id (str, optional) id of the dataset (if not fetched)

Returns:

list of Picture objects

add_data

Add the fetch pictures to a dataset

dataset.add_data(
    name="myDataset",
    version='myVersion',
    pictures=[]
    )

If you fetched a dataset before, you won't have to specify the name and version of the dataset

Parameters:

  • name (string, optional) the name of the dataset

  • version (string, optional) the version name of the dataset to fetch if None, we'll take latest

  • pictures (list, required) The list of pictures to attach to add

Return:

None

delete

Delete the a dataset

dataset.delete(
    name="myDataset",
    version='myVersion',
    )

Arguments:

  • name (string, optional) the name of the dataset to delete

  • version (string, optional) the version name of the dataset to delete if None, we'll take latest

Returns:

None

download

Download all the images from a dataset in a folder

dataset.download(
    dataset=None,
    folder_name=None
)

Arguments:

  • dataset (str, required) the name of the dataset you want to download written <dataset_name>/<version>

  • folder_name (str, optional) the name of the folder you want to download the pictures in, defaults to dataset_name/version if None

Returns:

None

add_annotation

Create an annotation for a picture in a dataset (or add objects to existing annotation)

dataset.add_annotation(
    picture_external_url=None
    dataset_id=None,
    data={},
    image_qa={}
)

Arguments:

  • picture_external_url (str, required) the name of the target image

  • dataset_id (str, optional) leave None if you already fetched a dataset

  • data (dict, required) annotation data

  • image_qa (dict, optional) Q&A data for image

Formats:

  • classification

data = [
    {
        "type": "classification",
        "label": "car"
    }
]
dataset.add_annotation("awsm_pic.jpg", True, data=data)

  • detection

data = [
    {
        "type": "rectangle",
        "label": "car",
        "rectangle": {
            "top": 16,
            "left": 10,
            "width": 50,
            "height": 60
        }
    }
]
dataset.add_annotation("awsm_pic.jpg", True, data=data)

  • segmentation

data = [{
    "type": "polygon", 
    "label" : "rose",
    "polygon": {
        "geometry": [
            {
                "x": 12,
                "y": 15
            },
            {
                "x": 178,
                "y": 151
            },
            {
                "x": 122,
                "y": 196
            },
            {
                "x": 112,
                "y": 10
            },
            
        ]
    }
}   
]
dataset.add_annotation("awsm_pic.jpg", True, data=data)

  • Q&A

data = [
    {
    "type": "polygon", 
    "label" : "rose",
    "polygon": {
        "geometry": [
            {
                "x": 12,
                "y": 15
            },
            {
                "x": 178,
                "y": 151
            },
            {
                "x": 122,
                "y": 196
            },
            {
                "x": 112,
                "y": 10
            },
            
        ],
        
    },
    "qa": [{
            "type": "text",
            "question": "What color ?", 
            "answer": "red"
            
        },
        {
            "type": "mc",
            "question": "What color ?", 
            "answer": ["red"], 
            "choices": ["red", "yellow", "blue"]
        },
        {
            "type": "select",
            "question": "Is it raining ?", 
            "answer": "yes",
            "choices": ["yes", "no"]
        },
        {
            "type": "range",
            "question": "size ?", 
            "answer": 68, 
            "max": 100, 
            "min": 0
        }]
    }   
]

image_qa = [
    {
        "type": "text", 
        "question": "How much is the image rotated ?", 
        "answer": "approx. 32 deg."
    },
    {
        "type": "mc",
        "question": "image attribute", 
        "answer": ["high contrast"],
        "choices": ["high contrast", "saturated"]
    },
    {
        "type": "select",
        "question": "image color", 
        "answer": "blue", 
        "choices": ["red", "blue", "green"]
    },
    {
        "type": "range",
        "question": "brigthness", 
        "answer": 36,
        "max": 100,
        "min": 0
    },
]
ds.add_annotation("awsm_pic.jpg", True, data=data, image_qa=image_qa)

Network

Networks are trained architectures that you can either deploy for inference (if available), use to start new experiments and share within your Organization's models.

The Network object

{'organization': {'name': 'picsell'},
  'model_id': 'b76ececa-274d-48de-b39e-70cf73941aba',
  'serving_id': 'b76ececa-274d-48de-b39e-70cf73941aba',
  'tag': ['efficientdet', 'd2', 'COCO', 'base'],
  'private': False,
  'network_name': 'efficientdet-d2',
  'description': 'This is a real game changer',
  'model_object_name': '',
  'checkpoint_object_name': '',
  'origin_checkpoint_objects': {},
  'type': 'detection',
  'files': {'config': 'b76ececa-274d-48de-b39e-70cf73941aba/pipeline.config',
   'model-latest': 'b76ececa-274d-48de-b39e-70cf73941aba/0/saved_model.zip',
   'checkpoint-data-latest': 'b76ececa-274d-48de-b39e-70cf73941aba/ckpt-0.data-00000-of-00001',
   'checkpoint-index-latest': 'b76ececa-274d-48de-b39e-70cf73941aba/ckpt-0.index'},
  'thumb_object_name': 'b76ececa-274d-48de-b39e-70cf73941aba/effdet.png',
  'framework': 'tensorflow2'
 }

Attributes

  • model_id (string) Unique identifier of your model

  • owner (hash, user_object) The creator of the model

  • network_name (string) The name of your model

  • description (string) A short description of what your model does

  • type (string) The type of application for your model, if you want to perform pre-annotation on Picsellia it has to be one of the following (but you can set your own type otherwise):

    • 'detection'

    • 'segmentation'

    • 'classification'

  • organization (hash, organization object) The organization under which your model is stored

  • private (boolean) Tells if your model is available for everyone in the public HUB or not

  • framework (string) The framework used for training

    • 'tensorflow1'

    • 'tensorflow2'

    • 'pytorch'

  • tag (list) List of tags to identify and sort your models

  • files (dict) Dictionary containing the list of files of your model

  • labels (dict) Dictionary of the labelmap of your model

  • base_parameters (dict) Dictionary of the base parameters allowing anyone to reproduce the training or iterate with already existing parameters

  • readme_text (str) A markdown text containing more information about your model

__init__

If you want to interact with your models, you have to initialize the Network class. It is a subclass of the Client.

from picsellia.client import Client

network = Client.Network(
    api_token=None,
    organization=None,
    )

Arguments:

  • api_token (string) Your personal API token

  • organization (string, optional) the name of the organization you want to work with (None default to your organization)

list

List of all the models for an organization

By default it will list the models of your own organization but you can specify the name of another organization where you are part of the team.

network.list()

Returns:

A list containing the models of the chosen organization.

[
 {'organization': {'name': 'picsell'},
  'model_id': 'b76ececa-274d-48de-b39e-70cf73941aba',
  'serving_id': 'b76ececa-274d-48de-b39e-70cf73941aba',
  'tag': ['efficientdet', 'd2', 'COCO', 'base'],
  'private': False,
  'network_name': 'efficientdet-d2',
  'description': 'This is a real game changer',
  'model_object_name': '',
  'checkpoint_object_name': '',
  'origin_checkpoint_objects': {},
  'type': 'detection',
  'files': {'config': 'b76ececa-274d-48de-b39e-70cf73941aba/pipeline.config',
   'model-latest': 'b76ececa-274d-48de-b39e-70cf73941aba/0/saved_model.zip',
   'checkpoint-data-latest': 'b76ececa-274d-48de-b39e-70cf73941aba/ckpt-0.data-00000-of-00001',
   'checkpoint-index-latest': 'b76ececa-274d-48de-b39e-70cf73941aba/ckpt-0.index'},
  'thumb_object_name': 'b76ececa-274d-48de-b39e-70cf73941aba/effdet.png',
  'framework': 'tensorflow2'
 },
 {'organization': {'name': 'picsell'},
  'model_id': 'b76ececa-274d-48de-b39e-70cf73941aba',
  'serving_id': 'b76ececa-274d-48de-b39e-70cf73941aba',
  'tag': None,
  'private': True,
  'network_name': 'vizdrone-test',
  'description': 'This is a real game changer',
  'model_object_name': '',
  'checkpoint_object_name': '',
  'origin_checkpoint_objects': {},
  'type': 'detection',
  'files': {'config': 'b76ececa-274d-48de-b39e-70cf73941aba/pipeline.config',
   'model-latest': 'b76ececa-274d-48de-b39e-70cf73941aba/0/saved_model.zip',
   'checkpoint-data-latest': 'b76ececa-274d-48de-b39e-70cf73941aba/ckpt-101.data-00000-of-00001',
   'checkpoint-index-latest': 'b76ececa-274d-48de-b39e-70cf73941aba/ckpt-101.index'},
  'thumb_object_name': '',
  'framework': ''
  }
 ]

get

This method allows you to retrieve a particular model in order to update it or store some files.

network.get(identifier=None)

Arguments:

  • identifier (string) Either the name or the id of the model you want to retrieve

Returns:

The Network object

create

This methods allows you to create a new Network from the SDK

network.create(
    name=None,
    type=None
)

Attributes:

  • name (str, required) The name of your Network

  • type (str, required) the type of your Network, it is the task it performs such as :

    • 'detection', for object detection

    • 'segmentation', for object segmentation

    • 'classification', for image classification

Returns:

Network object

update

This methods allows you to update the properties of a Network from your Organization

You must get or create a network before calling the update method.

network.update(**kwargs)

Arguments:

  • **kwargs (required) can be any property from the Network object described in the Network Object.

For example, if you want to update the description of your Network, you code will look like this :

network.update(description="A brand new description for your Network")

You can update as many properties as you want in one single time.

store

This method allows you to upload a file and save it under your model so you can use it later such as checkpoint files or config files.

network.store(name="", path=None, zip=False)

Arguments:

  • name (string,) the name of the file, please check this page to know about the Picsellia namespace

  • path (string, optional) the path to the file you want to upload

  • zip (boolean, optional) set to True if you want to zip your file or a folder to an archive before upload

update_thumb

This method allows you to set the image of your choice as the thumbnail displayed on the platform like below.

network.update_thumb(path=None)

Arguments:

  • path (string) the path to the image file

labels

This methods allows you to set the labelmap of your model.

network.labels(labels=None)

Arguments:

  • labels (dict) Dictionary containing your labelmap (index starts at 1 not 0)

Here is the COCO labelmap as example :

labels = {
    "1": "person", 
    "2": "bicycle", 
    "3": "car", 
    "4": "motorcycle", 
    "5": "airplane",
    ... 
    "86": "vase", 
    "87": "scissors", 
    "88": "teddy bear", 
    "89": "hair drier", 
    "90": "toothbrush"
}

Returns:

None

Projects

The Project object

{
  "organization": {
    "name": "pierre-nicolas"
  },
  "project_name": "test-logging",
  "description": "No description",
  "project_id": "3d039f84-7224-4442-b499-652f2f77f1a1",
  "project_manager": {
    "username": "pierre-nicolas"
  },
  "contributors": [
    {
      "user": {
        "username": "pierre-nicolas"
      },
      "is_active": True,
      "creation_date": "2021-01-18T12:08:51.573406Z"
    }
  ],
  "datasets": [
    {
      "dataset_name": "paginate",
      "description": "",
      "private": True
    }
  ],
  "type": "classification",
  "creation_date": "2021-06-16"
}

Attributes:

  • organization (hash) The parent organization of the project

  • project_name (string) The name of the project

  • description (string) A brief description of the project

  • project_id (UUID) The unique ID of the project

  • project_manager (hash) The user that created the project

  • contributors (hash) The list of worker objects for contributors of this project

  • datasets (hash) Short dataset objects for datasets attached to the project

  • type (string) Type of the project, you can set it arbitrarily

  • creation_date (string) Date of creation of this project

__init__

If you want to interact with your projects, you must initialize the Project class for our client.

from picsellia.client import Client

clt = Client.Project(
    api_token=None,
    host="https://app.picsellia.com/sdk/", 
    project_token=None, 
)

Arguments:

  • api_token (string) Your personal API Token

  • host (string) The URL of Picsellia platform, you shouldn't have to change it

  • project_token (string) The token of the project you want work with (you can also let None and use get() or create() methods later)

Returns:

None

list

This method allows you to list every project in your organization.

project_list = clt.list()
Project Name: scans-file
Description: No description
Contributors: [{'user': {'username': 'pierre-nicolas'}, 'is_active': True, 'creation_date': '2021-01-18T12:08:51.573406Z'}]
Datasets: [{'dataset_name': 'public', 'description': None, 'private': True}, {'dataset_name': 'paginate', 'description': '', 'private': True}]
-------------
Project Name: scan-file2
Description: This is a real game changer
Contributors: [{'user': {'username': 'pierre-nicolas'}, 'is_active': True, 'creation_date': '2021-01-18T12:08:51.573406Z'}]
Datasets: []
-------------
Project Name: test-logging
Description: No description
Contributors: [{'user': {'username': 'pierre-nicolas'}, 'is_active': True, 'creation_date': '2021-01-18T12:08:51.573406Z'}]
Datasets: [{'dataset_name': 'tttt', 'description': 'None', 'private': True}, {'dataset_name': 'paginate', 'description': '', 'private': True}]
-------------
Project Name: long-projet
Description: No description
Contributors: [{'user': {'username': 'pierre-nicolas'}, 'is_active': True, 'creation_date': '2021-01-18T12:08:51.573406Z'}]
Datasets: [{'dataset_name': 'confusion', 'description': None, 'private': True}]
-------------

... 10  more...

Project Name: Benchmark-sdk
Description: No description
Contributors: [{'user': {'username': 'pierre-nicolas'}, 'is_active': True, 'creation_date': '2021-01-18T12:08:51.573406Z'}]
Datasets: [{'dataset_name': 'cloned', 'description': 'None', 'private': True}]
-------------

The project list is also stored in the project_list attribute in the Project object you instantiated so you can access it as shown below 👇

clt.project_list
[{'contributors': [{'creation_date': '2021-01-18T12:08:51.573406Z',
    'is_active': True,
    'user': {'username': 'pierre-nicolas'}}],
  'creation_date': '2021-04-20',
  'datasets': [{'dataset_name': 'public',
    'description': None,
    'private': True},
   {'dataset_name': 'paginate', 'description': '', 'private': True}],
  'description': 'No description',
  'organization': {'name': 'pierre-nicolas'},
  'project_id': 'a0b90c90-3a7e-4dda-b6d0-59d6c4c19804',
  'project_manager': {'username': 'pierre-nicolas'},
  'project_name': 'scan-file',
  'type': 'classification'},
 {'contributors': [{'creation_date': '2021-01-18T12:08:51.573406Z',
    'is_active': True,
    'user': {'username': 'pierre-nicolas'}}],
  'creation_date': '2021-07-27',
  'datasets': [],
  'description': 'This is a real game changer',
  'organization': {'name': 'pierre-nicolas'},
  'project_id': 'af76d4fe-3b8a-4587-a49f-d3bc69c19499',
  'project_manager': {'username': 'pierre-nicolas'},
  'project_name': 'scans-file2',
  'type': 'classification'},

......

 {'contributors': [{'creation_date': '2021-07-16T11:58:41.868208Z',
    'is_active': False,
    'user': {'username': 'pierre-nicolas'}},
   {'creation_date': '2021-01-18T12:08:51.573406Z',
    'is_active': True,
    'user': {'username': 'pierre-nicolas'}}],
  'creation_date': '2021-06-29',
  'datasets': [{'dataset_name': 'cloned',
    'description': 'None',
    'private': True}],
  'description': 'No description',
  'organization': {'name': 'pierre-nicolas'},
  'project_id': 'e89d03d0-77d9-4920-8899-202b1bb78cfd',
  'project_manager': {'username': 'pierre-nicolas'},
  'project_name': 'Benchmark-sdk',
  'type': 'classification'}]

get

This method allows you to retrieve a project so you can perform actions on it later.

clt.get(
    name="",
    organization="null"
)

Arguments:

  • name (string) The name of project you want to retrieve

  • organization (string) The name of the organization the project belongs to, leave "null" to stay in your organization

Returns:

self (The project object)

project = clt.get(name='test-logging')
Successfully fetched project test-logging
{
  "organization": {
    "name": "pierre-nicolas"
  },
  "project_name": "test-logging",
  "description": "No description",
  "project_id": "3d039f84-7224-4442-b499-652f2f77f1a1",
  "project_manager": {
    "username": "pierre-nicolas"
  },
  "contributors": [
    {
      "user": {
        "username": "pierre-nicolas"
      },
      "is_active": True,
      "creation_date": "2021-01-18T12:08:51.573406Z"
    }
  ],
  "datasets": [
    {
      "dataset_name": "paginate",
      "description": "",
      "private": True
    }
  ],
  "type": "classification",
  "creation_date": "2021-06-16"
}

create

This method allows you to create a new project.

You can't have 2 projects with the same name in your organization, you will have an error if the project already exists.

clt.create(
    name="",
    organization="null",
    **kwargs
)

Arguments:

  • name (string, required) The name of the project you want to create

  • organization (string, optional) The organization where you want to create the project, leave "null" if you want it to be your organization

  • **kwargs , can be any attribute of the Project object within the following list :

    • description (string)

    • type (string)

Returns:

self (the Project object)

Example:

Here we will create a new project and specify the type in the **kwargs argument

project = clt.create('new_project', type='detection')
Project new_project2 created successfully.
{
  "organization": {
    "name": "pierre-nicolas"
  },
  "project_name": "new_project",
  "description": "This is a real game changer",
  "project_id": "e2b66355-f6b2-4db9-a6c9-75de43aea428",
  "project_manager": {
    "username": "pierre-nicolas"
  },
  "contributors": [
    {
      "user": {
        "username": "pierre-nicolas"
      },
      "is_active": true,
      "creation_date": "2021-01-18T12:08:51.573406Z"
    }
  ],
  "datasets": [],
  "type": "detection",
  "creation_date": "2021-07-27"
}

As we can see, it did create a new project called 'new_project' and set the type to 'detection', easy no ?

update

This methods allows you to update some properties of your project

Before calling the update() method, you must have retrieved a project using get() or create() methods.

clt.update(
    project_name="",
    description="",
    type=""
)

Arguments:

  • project_name (string, optional) the new name wanted for the project

  • description (string, optional) The description of the project

  • type (string, optional) The type of the project

Returns:

self (the Project object)

project = clt.get('new_project')
project = project.update(
    project_name='new_project_name',
    description='a descriptive text'
)
Project new_project_name updated.
{
  "organization": {
    "name": "pierre-nicolas"
  },
  "project_name": "new_project_name",
  "description": "a descriptive text",
  "project_id": "49fc1dc9-0eb9-4aa4-ad40-0c30be3868e9",
  "project_manager": {
    "username": "pierre-nicolas"
  },
  "contributors": [
    {
      "user": {
        "username": "pierre-nicolas"
      },
      "is_active": True,
      "creation_date": "2021-01-18T12:08:51.573406Z"
    }
  ],
  "datasets": [],
  "type": "classification",
  "creation_date": "2021-07-27"
}

As we can see, we have successfully updated the name and the description of our project !

delete

This method allows you to delete a project.

Before calling the delete() method, you must have retrieved a project using get() or create() methods.

clt.delete()

Returns:

None

project = clt.get('new_project')
project.delete()
Project new_project deleted.

attach_dataset

This method allows you to attach a dataset from your organization to your project, just like you would do in the platform.

Before calling the attach_dataset() method, you must have retrieved a project using get() or create() methods.

clt.attach_dataset(
    dataset=""
)

Arguments:

  • dataset (string, required) The dataset name and version separated by as '/' : <dataset_name>/<version>

Returns:

None

project = clt.get('new_project')
project.attach_dataset('Tutorial/first')
Dataset Tutorial/first successfully attached to project new_project

Experiments

The experiment object

{
   'id': 'b8995c82-f304-40d2-a91f-7e2bcc5df429',
   'date_created': '2020-12-06T20:09:29.703959Z',
   'last_update': '2020-12-06T20:09:29.703689Z',
   'owner': {'username': 'Pierre-Nicolas'},
   'project': {
     'project_id': '9c68b4ae-691d-4c3a-9972-8fe49ffb2799',
     'project_name': 'project 21'
   },
   'name': 'exp2',
   'description': '',
   'status': 'started',
   'logging': None,
   'files': [
     {
       'id': 13,
       'date_created': '2020-12-08T23:31:13.389276Z',
       'last_update': '2020-12-08T23:31:13.388914Z',
       'large': True,
       'name': 'model-latest',
       'object_name': 'b8995c82-f304-40d2-a91f-7e2bcc5df429/saved_model.zip'
     },
     {
       'id': 14,
       'date_created': '2020-12-08T23:35:42.964512Z',
       'last_update': '2020-12-08T23:35:42.964274Z',
       'large': False,
       'name': 'config',
       'object_name': 'b8995c82-f304-40d2-a91f-7e2bcc5df429/pipeline.config'
     }
   ],
   'data': [
     {
       'id': 6,
       'date_created': '2020-12-07T13:14:15.846681Z',
       'last_update': '2020-12-07T13:14:15.846467Z',
       'type': 'value'
       'name': 'acc',
       'data': 0.8
     },

   ]
}

Attributes:

  • id (string) Unique identifier for the object

  • owner (hash, user object) The creator of the object

  • name (string) The name of the experiment

  • description (string) A short description of the experiment

  • status (string) Describes the experiment state

  • date_created (date) Date and time at which the object was created

  • last_update (date) Date and time of the last update of the object

  • logging (hash, logging object) The telemetry of the experiment

  • files (list, file-asset object) The files stored for this experiment

  • data (list, data-asset object) The data saved for this experiment

__init__

If you need to interact with your experiments, you must initialize the Experiment class. It is a subclass of the Client. It is also used to initialize some class parameters such as project_token, id (experiment id) or name (experiment name), so you don't have to specify them when you call a method of the Experiment class.

If you initialize the Experiment class with the id or name of an experiment or the project token of the project that your experiments belongs, you don't have to fill them when calling any method, they are now attributes of the class instance !

Please note that when calling an Experiment method, you can either specify the id or the name of the experiment, you don't have to enter both. 

from picsellia.client import Client

experiment = Client.Experiment(
    api_token=None,
    host="https://beta.picsellia.com/sdk/", 
    project_token=None, 
    id=None, 
    name=None, 
    interactive=True
    )

Arguments:

  • api_token (string) Your personal API token

  • host (string, optional) the URL of Picsell.ia platform, you shouldn't have to change it

  • project_token (string, optional) the token of the project you want to focus on

  • id (string, optional) the id of the experiment you want to work with

  • name (string, optional) the name of the experiment you want to work with

  • interactive (boolean, optional) if set to False, disable all prompt that requires user action

Returns:

None

checkout

This methods allows you to retrieve all the information and assets of an experiment. For more explanation on how this method works, please have a look at this page.

experiment.checkout(
    id=None, 
    name=None, 
    project_token=None, 
    tree=False, 
    with_file=False, 
    with_data=False
    )

Arguments

  • id (string, optional) the id of the experiment you want to work with

  • name (string, optional) the name of the experiment you want to work with

  • project_token (string, optional) the token of the project you want to focus on

  • tree (boolean, optional) set to True if you want to create training-ready folders (see the training reference)

  • with_files (boolean, optional), set to True if you want to download all the files of your experiment

  • with_data (boolean, optional), set to True if you want to retrieve all data assets from your experiment

Returns:

JSON-like object

{
 'experiment_id': 'b76ececa-274d-48de-b39e-70cf73941aba',
 'date_created': '2020-12-12T16:26:59.343322Z',
 'last_update': '2020-12-12T16:26:59.343090Z',
 'owner': 1,
 'project': '9c68b4ae-691d-4c3a-9972-8fe49ffb2799',
 'name': 'my_new_experiment',
 'description': 'dog cat classificaiton with base parameters',
 'status': 'started',
 'logging': None,
 'files': [],
 'data': []
 }

list

List all of the experiments related to one project

This method needs to know your project token, you can either fill it when initializing the Experiment class or you can fill it when calling the method

experiment.list(project_token=None)

Arguments:

  • project_token (string, optional) the the token of the project you want to focus

Returns:

A list containing the experiment objects for your project.

[
 {'experiment_id': 'b76ececa-274d-48de-b39e-70cf73941aba',
   'date_created': '2020-12-12T16:26:59.343322Z',
   'last_update': '2020-12-12T16:26:59.343090Z',
   'owner': {'username': 'Pierre-Nicolas'},
   'project': '9c68b4ae-691d-4c3a-9972-8fe49ffb2799',
   'name': 'my_new_experiment',
   'description': 'dog cat classificaiton with base parameters',
   'status': 'started',
   'logging': None,
   'files': [],
   'data': []},
  {'experiment_id': 'b251ca51-ba03-4144-a1e7-9eb15db46984',
   'date_created': '2020-12-06T20:10:19.101950Z',
   'last_update': '2020-12-06T20:10:19.101772Z',
   'owner': {'username': 'Pierre-Nicolas'},
   'project': '9c68b4ae-691d-4c3a-9972-8fe49ffb2799',
   'name': 'desc',
   'description': '',
   'status': 'started',
   'logging': None,
   'files': [],
   'data': []}
 ]

create

Creates a new experiment for a project

The id parameter is used to iterate from a previous experiment, if you need more details, please look here for an explanation.

experiment.create(
    name=None,
    description='',
    previous=None,
    dataset=None,
    source=None,
    )

Parameters:

  • name (string, 60 char max.) the name of your new experiment

  • description (string, 1000 char max., optional) a short description for your experiment

  • previous (string, optional) set it to a previous experiment name if you want to iterate

  • dataset (string, optional) set it to <dataset_name>/<version> of the dataset you want to use for your experiment

  • source (string, optional) set it to <username>/<model_name> to clone the assets of a model from our model HUB or your organization

Returns:

An experiment object with your created experiment

 {
 'experiment_id': 'b76ececa-274d-48de-b39e-70cf73941aba',
 'date_created': '2020-12-12T16:26:59.343322Z',
 'last_update': '2020-12-12T16:26:59.343090Z',
 'owner': 1,
 'project': '9c68b4ae-691d-4c3a-9972-8fe49ffb2799',
 'name': 'my_new_experiment',
 'description': 'dog cat classificaiton with base parameters',
 'status': 'started',
 'logging': None,
 'files': [],
 'data': []
 }

update

From here, in order to run all the following methods, you mst have checkout or created an experiment first.

This methods allows you to update the information of an experiment

experiment.update(
    **kwargs
    )

Arguments:

  • **kwargs

    • name (string) the new name for your experiment

    • description (string) the new description

    • status (string) the new status

Returns:

The updated experiment object


{'id': 'b76ececa-274d-48de-b39e-70cf73941aba',
 'date_created': '2020-12-12T16:26:59.343322Z',
 'last_update': '2020-12-12T16:26:59.343090Z',
 'owner': 1,
 'project': '9c68b4ae-691d-4c3a-9972-8fe49ffb2799',
 'name': 'updated_name',
 'description': 'dog cat classificaiton with base parameters',
 'status': 'started',
 'logging': None,
 'files': [],
 'data': []}

delete

This method allows you to delete one experiment

experiment.delete()

Returns:

True if the experiment has been deleted correctly

delete_all

Delete all the experiments for a given project

experiment.delete_all(project_token=None)

Arguments:

  • project_token (uuid, optional) the project token of the project you want to delete all experiments

Returns:

True if the experiments has been deleted correctly

dl_annotations

Download the annotations JSON for the dataset attached to an experiment

experiment.dl_annotations(type='all')

Arguments:

  • type (string, required)

    • 'all' (default) download all annotations of the dataset

    • 'accepted' download only accepted annotations from the dataset

Returns:

JSON containing all of the annotations in the COCO format, the JSON is also stored in experiment.dict_annotations

File-assets

The file-asset object

{
   'id': 13,
   'date_created': '2020-12-08T23:31:13.389276Z',
   'last_update': '2020-12-08T23:31:13.388914Z',
   'large': True,
   'name': 'model-latest',
   'object_name': 'b8995c82-f304-40d2-a91f-7e2bcc5df429/saved_model.zip'
}

Attributes

  • id (int) Unique identifier for the object

  • name (string) The name of the file-asset

  • object_name (string) The path to the file in our object storage

  • date_created (date) Date and time at which the object was created

  • last_update (date) Date and time of the last update of the object

store

Create a file-asset for your experiment and upload the file on our object storage. Please check this page if you need more details about what it does in the background and how to use it properly.

experiment.store(name="", path=None, zip=False)

Arguments:

  • name (string,) the name of the file, it will be displayed on Picsell.ia

  • path (string, optional) the path to the file you want to upload

  • zip (boolean, optional) set to True if you want to zip your file or a folder to an archive before upload

list_files

List all the files belonging to one experiment

experiment.list_files()

Returns:

A list of all the file asset object of your experiment

[{'id': 22,
  'date_created': '2021-02-09T12:32:18.022694Z',
  'last_update': '2021-02-09T12:32:18.022465Z',
  'large': False,
  'name': 'config',
  'object_name': '9a141ede-03dc-4e6c-a695-38661d9a97c3/pipeline.config'},
 {'id': 23,
  'date_created': '2021-02-09T12:32:18.068900Z',
  'last_update': '2021-02-09T12:32:18.068723Z',
  'large': True,
  'name': 'model-latest',
  'object_name': '9a141ede-03dc-4e6c-a695-38661d9a97c3/0/saved_model.zip'},
 {'id': 24,
  'date_created': '2021-02-09T12:32:18.112582Z',
  'last_update': '2021-02-09T12:32:18.112410Z',
  'large': True,
  'name': 'checkpoint-data-latest',
  'object_name': '9a141ede-03dc-4e6c-a695-38661d9a97c3/ckpt-0.data-00000-of-00001'},
 {'id': 25,
  'date_created': '2021-02-09T12:32:18.156945Z',
  'last_update': '2021-02-09T12:32:18.156767Z',
  'large': False,
  'name': 'checkpoint-index-latest',
  'object_name': '9a141ede-03dc-4e6c-a695-38661d9a97c3/ckpt-0.index'}]

delete_all_files

Delete all the files belonging to one experiment

experiment.delete_all_files()

Returns:

True if all the files has been deleted correctly

get_file

Retrieve one particular file of an experiment by its name

experiment.get_file(name=None)

Arguments:

  • name (string) the name of the file asset you want to retrieve

Returns:

A file-asset object

{'id': 22,
 'date_created': '2021-02-09T12:32:18.022694Z',
 'last_update': '2021-02-09T12:32:18.022465Z',
 'large': False,
 'name': 'config',
 'object_name': '9a141ede-03dc-4e6c-a695-38661d9a97c3/pipeline.config'}

delete_file

Delete a file from your experiment

experiment.delete_file(name=None)

Arguments:

  • name (string) the name of the file asset you want to delete

Returns:

True if the file has been deleted correctly

update_file

Update the information of a given file

experiment.update_file(file_name, **kwargs)

Arguments:

  • file_name (string) the name of the file asset you want to update

  • **kwargs

    • name (string) the new name for your file

Returns:

A file-asset containing the updated object

Data-assets

The data-asset object

{
   'id': 72,
    'date_created': '2021-02-09T12:32:18.293746Z',
    'last_update': '2021-02-09T12:32:18.293556Z',
    'name': 'parameters',
    'data': {'steps': 200000,
     'nb_gpu': 1,
     'batch_size': 8,
     'learning_rate': 0.005,
     'annotation_type': 'classification'
     },
    'type': 'table'
   }

Attributes

  • id (int) Unique identifier for the object

  • name (string) The name of the data-asset

  • data (hash, data object) The data logged in this object

  • date_created (date) Date and time at which the object was created

  • last_update (date) Date and time of the last update of the object

log

Log your results to Picsell.ia, check this page for more details on how to use it and what you can do with logs.

experiment.log(name=None, data={}, type="")

Arguments:

  • name (string) the name of the data-asset you want to create

  • data (dict) a dictionary containing the data you want to display on the platform

  • type(string, optional) the type of asset you are logging, see all the types available here

list_data

List all the data-assets belonging to one experiment

experiment.list_data()

Returns:

A list of all the data-assets object of your experiment

 [
  {
   'id': 72,
    'date_created': '2021-02-09T12:32:18.293746Z',
    'last_update': '2021-02-09T12:32:18.293556Z',
    'name': 'parameters',
    'data': {'steps': 200000,
     'nb_gpu': 1,
     'batch_size': 8,
     'learning_rate': 0.005,
     'annotation_type': 'classification'
     },
    'type': 'table'
   }
  ]

delete_all_data

Delete all the files belonging to one experiment

experiment.delete_all_data()

Returns:

True if all the data-assets has been deleted correctly

get_data

Retrieve one particular file of an experiment by its name

experiment.get_data(name=None)

Arguments:

  • name (string) the name of the data-asset you want to retrieve

Returns:

A data-asset object

You will only return what is stored in the data field of the data asset, not all the information about the asset.

{'steps': 200000,
 'nb_gpu': 1,
 'batch_size': 8,
 'learning_rate': 0.005,
 'annotation_type': 'classification'}

delete_data

Delete a file from your experiment

experiment.delete_data(name=None)

  • name (string) the name of the data-asset you want to delete

Returns:

True if the data-asset has been deleted correctly

update_data

Update the information of a given file

experiment.update_data(data_name, **kwargs)

Arguments:

  • data_name (string) the name of the data-asset you want to update

  • **kwargs

    • name (string) the new name for your data-asset

    • data (dict) the updated data for your asset, please refer to this page for details on the data objects

Returns:

A data-asset containing the updated object

{
 'id': 72,
 'date_created': '2021-02-09T12:32:18.293746Z',
 'last_update': '2021-02-09T12:32:18.293556Z',
 'name': 'parameters',
 'data': {'steps': 5000000.0,
  'nb_gpu': 8,
  'batch_size': 64,
  'learning_rate': 0.0055,
  'annotation_type': 'detection'},
 'type': 'table'
 }

Logging (coming soon)

Logging is the telemetry (if existing) of your experiment. It allows you to record in real time what's happening in the interpreter during an experiment and to monitor it from the platform.

The logging object

Attributes

  • logs (dict) The raw stdout of your interpreter

  • steps (dict) The categories of your logs, useful to navigate inside your logs

  • timestamps (dict) The timestamps for each step

  • exit_code (dict) The exit code and the timestamp of the end of your run

PreviousAPI ReferenceNextPython Training Reference

Last updated